[itsoktocry]

>that if you give everyone access all the time the magic will happen

There's much ongoing discussion about this is the data world, often revolving around "self-service analytics".

Unless you're talking about "our analysts don't have to clean data all the time", which, for a large enough organization makes sense, "self-service" for non-technical folks is futile and pointless. They need specific answers to specific questions, not the ability to infinitely explore the data. Organizations should desire that kind of focus, not prevent it.

[datavirtue]

They idea was that they were going to hire an army of data scientists and become google...magically.

Reality smacked that shit down hard. I left data engineering because the projects were all over the place, wildly undisciplined and unfocused.

You were lucky to have source control let alone an understanding from the business that these projects were in fact software development.

I switched back to software engineering because at least there is a faint realization that we are...building software.

I might go back when the dust clears.

"Why do we need to hire programmers...I thought we needed data engineers?"

"Because the data pipelines are all built with thousands of lines of code. Java, python, Fortran, you name it...and your job post only mentioned SQL and data modelling"

I could go on forever.

[mason55]

This is the constant argument I have with people about data products.

You don't need to expose more dimensions or get the users more access to the raw data. You need to understand what their business is and what their business problems are and help them answer those specific questions quickly and succinctly.

Yes, there are certainly times where people use huge amounts of raw data to uncover the answer to a question they didn't know they had. But it's rare, it's expensive to support, and most businesses are going to be able to do anything with it anyway (a whole org built to do X isn't suddenly going to shift to do Y because you discovered some insight in a random report).