[apohn]

>I spend a lot of my time explaining that there must first be a business objective, a key question, or hypothesis that can then be understood through data. I cannot take a haystack and find the needle that is interesting to you. And if I do find that needle, many times there are no resulting changes made to our strategy.

IMO a number of data science positions should be considered partly research positions. You are hiring somebody think critically about how to generate high value/impact from data. This includes exploring if there is a different way to think about a business problem than it has been formulated in the past. This may include defining and collecting data when you discover the existing (or non-existent) data isn't appropriate. As with any research, you'll sometimes realize the path you are on is wrong and a correction is needed.

The "find all the needles in this haystack" is a totally different worldview and throws a lot of critical thinking out the window. I think this really plays into the idea that an organization can hire a person who is going to do immediate "magic" with algorithms and zero effort beyond that. You can slice/dice and p-hack your way into a million thoughtless and useless "insights."

[itronitron]

The organizations I have seen that do best at this have teams of data scientists collaborating with devs/engineers and business analysts... there need to be a lot of different research activities going on most of which are working off the same data/compute infrastructure but with some people dissatisfied and pushing the edge of course. Also regarding hiring pipelines I would discourage hiring based on technology keywords as anyone that is a good fit should be intelligent and curious enough to pick up their new employer's tech stack relatively quickly.