[ElysianEagle]

I think the same thing that happened to quants on Wall Street will happen to those pursuing Data Science/ML today (if it hasn't already happened). Post-2008 there was such a glut of qualified quants that companies moved the goalpost and now it's very difficult to even be considered for a role if you don't have a PhD.

With so many programs and courses springing up about ML, in a few years I suspect there will be a glut of Data Scientists on the market, and once again companies will use the PhD as a filter. Already it seems like a good number of places will only hire PhDs for such roles.

[Homunculiheaded]

As far as I can tell Wall Street has always had a penchant for pedigree in addition to skills. Large companies are often the same. People used to talk about all software engineers needing CS degrees in the future. Again in Wall Street and large companies you'll see this, but there are still plenty of well paid, talented software developers with no formal CS training.

In my experience hiring and chatting with other people hiring data scientists, there's the same trouble as there is with software engineers. No matter how many people have the training there's still a dearth of applicants that are truly talented and can actually do things. PhDs fleeing academia for a promise of easy employment and money are a huge bulk of new data scientists that I've seen and most of them have a very hard time taking deep knowledge and applying it to solve real-world problems.

At least in tech I think the future of Data Science lies in the perpetually small group of people that will have a proven track record of coming into companies and actually solving problems, just as it has been in software development.

[mindcrime]

I think the same thing that happened to quants on Wall Street will happen to those pursuing Data Science/ML today (if it hasn't already happened). Post-2008 there was such a glut of qualified quants that companies moved the goalpost and now it's very difficult to even be considered for a role if you don't have a PhD.

I'm not sure that's a valid comparison. There's a relatively fixed and fairly small pool of companies who need quants. Machine Learning, OTOH, can be used by almost any company in existence (even if most of them don't realize it yet). And plenty of companies don't need somebody doing cutting edge academic research in ML... they need somebody who can use a pre-packaged library or service and apply linear regression, or k-means, or build a simple neural network with backprop.

[nostrademons]

It can't, though. ML works best when you have enough data to build models that are robust and resilient to noise, and enough customers (or users) that these models will move the needle.

The vast majority of startups and small businesses - those whose customer base measures in the dozens to hundreds - should be going out, engaging their customers person-to-person, and looking for qualitative data, because that's what'll move the needle on their sales. There's no point in understanding "your customer base" as a unit until it's big enough that it behaves, statistically, as a unit; instead, you should be focusing on "your customers", individually. Once you get into the thousands of customers you can start applying some basic learning models, and once you get into the millions machine-learning becomes as fundamental as pricing.

But you gotta get there first, and many businesses haven't. And even if they have, userbase-wise, they need to build the infrastructure (through web & mobile devs, backend engineers, data scientists, etc.) to log, store, and clean all that data before they can apply machine-learning to it.

[mindcrime]

But you gotta get there first, and many businesses haven't.

Agreed. But many have as well. So I'll still argue that there are more potential positions for people doing "applied ML" than there are for quants. I'm open to being proven wrong though.

And even if they have, userbase-wise, they need to build the infrastructure (through web & mobile devs, backend engineers, data scientists, etc.) to log, store, and clean all that data before they can apply machine-learning to it.

We're working on a MLaaS offering to help reduce the need for a lot of that stuff. And there are some offerings in that space already.

[sbisker]

Once it becomes straightforward to do ML with a pre-packaged library, you'll quickly start to see Amazon or a third-party offer effective MLaaS. That will suck quite a bit of oxygen out of the room.

[nostrademons]

Already exists:

https://aws.amazon.com/machine-learning/

Problem is, writing an effective machine-learning model already doesn't require knowing the algorithms well. It requires knowing your data well. You can provide tools for this, and AML does, but there's no substitute for actually working with the data day-in-and-day-out and developing an intuition for it.

(Deep learning promises to change that a bit, since the relevant features are extracted for you by the algorithm and you don't need to do any particular data cleaning or feature extraction work. You still need to understand your data well to understand how to train the model, though, and how to apply primitive ML operations - classification, regression, clustering, etc. - to a real-world problem.)

[mindcrime]

You still need to understand your data well to understand how to train the model, though, and how to apply primitive ML operations - classification, regression, clustering, etc. - to a real-world problem.)

And this is the kind of stuff that I believe can be done by people who don't necessarily need phd's in Stats or ML. A decent grounding in statistics / ML, and good domain knowledge should be enough to support using pre-packaged algorithms to solve business problems.