[joe_the_user]

Even more,

I don't notice an effort to expand the workforce by training or by recruitment of non-traditional workers, etc.

The contra-logical statement "99% of programming applicants are unqualified" gets a lot of play in this field. But I would suggest something like "we can make 99% of applicants look like idiots with our circus-like hiring process".

Yes, we've decided we have a shortage once we decide on five arbitrary disqualifications, expect all applicants to work 18 hours a day and start yesterday having no time to get up to speed (so experience on earlier large systems, say, is indeed not useful).

[jandrewrogers]

The base-level skill set is being a very good applied mathematician with some good computer science skills. This is why a lot of "data scientist" types have degrees in things like physics. A lot of the database ETL stuff can be learned.

This is the reason why I cannot be a "data scientist", despite being an expert in parallel algorithm design and with strong database ETL experience. It would require me spending a couple years studying mathematics in depth that I do not currently know. The vast majority of programmers are at least as deficient as I am in critical skills for these positions.

We train our data scientists at my company but we usually do not start with software engineers. Our feedstock is strong applied mathematicians with some programming skills because the mathematics part is by far the most difficult to train for someone who has not already been doing it for years.

[achompas]

This is the reason why I cannot be a "data scientist"

Are you worried about this outcome at all? Do you see yourself playing an important role on a data team, one with less modeling responsibilities but more infrastructure/DB responsibilities?

I'm considering this path and would love to hear your opinion.

[jandrewrogers]

To be clear, I chose this outcome. I am good with mathematics but not the mathematics usually needed as a data scientist and I have relatively little interest in investing the time to learn. Being a data scientist is a great job for some people but probably not what I would choose even if I was a developer again.

There is a continuum of skill balances; some people are more "data" than "scientist" and vice versa. The most useful balance varies from job to job. There are plenty of opportunities for people that have strong skills standing up clusters even if you have relatively weak analysis and model building skills. I would not dissuade anyone from becoming a data scientist, it will pay very well for the foreseeable future, but the skill set requires real effort to acquire. At a small company there is likely opportunity to learn the trade by coming at it from the infrastructure side of things.

It is a young enough area that it should be pretty easy for talented individuals to invent a career if they apply themselves.

[achompas]

Thanks for the excellent summary. This line:

I am good with mathematics but not the mathematics usually needed as a data scientist

resonates with me. If you're thinking of data science, you're facing a loooooong road of coursework (scientific computing or numerical methods, linear algebra, PGMs, machine learning, AI, possibly some optimization too) to get your foot in the door. I'm going to try, but one could spend years finishing that work.

In some ways, getting a data science gig is the opposite of getting a web developer gig. In DS you're competing with a large supply of intelligent PhDs, so credentials are very important; for web dev, your portfolio goes much further than any credentials.

[greenyoda]

That's a very good point: insisting that a single person must have skills in math, computer science and data interpretation is creating a purely arbitrary set of qualifications.

If we make a comparison with other fields, we can see that trying to find a single person who has skills in several diverse areas is not something that's usually done. For example, do companies try to hire bond traders who can implement their own trading software? Or do we insist that airline pilots or surgeons or CEOs should be able to build and repair the technology they use?

And if a company did manage to find a person who was both a good statistician and a good software developer, wouldn't the combination of responsibilities pull this person in too many directions, making it hard to focus on on what they were doing? Also, it would take a lot of effort to stay current with the latest developments in both math and computer science.

If a company was too small to be able to afford to hire three full-time specialists to analyze their data, they could outsource their data crunching needs to consulting companies that specialized in this kind of work.

I also have a problem with the newly-coined term "data science". Scientists are engaged in discovering fundamental truths about the way the physical world works. I don't think that finding trends in a company's data counts as science. (I don't think that 99% of "computer scientists" are scientists either, including the professors I knew in grad school.) I liked the older term "data mining" much better, but I guess it's not trendy enough anymore.

[achompas]

I don't notice an effort to expand the workforce by training or by recruitment of non-traditional workers, etc.

"Code Year" for data scientists would actually need:

-- "Code Previous Year," where everyone would need to kick ass on Bayesian inference, linear algebra, and production-level software development; then,

-- during "Code Year," they'd proceed to learn about distributed algorithms, graphical models, and HMMs, then learn about distributed frameworks like Hadoop.

See Joseph Misti's comment here for more--it's really the most accurate list of skills one needs to become a data scientist:

http://www.quora.com/What-skills-are-needed-for-machine-lear...