[_delirium]

I'm not sure that the kinds of employees that this article describes will ever be a large number. There could be more of them in the future, but someone who is top-notch at all of statistics, programming, and data-presentation has long been less common than someone who's good at one or two of those. Companies might consider looking at better ways to build teams that combine talent that exists, instead of pining for more superstars.

I'm reminded indirectly of an acquaintance of mine who works on repairing industrial machinery, where companies complain of a big skills shortage. They either fail to realize or are in denial about what that means in the 21st century, though. It might've been a one-person job in the 1950s, a skilled-labor type of repairman job. But today they want to find one person who can do the physical work (welding, etc.), EE type work, embedded-systems programming (and possibly reverse engineering), application-level programming to hook things up to their network, etc. Some of these people exist, but it's more common to find boutique consulting firms with 3-person teams of EE/CE/machinist or some such permutation. But companies balk at paying consulting fees equivalent to three professional salaries for something they think "should" be doable by one person with a magical combination of skills, who will work for maybe $80k. So they complain that there is a shortage of people who can repair truck scales (for example).

[mgkimsal]

I see this in software dev consulting, and it's probably in many other fields as well too.

I see companies looking for one person who's a highly-skilled DBA, sysadmin, developer and who can interact affably with all levels of people in a company, including customers, at the drop of a hat. Often it's because they had one 'magical' person who did all that, though usually not very well, and the next person (or team) who comes in after on that project has to deal with a bundle of undocumented crap that never really 'worked', but worked well enough to keep some people happy.

This scenario has been surprisingly common, and I worked with a company last year who was committed enough to hire dedicated dba and sysadmin positions, vs continuing to rely on app devs to handle all that stuff. The separation of concerns has worked out pretty well, though at first there was some concern about the cost of adding 'dedicated' people. In reality, all that work was being done anyway, often in ways that weren't terribly understandable by anyone outside the project.

The time it takes to get feature X done, tested, db upgraded, rolled out, servers maintained, patches applied, etc... is going to be roughly the same. If that's going to take, say, 100 hours, it's either 3 weeks of one person, or 1 week of 3 people. It's not worked out quite that cut-and-dried, but it's coming close. And the ability for each person/team to focus on their core skills and let someone else handle the "other stuff" has meant that, generally, the quality of things is better all around.

[nahname]

There is a dark side to the separation of concerns. Lack of understanding. One person doing three jobs can fully understand what choices make the overal work more efficient. One DBA, one sysadmin and one dev only have direct insight into their own work. People, in my experience, can only optimize for what they understand and generally only care about their own work.

I believe this is why we are seeing roles, like devops becoming popular. Specialization introduces a communication overhead and most companies already do a terrible job of employee communication. Merging tightly coupled roles back together helps reduce friction and improve productivity.

In a small example such as your own, it probably works out closer to pipelining. The more common case, from my experience, is that the communication overhead and lack of understanding eat up more and more of your time as each new person is introduced. Law of diminishing returns takes effect and a year later everyone is always in meetings.

[mgkimsal]

There's a balance to be struck. Specialization brings with is tunnel-vision, and having the bulk of people on a dev team have a decent understanding of the other parts of the stack is certainly useful to help avoiding that.

Jack-of-all-trades devs have a place, but dedicated people in specific roles also have a place. Those places will change and move over time as the nature of the project and the business changes (initial dev in to maintenance, early market upstart vs established leader, etc). Understanding and accepting that role changes may be necessary is probably the hardest thing for some business to accept, and properly making those changes (filling roles with good hires) is arguably one of the hardest things to execute on.

[bmelton]

Perhaps slightly offtopic, but at least in Software dev consulting, there's a market for the multi-hatted individual in consulting directly, and pay is generally commensurate with the number of hats you can speak intelligently on to a customer.

This isn't necessarily true everywhere -- when I lived in Memphis, TN, I often felt that I was dooming my professional career as, every time I ran into a challenge, I'd fork my efforts and start tackling it. This means that I was getting skilled in (but not expert in) a wide variety of things. In short, my knowledge set was very wide, but somewhat shallow.

It wasn't until I moved to the DC area that I realized there's not only a market for that type of person, but a fairly lucrative one.

There are other tracks too, Architect, Management, whatever... whereas the scale repairman who can weld, machine and EE on a scale system is not exactly limited to just working on scale systems, but isn't necessarily opened up to as many positions as with software dev.

[OzzyB]

I don't think your off-topic at all. I think your point speaks to the core of the issue at hand, which is, companies only want to hire folks who have mastered a particular skillset that they only deem important to them at that specific point-in-time.

I don't for one minute think there aren't enough people who can't do "data driven analysis" or whatever it is that companies find important "today". Companies just don't want to invest their time and money in creating that someone who will be able to do the specific thing that they need.

They always seem to think that someone will be "out there" to solve their specific need, and all it takes is a job posting on some job board.

Take a note from the old record label industry, that would have A&R departments, who would take on an individual artist who had merit, and then nurture them to become the world-class musician that makes them a ton of money.

[mitjak]

The record industry analogy is brilliant. Major labels of the past were not unlike production lines. They weren't just management and promotion companies, but rather more like schools where promising resident talent was honed and music would be written for and played by most fitting artists. If one goes through hits of the 50-70s, they will find that the same songs by one label would often be performed by multiple artists, sometimes for years, before that final breakthrough performance would come along when the perfect match between the singer and the song would be found.

It was, then, a matter of knowing what to look for in new talent and finding ways to bring out their strengths through good and appropriate songwriting. People these days often complain about the quality of music being lower than it used to be, but the truth is that for every hit of the early times there would be many times the number of "duds" and just unsuccessful attempts that would either be mediocre songs or bad matchings between the artists and the records they perform.

IMO, despite even that, I still personally think records of the past are a lot more full of substance and soul than anything released today.

[JeffBlauser]

That's good to hear you found your niche. I sometimes look back with pangs of regret, to be honest. When my compan(ies) needed visual basic help, I figured it out. Then never used it again. Then repeated that over and over with asp, php, sas, sysadmin, graphic design, crm, erp, even non-tech things such as accounting, loss prevention in varying degrees... So I can't really apply as an 'expert' at any. I don't dwell on it, but I am not sure if I would do it exactly the same way again.

[dredmorbius]

It also depends on what it is you "go wide" on.

Early in my career I had the option of taking the proprietary track (mainframes, DEC/AXP, Microsoft Windows NT, several other proprietary platforms), or the open one (Unix, Linux, shell tools, GNU toolchain). One thing I realized was that just on an ability to get my hands on the tools I wanted to use, the open route was vastly more appealing. It's gotten somewhat better, but you could still easily pay $10-$20k just for annual licenses for tools you'd use.

The other benefit I found was that there was a philosophy of openness and sharing which permeated the open route as well. I've met, face to face, with the founders of major technological systems. And while there are many online support channels for proprietary systems, I've found the ones oriented around open technologies are more useful.

Dittos on training for proprietary systems. It's wonderful ... if you want to learn a button-pushing sequence for getting a task done, without a particularly deep understanding of the process. The skills I've picked up on my own or (very rarely) through training on open technologies have been vastly more durable.

[robomartin]

I think you are very much on point with your comments.

I am one of those people who are actually capable of going from MIG welding to designing websites, writing iOS apps, developing embedded hardware and software as well as GHz-range electronics with FPGA's, mechanical design and FEA.

The only option for someone like me seems to be to run your own business. Nobody is likely to pay for the combined skill set. Which means that having these skills is both a blessing and a curse depending on your point of view. I can take any product from drawing to completion. My resume scares most employers. And, in many ways, rightly so.

Hiring someone who can do the work of five people is very risky. You loose one person and your entire team is gone. And, of course, there is no way that one person can have the productivity offered by a team of specialists.

In the end, someone with my skill set either ends-up doing their own thing or in a managerial position where the wide knowledge base and context gained from actually being able to do the work can be harnessed to guide and assist a team in achieving the required goals.

As an entrepreneur, having a wide skill set can be priceless so long as you start letting go as soon as you can start hiring specialists. This can be hard for some. It's great to be able to do it all when you want to launch something and save a bunch of money. Once launched, you need to divest yourself from responsibilities as quickly as possible because you will hit productivity walls and you simply can't focus on everything at the same time.

The age of the generalist is pretty far gone. If employment is the goal it is best to focus on one subject and become really good at it.

[ohyes]

I think another issue is that no one is interested in doing 'on the job' training. Few people learn statistics, programming, and data-presentation in college (not all three anyway). Companies might consider finding someone smart with one or two of the skills and expecting them to learn the other skills on the job. And what is talent? To me, talent is the ability to learn to do something quite well. To say there is a lack of 'talent,' when you have only searched for people who /already/ have those specific skills is disingenuous. You have barely looked.

Who is complaining about the lack of talent? Companies! Who produces talent? Companies! Who can fix this? Companies!

Part of running a company efficiently is is hiring, grooming and retaining juniors (who will become seniors). For God knows what reason, few companies actually understand this. If the skills you are looking for are lacking, hire someone smart and teach them.

[_delirium]

I agree somewhat, but retention is a fairly major problem. The shift away from career-length employment means that neither employers nor employees assume there will necessarily be much loyalty or longevity in the relationship. I think the decline in on-the-job training is directly related. Engineering firms used to be able to assume that it's okay to lose money on the first five years or so of an employee's work, if they built up skills that will make the company lots of money over the next 30-40 years of the employee's career. But if the company invests five years of significant training in a junior employee, and then the employee jumps ship to do freelance consulting or work for a competitor, the training never ends up paying for itself.

[ohyes]

That's definitely true, but I think that a part of on the job training is building loyalty. If you like the people you work with and the salary and benefits are pretty decent, you are not likely to want to go looking for another job.

For my last job (at a really big company), the only time that substantially increasing my salary came up was when I was already on my way out the door, and they realized 'oh shit, we really depend on this guy' and tried to counter offer.

I would get glowing reviews but the maximum my salary could possibly ever increase in a year was 5%. Changing companies it could increase by as much as 50-60%. So we can say that it is a 'bad relationship' but mostly it is simple math. Of course you are going to have to pay the highly skilled person a salary that is commensurate to their skills. Additionally, you should work to keep them at the salary the market will bear rather than waiting for them to get a better offer from someone else. It will cut into your profit margin, but it is a lot better than being on the defensive and having to counter offer against a company that the employee has already talked herself into wanting to be at.

[sliverstorm]

The whole thing seems to be a problem created by the volatility and rapid growth/movement of the tech industry. Job loyalty originally started to decline in part because companies didn't last, and began a downward spiral as logical "next steps" were taken, such as cutting on the job training.

As for salaries, I think the gradual increase was historically normal, but the potentials for dramatic growth in skill and effectiveness is new and enabled by the tech sector. We haven't figured out how to cope with it yet. (No, you can't just give everyone 50% raises YOY).

[mr-ron]

What prevented employees from jumping ship before? Was there better long term benefits associated with staying with a company?

[_delirium]

Yes, on the latter point: vacation time and pensions typically increased based on length of service with the company. You ended up with a much worse pension if you had 10 years' service with each of four companies, than if you had 40 years' service with one company, under the traditional defined-benefit pension schemes.

There are probably a lot of cultural changes influencing it though, perhaps more; changing jobs frequently as a salaried professional just wasn't something many people in my dad's generation actively considered. One of many factors might be the change in how promotions are done; it used to much more often be "within the ranks". You worked your way up to FooBar VP or even FooBar CEO by starting in a regular job and getting promoted up the ladder, which required staying at the company for a long time. Now it's more common to hire external people right into senior posts.

[easp]

Yeah, one of the long term benefits was that you could work there your whole career, if you wanted to. By the mid-80s though, it was clear that layoffs were to become a regular fixture of corporate life.

[disgruntledphd2]

I completely take most of your points, but I think that pretty much all quantitative PhD's are going to be close to "data scientists". Given that stats and explaining your research are requirements, all that's left is to train them to program, which a lot of people are already doing. As a matter of fact, since I heard about this big data stuff I've been honing my skills in this area, in case the hype actually manifests.

[Dn_Ab]

_delirium's post acknowledges your point but is looking for an even rarer person:

"There could be more of them in the future, but someone who is top-notch at all of statistics, programming, and data-presentation has long been less common than someone who's good at one or two of those".

Someone that can program, understands statistics and can present the data in an appealing manner without losing significant fidelity. Many people underestimate the difficulty and skill required in presenting data in a way that makes sense and also actually says something.

There is a significant gap between presenting data that is satisfactory to a research advisor and something that a business person with barely enough time to think can grasp without misconception.

[disgruntledphd2]

Again, I completely see the difference (and am actually in the process of moving full time to the private sector from academia, so will probably understand a lot more in six months) but visualising data well is not that hard. Step 1: learn R Step 2: Learn PCA Step 3: Learn ggplot2 Step 4: play with the different geoms until you understand them (seriously though, everyone's eyes are optimised to find patterns, and if you can apply significance testing to these then you should be good) Step 5: profit!? Note that I am being somewhat facetious here, but I suspect that the mathematical knowledge and ability to apply this to business problems will be the real limiting factors, as good practices in data analysis, programming and visualisation can be learned. Granted that will take a long time to learn, and there will be individual differences, but its doable.

Whether or not it will be done at all though is another matter.

Again, delirium's point is trivially true if one requires these people to know all of statistics, programming and data presentation as I don't think there's anyone who knows all of any one of these subjects.

I suppose it somewhat depends on what the skill levels for each of these areas need to be, and that varies from person to person as well as from application to application.

[ryanlchan]

Allow a short vignette from a former academic and now management consultant.

We spent six months at a major pharmaceuticals client examining their reimbursement data. Poring over many millions of rows of transaction data and thousands of payment codes (which, of course, were unique across sales geographies), we determined the ten regions at highest risk of reimbursement collapse. R was used, maps were created, beers all around.

But almost none of it was used for the executive presentation. In fact, the only part that was included was that we had ten regions that needed fixing, and our suggestions on how to fix it. You see, the CEO was dyslexic, the chairman of the board was colorblind, and the COO was a white-boarding kind of gal, so given this audience the nuts and bolts of our advanced statistical analysis were simply irrelevant.

This is hardly surprising. If we are having so much trouble hiring people who are fluent in Big Data, how can we expect business leaders to be even conversant? With only slight exaggeration, the way you do your analysis and the visualizations that you create are not important.

Companies are demanding Big Data scientists because they suddenly have lots of data and see the term Data Scientist in the news. But what they really want is not Data Scientists, it's business insights and implications from Big Data. The customer needs 1/4" holes, but we're all arguing over which brand of laser powered diamond drill they should buy.

[thuzarsky]

Nailed it.

[btilly]

I still remember my first business presentation. I had a slide talking about how I did a statistics study. I was told to take the word "study" out because it had bad connotations for the target audience (middle managers at Bristol-Myers Squibb if you're curious).

The comment was probably right. But I was horrified.

[fauigerzigerk]

I agree, and it's all the more true if you consider that "presenting" data may actually be more like creating an interactive environment to explore data.

I believe that data analysis yields the best results when perusing the data and tuning the models are closely connected tasks.

[timr]

"I think that pretty much all quantitative PhD's are going to be close to "data scientists"."

Exactly. Fundamentally, "data science" is known far more widely by its other name: science. Yet we've reached the bizarro-world place where there are huge numbers of un- or under-employed scientists looking for work, while companies are freaking out about hiring the "rare" computer programmer who happens to know some statistics and self-identifies as a "data scientist". It's rather absurd.

[tgflynn]

Anyone who can earn a PhD can learn to program but being good at engineering the very complex processes that are needed for effective machine learning applications is a skill that is not so easily acquired.

I've worked with quite a number of quantitative PhD level people in my career and most often the quality of their code leaves much to be desired.

[chimeracoder]

> I think that pretty much all quantitative PhD's are going to be close to "data scientists"

Having taken all but one of the core requirements for a masters' degree in statistics at a university with a well-respected statistics department, I can tell you that's very much not true.

The true challenges in data science have almost nothing to do with what you spend 90% of your time as a graduate student studying (whether you're getting an MA or a PhD, this applies the same). You may happen to end up a qualified data scientist, but that's not by design of the program.

The big problems in data science are almost a disjoint set from the big problems in statistics (at least the solved ones), and that's because the things that are tractable from a theoretical/mathematical perspective are very different from the ones that we hope to solve in the workforce. We're just starting to bridge this gap in recent years (particularly with the advent of computers), but that's a very, very nascent trend.

This isn't unique to my university, either - most schools just simply aren't teaching the type of skills that a data scientist - not a statistician, but a data scientist - would need to be competitive in the work force. Those that do know these skills mostly do by chance - either because they branched into statistics from another discipline, because they were forced to learn it on the job, or because they took the time to learn it themselves.

All three of those are pretty rare - I recently took a class in applied data mining and Bayesian statistics. Except for a few undergraduates majoring in comp sci, the class was mostly graduate students in statistics, and those who knew how to program were in the stark minority (and were very popular when we were picking project groups!)

> all that's left is to train them to program

And to turn everything that they've learned and studied for the past two, four, or more years on its head so that they can actually put it to use. Okay, not everything, but at least 80% of it. Seriously, studying statistics at a high level is incredibly valuable, but it's not sufficient - it's not even going to get you half of the way there.

[tel]

I'm in the same boat and one of the funnier professors in math stat loves to talk about the students he hasn't "ruined" because they manage to learn programming and practical finite sample wisdom and go on to be successful in the industry.

And then he talks about his other students, with great love, who just like proving theorems.

[disgruntledphd2]

Again, I completely see where you are coming from. However, the difference between a Masters and a PhD are huge, far bigger than the gaps between any other form of education.

In a PhD, essentially everything you learn is to master a particular topic, or solve some kind of problem. This can often involve programming (it did for me) and almost certainly involves statistics (again, it did for me). The most important characteristic of a PhD is that you learn all this yourself (I certainly did). For instance, I was the only person in my department to learn R (although there were some oldtime Fortran and C programmers in my department), and then I ended up learning some python and java along with bash to deal with data manipulation problems and administering psychological measures of the internet. These are the kinds of skills that lead into me possessing some of the skills needed to be a data scientist, and with some experience in the private sector, I'll get there.

Bear in mind that I (almost) have a Psychology PhD, and this would all have been far easier for me if I had worked in physics, chemistry or any of the harder sciences. So from my perspective, I can see that this is where the data scientists of the future are going to come from.

Note that I looked up the job market, and made a conscious decision to train myself in these kinds of skills throughout my PhD, but if you are not capable of performing this kind of analysis that you probably shouldn't be doing a PhD anyway.

I really don't see how programming upends all that grad students learn (though I would be delighted to hear your thoughts), as to me it just seemed like the application of logic with the aid of computers. I'm not that good a programmer though, certainly not outside the application of stats, but within the next few years I will be.

[chimeracoder]

> In a PhD, essentially everything you learn is to master a particular topic, or solve some kind of problem. This can often involve programming (it did for me) and almost certainly involves statistics (again, it did for me).

Yes, but the original point was that more or less any quantitative PhD would be expected to have these skills.

> For instance, I was the only person in my department to learn R

Case in point - and I can tell you from knowing the PhD students that I if I found someone who knew how to program in any language not used primarily for statistical computation (R, Stata, Matlab, SAS etc.), I would consider them the exception, not the norm.

The exact opposite is true about a data scientist.

> but if you are not capable of performing this kind of analysis that you probably shouldn't be doing a PhD anyway.

Or you just don't care about those types of jobs - and apparently there are plenty of those, because many, if not the majority, of PhD students I can think of aren't looking for data science jobs.

> I really don't see how programming upends all that grad students learn (though I would be delighted to hear your thoughts), as to me it just seemed like the application of logic with the aid of computers.

It's not programming per se, but the computational power that it brings makes certain techniques feasible, and other concepts and methods aren't obsolete - just no longer optimal. This is really a comment about statistics specifically. Most job postings for data science positions mention some form of the phrase 'machine learning' - and if they don't, they often have that in mind. Unfortunately, while demand for machine learning dominates the job market, in the grand scheme of things, it's just one branch in the field of statistics, and its 'parent' branch was relatively obscure until very recently. To this day, if a PhD student finished their program having next to none of the required academic background for machine learning, I doubt most academics would bat an eyelid. It's just not considered important from an academic standpoint. It's unfortunate that we have such a disconnect between academic interest and industry demand, but it's very much the case.

A basic example that I often cite about how computational power has fundamentally changed statistics from how it was for the previous few decades is in our selection of estimators. (I often cite this because anybody who's ever taken a statistics class probably had this experience). In every introductory statistics class (and for many non-intro classes as well), when studying inference, you spend 90% of your time talking about estimators for which the first moment has an expectation of zero, and the 10% is a 'last resort' when no 'better' estimator exists. Who decided that the first moment was the most important? What about the second? Third?

Well, it turns out that the first moment is easier to calculate, and, by coincidence, it happens to be the most relevant when your dataset is small (say, between 30 and 100). But once you're talking about datasets with observations which number in the thousands (which is still 'small' by some standards today!), you'd be insane to throw out any estimator that converges at a linear rate (rather than at the rate of \sqrt{n}) just because it introduces a small bias.

But we do - and that's reflected in the the sheer amount of academic research and literature that discusses the former, and the sheer lack of that reflects the latter. In many cases, the theory exists, but it was developed in an era in which it could never feasibly be applied.

Vestiges of this era are visible even in many statistical software packages - for another basic example, regressions by default assume homoskedasticity in errors, even though this is almost never valid in real life. Why? Because in a previous era, everyone imposed this assumption, because while the theory behind the alternative had been developed, it was expensive to carry out in practice (it involves several extra matrix multiplications).

I'm painting with a broad brush, but the general picture still very much holds.

[disgruntledphd2]

I completely see your point on estimators and the nature of many if not most statistics courses. I suppose that I was lucky enough to study non-parametric statistics in my first year of undergrad, as there's a subset within psychology that's very suspicious of all the assumptions required for the traditional estimators.

That being said, I think you're missing my major point which is that a PhD should be a journey of independent intellectual activity, so the courses one takes should be of little relevance, and so can therefore be downweighted in considering what PhD students actually learn. I accept that this is an idealistic viewpoint (FWIW, the best thing that ever happened to my PhD in this context was my supervisor taking maternity leave twice during my studies, which forced me to go to the wider world for more information about statistics).

I accept your point about machine learning not being a major focus of academia (well except for machine learning researchers), and I think its awful. Its very sad that the better methods for dealing with large scale data and complex relationships are only used by private companies. Its not that surprising, but it is sad.

That being said, I firmly believe that any halfway skilled quantitative PhD can understand machine learning, most of which is based on older statistical methods. It may not be taught (yet), but its not that much of a mindblowing experience. I do remember that when I first heard about cross-validation I got shivers down my spine, but that may just be a sad reflection on my interests rather than a more general point.

[randomStuff]

This is a total ploy by large companies to increase the H1-B visa cap. I have seen companies post job openings with starting salaries of 40K for experienced developer positions so they can then claim there were no American applicants.

Its a self fulfilling prophecy, if companies outsource the jobs then people don't study those skills for fear of their job going to India; then the companies complain that there aren't enough Americans with CS degrees so they need more H1-B visas.

The US needs tariffs in the IT industry to save our skill base so we can be competitive long term.

[yummyfajitas]

Protecting the industry with trade barriers definitely sounds like the way to make it competitive.

If we do this our IT industry might one day lead the world, like our sugar industry, automotive manufacturing and softwood lumber do.

[v0cab]

Or: Like the South Korean electronics industry does.

[gaius]

One other thing is that the traditional entry route to software for the non-traditional candidate was via the helpdesk or QA department. You got your foot in the door, impressed the established engineers by turning around tickets quickly or by writing comprehensive bug reports, then when they needed another developer, you got the tap on the shoulder. Some of the best engineers I've worked with have come in through this route, including the best manager I've worked for.

If these kinds of positions are outsourced, how do we tap into that seam of talent anymore?

[read_wharf]

If dev is outsourced, then they'll be right there in the building when they need to tap a QA shoulder.

[darkmethod]

I've seen this method of advancement with those around me at where I work. However, I've had the exact opposite experience that you have had. Although, I'm willing to admit to selection bias regarding the sample of candidates.

[jimmytucson]

That's a great point. In theory, having a shortage of X at price Y is nonsensical. It doesn't make any sense to say there's a shortage of worker drones with a mastery of multiple advanced disciplines. There are "enough" of them, they're just called consultants and they make like a thousand dollars an hour. Or if there aren't enough of them, offer a salary that will enable them to at least pay off the student loans they'll accumulate getting a Ph.D and a couple masters degrees on top.

...in theory.

[mturmon]

Good take on the article. I.e., there's a sense that money is to be made through "big data" if only we could find the right, highly skilled person.

Now, we notice that we're not making those gobs of money, as promised in the business press (and shown through a few example successes), and conclude that there must be a shortage of such people.