[snowwrestler]

This argument comes up all the time on HN, but I don't think it means anything. It seems to me that the ability to fill an opening by offering more salary can't disprove a talent shortage, because it is always possible to do so.

Thought experiment: If 100 companies had openings for a skill set that only one person could deliver, all 100 companies could eventually fill their openings by sequentially outbidding each other for the services of that one person.

So how would we know if a talent shortage really exists for a certain job? I can think of a couple potential hints: if starting salaries are going up much faster than the national average, or if the unemployment rate for that job is much lower than the national unemployment rate. Either would seem to indicate that, relative to the job market as a whole, there was a greater demand than supply for that particular job.

[trendia]

In this case, though, there are way more than 6,600 people in the US that would be able to get do that data engineering job, including:

1. physicists

2. Wall St. quants

3. game programmers

4. PhD statisticians

So, the problem is not that there aren't 6,600 people in the US that can do it, it's that the companies can't pay or don't want to pay the $200,000 + that would be required to hire them.

[Declanomous]

This comment will sound a bit self-serving, but it supports your point. I have most of the skills necessary to be a data engineer. My degree in biology, but I nearly got a double major in computer science with a minor in math. (I wanted to work in bioinformatics, but it's nearly impossible to make more than a pittance without a PhD) I didn't pursue the double major because I felt taking classes outside of those three fields was more useful to my development.

Instead of working as a data engineer, I'm working at a non-profit doing pretty much everything involving data for them, as well as running their appeals, and doing almost all of the analysis. I'll lead off by saying the biggest downside of working for this particular non-profit is the salary. However, there are a lot of things I like about this job:

1) Location: I want to be located in Chicago. I have 0 interest in moving out to the West Coast. I'm up in the air about working remotely, because I feel like there is a lot of value in working with people in person.

2) The role is very broad. I get to do a lot of exciting things with data, but it is also a marketing and communication role as well. I am included in nearly every strategic discussion, not just those pertaining to data or technology.

3) Work life balance is very good. I am never expected to work more than 40 hours a week. My boss makes sure that everyone is focused on their lives, to the point where he basically kicked me out of the office for a week because I was waffling about taking a vacation. He makes sure that people know they aren't expected to check their email or do work on the VPN during off-hours.

4) The work I do makes a difference. Not in a "I make something people use" difference, but in a "my work has rescued people from being homeless and fed starving kids" difference. My first couple of jobs out of college were totally lacking this aspect, and I didn't realize how much it meant to me until I started working at a place like this.

I've been here a few years now, and so it's approaching the time where I should start looking for a new job if I want to continue to grow, but I'm having trouble visualizing what that would be. From my perspective, the problem with hiring is that job listings really focus on titles rather than roles, even in smaller organizations. I think my best bet of finding an organization matches the first two points, if not all four, is through my network rather than through job postings. So, to your point, the only way I see myself in a narrow-title role like a "data engineer" is if I really need money.

[infinite8s]

> (I wanted to work in bioinformatics, but it's nearly impossible to make more than a pittance without a PhD)

Just wanted to comment on this part - sadly it's difficult to make more than a pittance even with a PhD.

[Declanomous]

Yeah, totally. The difference is a bioinformaticist with a PhD generally gets to choose what they research, whereas a bioinformaticist without a PhD has to work under someone else's grant. Biology actually has the lowest pay of any major for people working in their field with a four-year degree. You are lucky to make more than minimum wage with a four-year degree, especially if your interest is field ecology or something similar.

If you want to talk about a shortage of labor where it would matter, biology as a field is probably hurting way more for talented software engineers than any company that needs a data engineer. There are so many great applications for programming in biology, and unlike other sciences, say physics, researchers don't tend to pick up on any amount of programming skill on their way to their PhD.

I've tried getting involved in bioinformatics on the side, but it's really difficult to keep up with the field if you don't have thousands of dollars to drop on journal subscriptions. It's also really hard to get access to the data researchers use in general (in any field), but it is made even harder when dealing with researchers involving people due to concerns about privacy. I don't think a focus on privacy is a bad thing, but a lot of publicly available data is sanitized to the point where your sample size would need to be in the billions to draw any inferences. You can request access to less general data, but good luck doing that without the support of a research organization.

Anyways, unless you have a martyr complex, there really isn't any reason to go into bioinformatics.

[x0x0]

I happily worked in a wetlab writing stats software to support breast cancer research. I now do better ad targeting. My salary tripled.

[Grishnakh]

>The work I do makes a difference. Not in a "I make something people use" difference,...

I'd just be happy with that. Most of the work I've done professionally hasn't gone anywhere; it's always "we missed the market window" or "upper management decided on a new strategy". I can't point to that many things I got to work on that actually made it into the market and were used by people for long. One place (a semiconductor company) had a successful though buggy product and large customers in place, with the product already deployed into the field, and the software I wrote got used by some customers, but then suddenly the company decided they weren't making a big enough profit margin on this part (even though the profits were guaranteed and extremely low-risk as the customers had the part designed-in), so they simply quit the market and laid off our entire team.

Making something people use would be a step up. Rescuing people and feeding starving kids is a pipe dream, but then again I work on embedded devices, not big data or analytics or anything like that so that's not exactly a position that'd be easy for me to find if I really wanted it.

[huac]

I'm going to guess most of those people couldn't set up and scale a Hadoop cluster. Are they smart enough that they could learn this stuff? Sure! But there's still a skill mismatch here.

[walshemj]

so you find some real full stack devs ie layers 1-7 or you buckle down and do a CCNA or similar is there a CCIE track for big data?

And I would bet across all of the tech workers in the USA there are well more than 6k that could do this.

[huac]

So, obviously a LinkedIn search for exact title of "data engineer" isn't exhaustive. And as I understand your point, there's certainly no agreed upon group of skills/certification that qualify someone to be a data engineer (or data scientist, or software engineer, for that matter).

But the GP was particularly amusing to me because of its assertion that 'smart, quantitative people, regardless of industry, can build data infrastructures for startups.' I guess we could also say, there's little incentive to pay to train them (or for them to pay to train) to become a data engineer.

[walshemj]

Ah the British Disease ie don't want to pay for training and we don't want those uppity engineers getting above themselves :-(

[ISL]

Or, offer the comparably-compensated part-time job that an academic physicist would accept in parallel with continuing to work in academia.

Source: Am physicist who'd love to find sustainable part-time work at market rates.

[__derek__]

This is on-point. For comparison's sake, look at the number of economists, political scientists, and business professors who have side gigs in consulting.

[blahi]

phd statistician can write ETLs and data infrastructure?

[ronald_raygun]

There can be a lot of trickiness in this. I worked on A/B testing framework at one of the big software cos and me and all my team had a masters or phd in math or stats. While 95+% of our job was data infrastructure and ETLs there is another dimension to making it work and be correct from a statistical point of view.

[snovv_crash]

Bullshit. I have the skills for an intermediate-level data engineer, but I find it bland and I'd rather work in computer vision. However, offer me enough and I may reconsider, and I don't think I'm alone in this.

[Declanomous]

I basically wrote the same thing as a reply to your sibling comment. Data engineering would have to pay a lot more than I currently make for it to be an option, and even then I'd probably change fields once I paid off my student loans and saved some money.

[snowwrestler]

You're not alone, but you're just reiterating that it's always possible to fill an opening by running the salary up high enough.

[snovv_crash]

But the pool of people who wouldn't otherwise take the job grows as the salary increases, pulling the people away from careers where they clearly aren't providing as much value to their employers.

Put it this way, the company isn't going to pay the employee more than the value they provide. That is the ceiling on salary. So until that ceiling is reached it is indeed a case of higher bidder takes all, as your thought experiment demonstrated. But once that ceiling is neared the company will make the decision not to bid higher, thus reducing the demand.

Thus, there is no shortage, just a shortage to work at the lower salary of companies with lower ceilings, because they aren't capable of leveraging the employee's talents sufficiently to draw from fields with related skill sets.

[JamesBarney]

Shortage has a specific term in economics, which pretty much only happens because of price controls.

However if people start liking kale, and the price goes up 20% and you start telling people about the massive kale shortage people will think you're being a little histrionic.