[jnordwick]

Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay."

I've worked with very large amounts of data and high performance computing for most of my career; I mostly had finance related jobs in the last decade or so. I have most of the skill you want, including some you don't know you want. However when salary comes up, that is where we start to part ways. If you are really serious about a shortage, you should be really serious about making offers that can be competitive, but I keep seeing the same $150k offers. That isn't a "shortage" kind of offer.

[dizzystar]

Are they looking for someone who must have every box ticked or are they looking for someone with enough qualifications yet needing work so much they are willing to undercut themselves? Are they justifying their salary offer because you tick 90% of the boxes and not 100%?

I've been looking for work in data engineering and databases for 9 months, and while I'm certainly not as qualified and experienced as you are, I consider myself capable. I've definitely passed the take home and whiteboard tests I've been given, etc.

When I read about a "shortage," I wonder if this is more indicative of unicorn searching than anything else.

[SmellTheGlove]

That to me is a classic recruiting problem in technical positions, data engineering included. Unless you have a manager handling it themselves, the person doing the initial screen really is ticking boxes because they may not know any better.

Once a resume gets to me, and I'm only speaking for myself here, I'm looking for the challenges you've faced and the problems you've solved. I actually care very little about what tech you used because odds are we'll have something different, but we'll need to solve problems. If someone is solid in some related technical skillset, can think critically, and communicate the details of what they've tackled in the past, learning our specific tech stack is going to be the easy part.

Let me put it another way - when I look for interns or entry level hires, the number of those that can do more than spell SAS or Teradata approaches zero very quickly. But if they've solved challenges of the magnitude that they'd be expected to solve with us initially, the tech is secondary to process and problem solving. As we look more experienced, I'd still be limiting myself to candidates from a set of "legacy" industries that prefer these sorts of tools if I insisted on checking those boxes at the outset. I'd prefer to teach a really smart person to use the things that they don't know yet if I have it my way.

[walshemj]

Quite I am sure my experience doing MR back in the early 80's for British Telecom would be usefull today - but I suspect that I might struggle to get past the hr screen.

That was when 17 top of the line supermini's Pr!me 750's was a huge cluster (we where the largest non back user in the UK) - probably about the same as a 10-20k core Hadoop setup would be today.

[blihp]

Of course they must. By demonstrating that they cannot find candidates with 100% of the 'required' skills at the price they are willing to pay, the path is cleared to go the route of 'highly skilled' H1B applicants etc. with a small percentage of these skills. It is not, and has never been, about the skills.

[SmellTheGlove]

I think it's definitely true. Functionally, I'm a director of data engineering (with a big company, so my real title is way more generic). Usually in the initial screen, we'll talk general dollars, and my number is always out of range. For my level and the fact that I'm reasonably happy where I live now, the number is 200k + relocation (more for Bay Area, but lets not go there), and I don't think that's unreasonable for a director level who is presumably going to also develop your more junior DEs.

I don't fancy it up too much, either. I build teams that make the data move and land it clean so that your PhDs can do the smaaaht stuff with it. I can stack BI and Analytics on top, but a lot of people can do that starting from clean data - and clean data is what I do. But I do get the impression that we're viewed as janitors and plumbers - who you'd be thrilled to see at 3am when your shit(ter) broke, right?

[sloxy]

"director of data engineering"

This is already generic.

[SmellTheGlove]

Oh boy. My HR title would drive you nuts then. It's just director information management. We also have info analysts and info managers. Our department color is gray.

[stale2002]

Although your statement is technically true, it is basically meaningless.

Yes, you can always always always find somebody to do a job is your a willing to pay 10 million dollars. That means that "shortages" are impossible. It means that you can never have a shortage in any situation, because you can always pay 10 million dollars for a single visit to the doctor.

But this line of logic isn't very useful when talking about "shortages".

If you had to pay a million dollars for a loaf of bread, is there a shortage of bread? IE, billions of people will starve to death by next week, because they can't afford to buy food.

Most people would say "Yes, there is a shortage of bread".

When people talk about shortages, they are obviously talking about a shortage at a certain price point. There is no other definition of the word shortage that makes sense.

A good definition that I use for the term shortage is "If the government could snap its fingers and instantly produce large amounts of X overnight, would the world be a better place"?

If the answer is "Yes, the world would be in a much much better place", then that means there is a shortage of X. If the answer is "No, the world would only be a little better". Then that means that there is NOT a shortage of X.

[whybroke]

There is no analogy to bread or 10miilon dollar salaries.

A company found something it could profit from more if it paid less than current market value. That is all. They are not saying there are no qualified applicants. They are not saying they want 10million dollars.

What is the case is that a business finds a resource (the perfect hire) that they wish to profit from but do not want to pay the market value for it because that would reduce profits. Rather than be satisfied with what would be an erosion of profit (or an admission of an unworkable business model) articles are posted to demand government pressure wages downward.

If you want a bread analogy, it's as if I found a cheap source of bread I can sell elsewhere at a profit but then complain there's a shortage solely because the cheap stuff isn't even cheaper.

[jnordwick]

If they were offering $300k and still couldn't find attract top talent from other industries, then we could have a discussion about a shortage, but the low six figures doesn't show a shortage situation. It shows a market with plenty of headroom for salaries to grow.

[sbov]

Haven't you just done a 180? I mean, I'm pretty sure the world would be in a much much better place if the government could snap its fingers and instantly produce large amounts of almost anything. Therefore there is a shortage of almost everything.

[sksnxjis]

Given the millions of unemployed Americans, it seems this is not true for at least some occupations.

Wal-Mart greeters can be wonderful people and I'm not saying they aren't valuable as humans. But in labor market terms, there is clearly not a shortage of them.

[tuna-piano]

Of course this is more or less always true - there are only shortages or excesses of things when prices don't or can't adjust freely.

If there was 1 gallon of water left on earth, Bill gates would buy that gallon for $50 billion, and everyone else would die of dehydration.

There has always been a shortage of maids willing to do all my house work for $10.

And there is a shortage of data engineers at $x, but there wouldn't be a shortage at $1M/year (because less companies would want one, and more people would be willing to do the work).

[beachstartup]

> If there was 1 gallon of water left on earth, Bill gates would buy that gallon for $50 billion, and everyone else would die of dehydration.

really? who would sell the last gallon of water on earth?

[snovv_crash]

Someone with a bunch of hydrogen, oxygen and a rudimentary knowledge of chemistry.

[branchless]

someone with liquid assets. I'll get my coat...

[spoonie]

Someone who just drank the last 2nd last gallon. ;)

[qaq]

and what would he use the money for :)?

[atombath]

bullets =)

[beachstartup]

all the bullets would have been fired by then.

[wickedlogic]

If they needed water to prime the last pump on earth?

[p4wnc6]

Disney would.

[biztos]

Maybe you should start a Data Science & Engineering consultancy. The same people who would offer $150K to an employee often have bosses who would love to spend $500K for a person-year of (contract) work if it comes with a high probability of success.

[SmellTheGlove]

I have thought about that. Many times. I have a couple of barriers, most of which are temporary:

1. I have student debt from my law degree, and I have a lower risk tolerance until that's paid off. 2. My daughter is 4, it's nice to be around for the early years, and the corporate gig is quite comfortable in terms of hours. 3. I'm in Maine. Most clients would require me to travel, which impacts #2.

I do have a former colleague here that started a data consultancy. I should grab a beer with him and see if we have common ground in the short term. It's not quite starting your own thing, but it might be fun.

[xapata]

Running an individual or small consulting company isn't that hard. Scaling a consulting company, on the other hand, is quite difficult.

[edoceo]

+1 for making family important!

[snowwrestler]

This argument comes up all the time on HN, but I don't think it means anything. It seems to me that the ability to fill an opening by offering more salary can't disprove a talent shortage, because it is always possible to do so.

Thought experiment: If 100 companies had openings for a skill set that only one person could deliver, all 100 companies could eventually fill their openings by sequentially outbidding each other for the services of that one person.

So how would we know if a talent shortage really exists for a certain job? I can think of a couple potential hints: if starting salaries are going up much faster than the national average, or if the unemployment rate for that job is much lower than the national unemployment rate. Either would seem to indicate that, relative to the job market as a whole, there was a greater demand than supply for that particular job.

[trendia]

In this case, though, there are way more than 6,600 people in the US that would be able to get do that data engineering job, including:

1. physicists

2. Wall St. quants

3. game programmers

4. PhD statisticians

So, the problem is not that there aren't 6,600 people in the US that can do it, it's that the companies can't pay or don't want to pay the $200,000 + that would be required to hire them.

[Declanomous]

This comment will sound a bit self-serving, but it supports your point. I have most of the skills necessary to be a data engineer. My degree in biology, but I nearly got a double major in computer science with a minor in math. (I wanted to work in bioinformatics, but it's nearly impossible to make more than a pittance without a PhD) I didn't pursue the double major because I felt taking classes outside of those three fields was more useful to my development.

Instead of working as a data engineer, I'm working at a non-profit doing pretty much everything involving data for them, as well as running their appeals, and doing almost all of the analysis. I'll lead off by saying the biggest downside of working for this particular non-profit is the salary. However, there are a lot of things I like about this job:

1) Location: I want to be located in Chicago. I have 0 interest in moving out to the West Coast. I'm up in the air about working remotely, because I feel like there is a lot of value in working with people in person.

2) The role is very broad. I get to do a lot of exciting things with data, but it is also a marketing and communication role as well. I am included in nearly every strategic discussion, not just those pertaining to data or technology.

3) Work life balance is very good. I am never expected to work more than 40 hours a week. My boss makes sure that everyone is focused on their lives, to the point where he basically kicked me out of the office for a week because I was waffling about taking a vacation. He makes sure that people know they aren't expected to check their email or do work on the VPN during off-hours.

4) The work I do makes a difference. Not in a "I make something people use" difference, but in a "my work has rescued people from being homeless and fed starving kids" difference. My first couple of jobs out of college were totally lacking this aspect, and I didn't realize how much it meant to me until I started working at a place like this.

I've been here a few years now, and so it's approaching the time where I should start looking for a new job if I want to continue to grow, but I'm having trouble visualizing what that would be. From my perspective, the problem with hiring is that job listings really focus on titles rather than roles, even in smaller organizations. I think my best bet of finding an organization matches the first two points, if not all four, is through my network rather than through job postings. So, to your point, the only way I see myself in a narrow-title role like a "data engineer" is if I really need money.

[infinite8s]

> (I wanted to work in bioinformatics, but it's nearly impossible to make more than a pittance without a PhD)

Just wanted to comment on this part - sadly it's difficult to make more than a pittance even with a PhD.

[Declanomous]

Yeah, totally. The difference is a bioinformaticist with a PhD generally gets to choose what they research, whereas a bioinformaticist without a PhD has to work under someone else's grant. Biology actually has the lowest pay of any major for people working in their field with a four-year degree. You are lucky to make more than minimum wage with a four-year degree, especially if your interest is field ecology or something similar.

If you want to talk about a shortage of labor where it would matter, biology as a field is probably hurting way more for talented software engineers than any company that needs a data engineer. There are so many great applications for programming in biology, and unlike other sciences, say physics, researchers don't tend to pick up on any amount of programming skill on their way to their PhD.

I've tried getting involved in bioinformatics on the side, but it's really difficult to keep up with the field if you don't have thousands of dollars to drop on journal subscriptions. It's also really hard to get access to the data researchers use in general (in any field), but it is made even harder when dealing with researchers involving people due to concerns about privacy. I don't think a focus on privacy is a bad thing, but a lot of publicly available data is sanitized to the point where your sample size would need to be in the billions to draw any inferences. You can request access to less general data, but good luck doing that without the support of a research organization.

Anyways, unless you have a martyr complex, there really isn't any reason to go into bioinformatics.

[x0x0]

I happily worked in a wetlab writing stats software to support breast cancer research. I now do better ad targeting. My salary tripled.

[Grishnakh]

>The work I do makes a difference. Not in a "I make something people use" difference,...

I'd just be happy with that. Most of the work I've done professionally hasn't gone anywhere; it's always "we missed the market window" or "upper management decided on a new strategy". I can't point to that many things I got to work on that actually made it into the market and were used by people for long. One place (a semiconductor company) had a successful though buggy product and large customers in place, with the product already deployed into the field, and the software I wrote got used by some customers, but then suddenly the company decided they weren't making a big enough profit margin on this part (even though the profits were guaranteed and extremely low-risk as the customers had the part designed-in), so they simply quit the market and laid off our entire team.

Making something people use would be a step up. Rescuing people and feeding starving kids is a pipe dream, but then again I work on embedded devices, not big data or analytics or anything like that so that's not exactly a position that'd be easy for me to find if I really wanted it.

[huac]

I'm going to guess most of those people couldn't set up and scale a Hadoop cluster. Are they smart enough that they could learn this stuff? Sure! But there's still a skill mismatch here.

[walshemj]

so you find some real full stack devs ie layers 1-7 or you buckle down and do a CCNA or similar is there a CCIE track for big data?

And I would bet across all of the tech workers in the USA there are well more than 6k that could do this.

[huac]

So, obviously a LinkedIn search for exact title of "data engineer" isn't exhaustive. And as I understand your point, there's certainly no agreed upon group of skills/certification that qualify someone to be a data engineer (or data scientist, or software engineer, for that matter).

But the GP was particularly amusing to me because of its assertion that 'smart, quantitative people, regardless of industry, can build data infrastructures for startups.' I guess we could also say, there's little incentive to pay to train them (or for them to pay to train) to become a data engineer.

[walshemj]

Ah the British Disease ie don't want to pay for training and we don't want those uppity engineers getting above themselves :-(

[ISL]

Or, offer the comparably-compensated part-time job that an academic physicist would accept in parallel with continuing to work in academia.

Source: Am physicist who'd love to find sustainable part-time work at market rates.

[__derek__]

This is on-point. For comparison's sake, look at the number of economists, political scientists, and business professors who have side gigs in consulting.

[blahi]

phd statistician can write ETLs and data infrastructure?

[ronald_raygun]

There can be a lot of trickiness in this. I worked on A/B testing framework at one of the big software cos and me and all my team had a masters or phd in math or stats. While 95+% of our job was data infrastructure and ETLs there is another dimension to making it work and be correct from a statistical point of view.

[snovv_crash]

Bullshit. I have the skills for an intermediate-level data engineer, but I find it bland and I'd rather work in computer vision. However, offer me enough and I may reconsider, and I don't think I'm alone in this.

[Declanomous]

I basically wrote the same thing as a reply to your sibling comment. Data engineering would have to pay a lot more than I currently make for it to be an option, and even then I'd probably change fields once I paid off my student loans and saved some money.

[snowwrestler]

You're not alone, but you're just reiterating that it's always possible to fill an opening by running the salary up high enough.

[snovv_crash]

But the pool of people who wouldn't otherwise take the job grows as the salary increases, pulling the people away from careers where they clearly aren't providing as much value to their employers.

Put it this way, the company isn't going to pay the employee more than the value they provide. That is the ceiling on salary. So until that ceiling is reached it is indeed a case of higher bidder takes all, as your thought experiment demonstrated. But once that ceiling is neared the company will make the decision not to bid higher, thus reducing the demand.

Thus, there is no shortage, just a shortage to work at the lower salary of companies with lower ceilings, because they aren't capable of leveraging the employee's talents sufficiently to draw from fields with related skill sets.

[JamesBarney]

Shortage has a specific term in economics, which pretty much only happens because of price controls.

However if people start liking kale, and the price goes up 20% and you start telling people about the massive kale shortage people will think you're being a little histrionic.

[achompas]

Yeah, this is a selling problem. It feels like you're far more likely to gain traction starting a data team than taking an IC-track DE role. It's easier for companies to justify $200k+ for your skillset in that case, even if it takes you away from pure engineering.

Alternatively, you can just join a large tech org. Netflix etc. have no problem paying good DEs north of $200k in total comp.

[tobyjsullivan]

"we're in the middle of a talent shortage [and don't believe in upskilling]."

[ThomPete]

Upskilling is one of the most ineffective costly ways to try and "re-program" workers and it mostly doesn't work because it's not about skills it's about talent.

[linkregister]

Talent that occurs through the genetic/epigenetic process of having attained a Data Science masters degree after earning a Computer Science degree?

I am a believer in inherent talent but Data Engineering is a skill set.

[ThomPete]

Engineering is a talent skill there is a world of difference between teaching someone starting from scratch and then starting someone first having to unlearn what they learned to then learn perhaps a completely new way of thinking.

Most of the reskill programs I have heard of failed miserably exactly because the skill isn't enough.

[crispyambulance]

I sort of agree in that orgs can't simply create massive education programs to re-purpose skill-sets/talent. That might have been possible "back in the day" before project managers were breathing down people's necks, but not today.

But the brightside is that talented people will find a way to "upskill" themselves in whatever environment they find themselves in. It is then up to the candidates to sell themselves and for the potential employers to be flexible about considering different backgrounds and nurturing the development of cross-functional skills that are needed for so-called data-engineers.

The skills listed in the article are all fairly common but its hard to find enough of these skills within individuals. For example, its not hard to find folks who can do the care and feeding of sql-server databases, or skilled programmers, or analysts who understand the business domain intimately. The problem is getting all of these together in one individual in a "know-enough-to-be-dangerous" level.

[ThomPete]

Yeah if up-skilling means "I used to do front end development now I do back end development" the up-skilling is fairly easy to do.

But if you used to work as a plumber and want to up-skill to data analyst (or vice versa) it's not that simple.

[etjossem]

That's not always the case. Talent doesn't exist in a vacuum.

Someone with a natural talent for picking up new development skills will still learn data engineering far faster when provided with proper resources and strong internal mentorship.

I can see how you might make this observation after observing a poorly conducted training program.

[ThomPete]

The problem is that there aren't that many people with natural talents. They exist but it's very hard to sync them up with where demand is.

Also this is not just one poorly conducted training program. Denmark spent billions up-skilling parts of their work force. The results where simply no there. Something like 6 out of every 1000 person or something like that.

[duaneb]

How would you distinguish talent from experience?

[ThomPete]

I would say that talent needs experience to be valuable.

[sdoowpilihp]

This has been my experience with any "senior" engineering / BI / DS role. There is a particularly high level of price sensitivity to anything above 200k.

[x0x0]

In particular, employers whining about lack of X need to ponder raising wages to where employees can afford homes in a city where prices are now within spitting distance of $1k/ft2. When your basic pitch is, "We desperately need [data engineers | machine learning engineers | computer vision engineers | what have you] so desperate to live in CA they'll accept never being able to afford a home unless our lottery tickets pay out", it should be unsurprising they have a hard time finding the talent they claim to need. Or, they could accept remote workers! Even remote workers near sfbay, who just don't want to burn 2.5 hours/day commuting in and out of sf...

[harichinnan]

We all like $400K the investment bankers make. But Finance Industry had developed a business where they could pay their workers $400K and still make a huge profit for their investors. Except for Googles and Facebooks, the average tech startup is not making Finance industry level profits.

Also Finance requires proper education and training. Not so much for App development. So for everyone who complains about getting $150K offers, there are a 100 thousand people right here in US applying for $60K technical analyst jobs.

[__derek__]

> Except for Googles and Facebooks, the average tech startup is not making Finance industry level profits.

And they don't have finance/Google/Facebook level needs for data engineers. They can't reasonably claim to need top-level skills and then beggar out on the cost.

[bobosha]

>Whenever I see these posts I immediate translate them in my head to "we're in the middle of a talent shortage at a price I am willing to pay."

That's true for just about anything.

"there is no epipen crisis, only a crisis at what you are willing to pay"

"There is no poverty , only poverty at a given income level"

*"there is no crime problem, only crime problem at a given crime level"

what you are saying is self-contradictory. If you (or others) are able to turn down 150K offers...you know what you are.

[wavefunction]

You must admit that the price of epipens is an artificially inflated one only possible due to government imposed monopoly, not one driving by true market forces.

Poverty is simply a description wealth and is always comparative. We can define poverty as any level we so desire.

One might argue that any crime is a problem, as long as it causes an issue for society or victims.

[whenwillitstop]

My experiences exactly, pinged by companies obsessively for my big data skills, all trying to pay me less than I am currently making.