Hacker News new | ask | show | jobs
by sanderjd 1068 days ago
The problem is that if you have all the skills of a software engineer, lots of people would pay you more to do different things than they will to do "data engineering", which is often a lot about cobbling different tools together and doing a bunch of custom transformations and validations that are more tedious than creative.
2 comments
iblaine 1068 days ago
Agreed. The job title of Data Engineer gets abused. Some DE's are Analysts, others are Backend SWEs. Their goals are aligned, to create value out of data, and their methods may differ greatly.
link
_bk90 1068 days ago
A data engineer is as much a software engineer as a back-end engineer or front-end engineer and paid equivalently.
link
tm-guimaraes 1068 days ago
Same as “game developer”

But that’s not how markets work.

Also a job title doesn’t really say much. On all of these 4 roles there’s people doing trivial work, as there’s people doing very deep technical work. Same with pushing the envelope on tech.

link
sanderjd 1068 days ago
I would certainly prefer it if that were the case, but it largely is not, in my experience.
link
_bk90 1068 days ago
It seems like your experience doesn't overlap with data engineering, in that case.
link
sanderjd 1068 days ago
Ok. I think this is just going to end up being "no true scotsman". I'll say "in many companies, data engineers are more doing BI data preparation and presentation than building distributed processing infrastructure or whatever and they are compensated more like analysts than general software engineers" and you'll say "well that's not data engineering".

My argument is not an "ought" argument, it's an "is". I agree with you that data engineering "ought" to be, if anything, a more difficult specialized practice of software engineering, because you need all the software engineering skills and then also specialized data skills. But I'm saying that what it often "is", is more like a specialization of BI and analyst roles.

I think the OP is a pretty good demonstration of this. If "all you need is SQL", does that sound more like a specialization of a software engineer skillset or an analyst skillset? I think the latter... And sure you can say, "well the article is wrong, if all you need is SQL, that's not data engineering", but we're just back to "no true scotsman"; I believe it is common to see the role this way.

link