[iblaine]

The Data Engineering track seems to attract data analysts and non-engineers. I’m one of them, have worked with many others in my same bucket. What makes Data Engineers excel, I think, is to have all the skills of a software engineer + the ability to think like a data analyst.

SQL is definitely part of that journey, and more skills should be mastered if you’re going to do things like use Kafka, airflow or spark.

[sanderjd]

The problem is that if you have all the skills of a software engineer, lots of people would pay you more to do different things than they will to do "data engineering", which is often a lot about cobbling different tools together and doing a bunch of custom transformations and validations that are more tedious than creative.

[iblaine]

Agreed. The job title of Data Engineer gets abused. Some DE's are Analysts, others are Backend SWEs. Their goals are aligned, to create value out of data, and their methods may differ greatly.

[_bk90]

A data engineer is as much a software engineer as a back-end engineer or front-end engineer and paid equivalently.

[tm-guimaraes]

Same as “game developer”

But that’s not how markets work.

Also a job title doesn’t really say much. On all of these 4 roles there’s people doing trivial work, as there’s people doing very deep technical work. Same with pushing the envelope on tech.

[sanderjd]

I would certainly prefer it if that were the case, but it largely is not, in my experience.

[_bk90]

It seems like your experience doesn't overlap with data engineering, in that case.

[sanderjd]

Ok. I think this is just going to end up being "no true scotsman". I'll say "in many companies, data engineers are more doing BI data preparation and presentation than building distributed processing infrastructure or whatever and they are compensated more like analysts than general software engineers" and you'll say "well that's not data engineering".

My argument is not an "ought" argument, it's an "is". I agree with you that data engineering "ought" to be, if anything, a more difficult specialized practice of software engineering, because you need all the software engineering skills and then also specialized data skills. But I'm saying that what it often "is", is more like a specialization of BI and analyst roles.

I think the OP is a pretty good demonstration of this. If "all you need is SQL", does that sound more like a specialization of a software engineer skillset or an analyst skillset? I think the latter... And sure you can say, "well the article is wrong, if all you need is SQL, that's not data engineering", but we're just back to "no true scotsman"; I believe it is common to see the role this way.

[woleium]

for a moment there I thought you said "what makes data engineers is excel"

[jeremyjh]

do not speaks its name