[iamjk]

I won't be surprised if DE ends up just falling under the "software engineering" umbrella as the jobs grow closer together. With hybrid OLAP/OLTP databases becoming more popular, the skillset delta is definitely smaller than it used to be. Data Engineers are higher leverage assets to an organization than they ever have been before.

[snapetom]

I think it's mostly already there, but your big, enterprise houses were late getting the memo. About 12 years ago, I switched to a DE role/title and held it ever since. I worked in a variety of startups doing DE - moving data from over here to over there, with a variety of tools from orchestration frameworks to homegrown code in a variety of languages.

About six years ago, I walked into a local hospital to interview for a DE role and it was very clear that their definition of DE was different than mine. The whole dept worked in nothing but SQL. I thought I was good with SQL, but they absolutely crucified me on SQL and data architecture theories. I ended up getting kicked over to a software engineering role, doing DE in another capacity, which made more sense for me.

Only now I'm hearing that they're migrating to other tools like dbt and requiring their DEs to learn programming languages.

[buscoquadnary]

Well my understanding is that a Data Engineer is basically just a DevOps engineer but instead of building infra to run applications they build infra to process, sanitize and normalize data.

[tbarrera]

Author here - Of course, data engineering involves building infra and being knowledgeable about DevOps practices, but that’s not the only area data engineers should be familiar with. There are many, many more! In my personal experience, sometimes we end up not using DevOps best practices because we spread too thin. That’s why I believe in specialization within data engineering and the surge of “data reliability engineer” or similar

[Avalaxy]

Imho that is absolutely not doing the role justice. For some people that may hold true, but I would expect a data engineer to know everything about distributed systems, database indexes, how different databases work and why you pick them, partitioning, replication, transactions/locking. These are topics a software engineer is typically familiar with. A DevOps engineer wouldn't be.

[hbarka]

Or to denormalize data, the distinction of which the data engineer would be most familiar with why and how.

[acjohnson55]

I don't think that's true, in general, just because I think software engineers will largely be focused on creating end user experiences and custom systems, whereas data engineers will be focused on analytics, reporting, ML, and business operations.

There's definitely some overlap, where companies are producing fundamentally data driven user experiences, such as user-visible analytics or live recommendations. But that's niche.

[3minus1]

Yeah, maybe this will happen. Where I work (FAANG), I know that DEs get lower compensation than SWEs and SREs.

[artwr]

Funny in my FAANG, it can be the reverse :)