[CalRobert]

Data engineering was software engineering from the very beginning. Then a bunch of business analysts who didn't know anything about writing software got jealous and said that if you knew SQL/DBT you were a data engineer. I've had to explain too many times that yes, indeed, I can set up a CI/CD pipeline or set up kafka or deploy Dagster on ECS, to the point where I think I need to change my title just to not be cheapened.

[kentm]

Yep, I specifically asked my company to make sure my job title was not “data engineer” when working on data infrastructure, because there was a growing trend of using it to mean “can write some sql”.

Likewise, we had to steer HR away from “data engineer” because we got very mixed results with candidates.

[itsoktocry]

Ironic, since "Data Engineers" are probably far more in demand right now than "Software Engineers".

[ozim]

Only in places silly enough to believe software devs/enga cannot write SQL.

[majormajor]

The "write SQL for ETLs all day" job is a risky one right now since LLMs really lower the barrier for dealing with gnarly SQL. So it's still not a bad time to have your resume be as clear as possible that you're the "deals with complex distributed systems" SWE type instead.

[mrugge]

That's more of an analytics engineer role. LLMs lower the barrier to entry, but popular SQL queries are about correctness and flexibility and this often requires deep understanding and ownership of each filter and window function. This lower barrier can quickly can turn into enough rope to hang yourself.

[CyberDildonics]

Why would a software engineer not be able to do both roles?

[omgwtfbyobbq]

They can, but it takes time away from software engineering.

Along the same reason, that's why there are DBAs, dev-ops engineers, etc...

[sdairs]

I think even before dbt turned DE into "just write sql & yaml", there was an appreciable difference in DE vs SE. There was defo some DEs writing a lot of java/scala if they were in Spark heavy co's, but my experience is that DEs were doing a lot more platform engineering (similar to what you suggest), SQL and point-and-click (just because that was the nature of the tooling). I wasn't really seeing many DEs spending a lot of time in an IDE.

But I think whats interesting from the post is looking at SEs adopting data infra into their workflow, as opposed to DEs writing more software.

[majormajor]

"Data Engineering" being considered a different role from "regular" SWE predates DBT by... at least one decade? If not two? Probably folks working with Hadoop vs RDMS DBA jobs.

[snthpy]

In work yes, but as a title? I only started seeing it called that around dbt origin.

[craneca0]

yeah, i've seen large fortune 100 data and analytics orgs where the majority of folks with data engineering titles are uncomfortable with even the basics of git.

[vjvjvjvjghv]

We have these at my company. They refuse to do any infrastructure work so you have to spoon feed the databases to them ready to go. It’s pretty annoying.

[Foobar8568]

Or the basics of SQL...

[omgwtfbyobbq]

Part of the problem is that a BA/BSA who writes Python, SQL, etc... as part of their day to day work will get lumped with those who don't and their salary doesn't reflect their skills and work product.

Obviously the same can apply in any given title, and does with data engineers like you pointed out, but it's not as simple as just title inflation.

[mrugge]

Titles in software engineering have never mattered less than they do today. Energy worrying about titles or jealosy over specific tech ownership is best channeled into focus on customer, on problem to solve and on finding the best way to solve it as a team.

[isaacremuant]

Agreed. Weird distinction to pay less to people who did certain things and you could a high variance between "data engineers". Some who had only done a course and others that had extensive knowledge of software engineering practices were considered the same.

Ridiculous.

[mandeepj]

> Data engineering was software engineering from the very beginning.

There it is! I found the post title was strange. Thanks for setting the record straight so succinctly.