[luckydata]

I feel the name "data engineer" is perfectly fine for both positions. It's the job of the job post to clarify the requirements.

I don't see an issue with looking for a "data engineer with deep experience building and maintaining data pipelines based on spark".

What would you call it? "Spark operator"?

[FZ1]

> It's the job of the job post to clarify the requirements.

I couldn't agree more. That's the exact thing I'm saying is a problem here.

When stuff is wrong, and we don't have enough people, and don't have a well-defined job, but need someone to just handle it .. "Data Engineer".

You get there, and everybody has a different idea as to what you should be doing, and what skills you should have.

[wlesieutre]

Is this any different from “software engineer”?

[FZ1]

SE usually at least knows what language they're expecting you to mainly work in. Not always the case for DE.

[wlesieutre]

Ah gotcha, the problem isn't that it's a vague job title, the problem is that the job description for a particular position may not set expectations clearly enough

[FZ1]

No - I'm trying to say that people often don't even know what the DE really needs to do. They know they have back-end problems, and need someone to clean them up.

But often the hiring manager believes it's probably mostly SQL or mostly python, and advertises that. They really don't know - because someone left, and they are trying to replace them.

So you get there, and talk to the hiring manager, and they're like "are you good in SQL?".

Then you talk to someone closer to the problem, and they're like "most of it's in Java - how's your Java?".

Then someone even closer to the position says, "well, most of the work we need done is really in Scala".

It's your job to just take it over, and make it happen. "Fix the data" - using our existing patchwork of tools ...

This isn't always the case - but given the nature of the problem, this happens a hell of a lot more often than with Soft Eng's.

[bfung]

Nope. Just someone that is willing to use tools in addition to build tools to deal with data.

Most who consider themselves software engineers only want to code and build new, not use existing. Mostly a mindset thing.