[collyw]

So I know SQL, Python, Django, Java (though its been a while), Javascrit, Linux, some cloud computing and a bit of devops. Am I a data engineer? Software engineer, with a lot of database background? What makes a data engineer different from a software engineer?

[njd]

- The challenge for an organization is to recognize that there is a significant difference between the 'data engineer' working on a vertical project and the 'data engineer' responsible for integrating data across the enterprise.

- The project 'data engineer', in today's world, most likely will be a software developer responsible for ETL, etc. The data design will be more or less up to the software developer.

- An enterprise 'data engineer' is more concerned with data that affects the enterprise. This typically involves some sort of data integration. For example, how to integrate relevant data from N projects (e.g. A,B,C .. Z) where each project has its own idea of how to represent similar concepts (e.g. person, user, customer), with different provenance, truth assertions, access rules, data retention periods, granularity of metadata (e.g. at the attribute level vs entity level), etc. The enterprise is interested in questions like 'What did we know and when did we know it?", etc. The enterprise 'data engineer' will probably levy requirements on the project 'data engineer' to meet the enterprise's needs.