|
|
|
|
|
by donall
566 days ago
|
|
Strongly agree. I have been doing data engineering for 14 years and, in my experience, new grads need a lot of on-the-job training. There are a lot of real-world problems (e.g. consistency problems, scale problems, distribution problems) that benefit from a little bit more than just theoretical knowledge. A lot of data systems are full of problems because they were designed and implemented by inexperienced people. People don't know what they don't know and there are a lot of teams that say things like "oh yeah, this takes 3 days to run because it's pretty big" or "we release code daily but it would be too expensive to re-process past data to fix bugs" or even "what's a schema?". YMMV, though. Some data engineers are writing basic SQL or playing with Azure Data Factory and there isn't too much complexity. Read Designing Data Intensive Applications. If that sort of thing resonates with you then find someone to work with who has experience with those kinds of problems! [Edit: no disrespect to ADF! Just pointing out that the data engineering discipline is broad and different practitioners will have different expectations of complexity] |
|