[LeetHacks]

It highly depends. I was hired for a small research group that didn't have a product in production. Got hired for programming, was on the table discussing and contributing to research within a couple months without any background in ML.

[listenallyall]

These types of anecdotes make the actual practice of both ML and AI seem rather, well, less than scientific. There is supposed to be Ph.D. level math behind all of this, yet an amateur with admittedly no ML background is part of the team.

In Star Wars, it takes Luke Skywalker years to learn to use a light saber skillfully. Then in The Force Awakens, some ex-Stormtrooper with no training picks up the light saber and within 5 minutes is a pro. Kinda ruined the mystique of Star Wars, just like people jumping into ML with no training ruins the mystique of ML.

[LeetHacks]

I am very against the idea that somebody with a PhD is the only one that can do a certain kind of work. But I am ofcourse biased given that you call me out.

Creative and critical thinking is not exclusive to people with a PhD. The ability to understand ones strengths and weaknesses is not exclusive to a PhD.

I would never attempt to write or publish a paper without help of somebody with stronger mathematical or statistical knowledge. On the other hand they should not write source code for a paper without consulting somebody with a strong background in sw engineering. You complement each other. Power is in recognizing that.

You would be surprised how many software bugs I have found that invalidated entire (draft) papers. A PhD in ML doesn't save you from that.

[mkl]

Is mystique worth having? In fiction, sure, but I think not in R&D. I think a lot of ML and AI isn't especially scientific, but I also think there's a lot of low-hanging fruit in applying it to new areas, and both of those make it easier for an amateur to contribute.

[data_maan]

Let me expound on this, as there's a lot of PhD hate in the comments parallel to mine.

What is unique to a PhD is that you took a very long time to master a small slice of the knowledge pie. The emphasis here is on long time: Most people simply aren't willing to go for years on a low salary and tedious job.

It doesn't mean PhD's are more (or less) creative, top coders and whatnot; it means we took the time to read all the papers, to know all previous solution attempts and who all the big players in the field are ("all" w.r.t. our niche). It also mean we can read papers much faster than other people because that is basically what we do all day.

There you have it! Now don't send me ML job offers, cause I gotta read this next obscure paper to figure out if they are legit. :D Just kidding.

[Werewolf255]

I think your metaphor kinda plays against your point.

Years and years ago in the movie universe, Luke had to painfully learn to use his lightsaber from a mentor who passed down techniques and philosophy to the student.

In the current day of the movies, lightsabers are understood to be powerful, yet temperamental and exotic, weapons. Mildly trained individuals can use them, even if it's to a limited extent (i.e. flick switch; shiny side is the business end; heat bad, ouch, no touch).

To belabor the metaphor, you've also had a tradition of people, from all walks of life, using vibroblades (basic to advanced standard statistical analysis and regression) in order to achieve some level of parity against users of lightsabers.

[rjdagost]

There is PhD level math involved. And yet, ML (deep learning in particular) is much more of an empirical endeavor than many would like to admit. A deep understanding of the underlying mathematics does not necessarily give you a better model. Modern models are so complicated that no one can reason through them. Parameter spaces are non-convex and fully of ugly pathologies that make neat and tidy analysis methods useless.

From one perspective, it is disheartening that a deep understanding of the underlying methods doesn't necessarily win the day. From another, it is quite remarkable that having good implementation skills and a methodical mindset can get you quite far.

[natch]

Fuck mystique.

[fisf]

Much 'applied' ML is based on tweaked existing models, getting them production ready and integrating them into a product. That's inherently a bit of non-trivial engineering work, but as you pointed out, not scientific per-se.

[jokethrowaway]

You're disillusioned if you think a PhD is what makes the difference.

Smart people will be able to contribute even if they don't have a PhD. Some PhD are useless and everyone is wondering how the hell they go through that.

[bitL]

In smaller teams/companies one gets to wear multiple hats. However, the term "Data engineer" was specifically created by/for ML folks to get rid of unpleasant repetitive work that has to be done but nobody looks forward to it.

[natch]

Sorry, no. The term "ML scientist" was specifically created by/for data folks to get rid of unpleasant repetitive work with math equations that has to be done but nobody looks forward to it.

If you've ever crafted a pipeline and tuned it to hum along, then watched it break with new/more/messier data, then figured out creative ways to fix it or replace parts of it with more robust parts, iterating on that and scaling it up, you would know some of the fantastic pleasure of whatever you call it, data engineering.

[bsenftner]

Sounds like what the media entertainment companies call a "pipeline engineer". Hmmm...

[chrstr]

> However, the term "Data engineer" was specifically created by/for ML folks to get rid of unpleasant repetitive work that has to be done but nobody looks forward to it.

This may indeed be how the term "data engineer" is used sometimes, but I have my doubts that it was originally created with this meaning. Not really sure where/when the term "data engineer" was actually created, but ICDE started in 1984 [1] and the Data Engineering Bulletin was renamed in 1987 [2] (from "Database Engineering"). It seems likely that the term "data engineer" has also been used since at least then.

Of course ML did also already exist then, but it's certainly a while before the current "big data" / "deep learning" time. And regarding the topics considered "data engineering" at that time, this is from the foreword of the December 1987 issue of the Data Engineering bulletin:

> The reasons for the recent surge of interest in the area of Databases and Logic go beyond the theoretical foundations that were explored by early work [...] and include the following three motivations:

> a) The projected future demand for Knowledge Management Systems. These will have to combine inference mechanisms from Logic with the efficient and secure management of large sets of information from Database Systems.

Which sounds just as relevant today as it did back then. It also does sound like a rather challenging task, and not exactly like "unpleasant repetitive work". Or at least not any more repetitive than: change some model parameters / retrain model / evaluate results / repeat ;)

[1]: https://ieeexplore.ieee.org/xpl/conhome/1000178/all-proceedi...

[2]: http://sites.computer.org/debull/bull_issues.html

[bitL]

Data engineering jobs named as such started to pop up only in the past few years, coinciding with Map Reduce/Spark availability. I wouldn't be surprised if it was re-introduced by one of the companies developing those systems to distinguish themselves (like Databricks, Cloudera etc.), a sort of a marketing. In the past we had DBAs, now DBA + DevOps + unspecified everything morphed into data engineering.

I used to be a member of SIGMOD and the "data engineering" you mentioned was just an academic term.

[hnhg]

Data engineers exist at organisations without any ML work.

[bitL]

Yes, but they are basically what DBAs were before with the addition of ETL. OP is asking about data engineers in the context of ML.