It is meant to be a showcase of what data scientists work on in companies whose businesses do not rely on accuracy of ML models.
For example, data science work at companies like LinkedIn or Facebook can end up requiring concentrated focus on model performance as they have highly developed ML capabilities.
Data scientists at smaller or less data-driven companies end up running ad-hoc queries for marketing or product teams. We are somewhere in-between at Automattic and we find value in sharing our day-to-day work so other data scientists and companies know what to expect when they hire or get hired as someone to do data science work - which is now such a broad definition that it barely means anything.
I also wanted to share our learnings about going from ad-hoc queries, one-off models and solutions to general frameworks, and make a case for custom ML pipelines by showing how this kind of work is so closely coupled with internal data.
It really isn't very magical, though. It is mainly data and software engineering work as mentioned in the post and we are still in the early stages.
I think there is some magic in the applications of ML models to actual business questions which is also a topic that I want to post about on the same blog.
It also wasn't meant as a way to recruit people, as far as I know, we are not currently explicitly looking to hire more data scientists (but also, of course, always welcome applications). I posted this to HN because people ask many questions about what exactly it is that data scientists do in the industry!
It mentions how it's 'impossible' to separate the data points in cartesian coodinates. Isn't logistic regression exactly the use case for this? Thus making the transformation irrelevant?
Anyone with ML experience have an opinion on this?
Please note I wrote 'Logistic Regression' and not 'Linear Regression' (as you seem to think).
Logistic Regression based classification (with quadratic theta parameters) would seem to certainly be able to handle the cartesian case (without having to resort to convert into polar coordinates).
I meant to write "logistic", but it's worth noting that logistic regression is a linear model from which you derive a linear decision boundary.
And yes, it can handle it, by finding a "optimal" boundary according to a criterion other than "is it separated or not?". But that's not the point. The data remains inseparable.
And yes, while logistic regression can technically handle this case (by returning a solution and not blowing up), it will perform poorly unless you transform the data, because the decision boundary is still linear.
