It most definitely is, but there's a large part of truth in it. You can try and train people to write good code or follow best practices but from what I have seen, you're swimming against strong currents.
What we have here is the: "I don't get it, why don't you read this 500 pages manual and type in those 10 commands to print your document" era of computing. People wondered why people had trouble "just" programming something.
That is not the job to be done. There are huge frictions and broken interfaces between roles: developers, engineers, data scientists, domain experts with everyone expecting the other to "just" do X.
If data scientists just wrote better code. If developers just were more familiar with machine learning. If ML engineers just could be more CICD aware. If domain experts just could get it right. This is not working.
> That is not the job to be done. There are huge frictions and broken interfaces between roles: developers, engineers, data scientists, domain experts with everyone expecting the other to "just" do X.
The only friction I see with data analysis roles is that data analysts typically don't need (or even outright refuse) to do collaborative work where there are multiple people working on the same code and data.
Once we take the collaborative aspect out, and we see data analysts who never had to check out someone else's work to apply minor changes or have someone else apply minor changes to their own code, then we get to the same naive criticism directed at basic organizational practices and tools as we see here.
>The only friction I see with data analysis roles is that data analysts typically don't need (or even outright refuse) to do collaborative work where there are multiple people working on the same code and data.
In your experience, what do you attribute this to?
One way to look at it is that programmers tend to see notebooks as code, and think it makes sense to use the tooling used for code.
Another way to look at it is as a document, and build around that.
Would fiction writers benefit from learning Git ? Sure. Could it be more productive to reduce the activation energy to accomplish the job of tracking changes with another implementation ? I think yes. There are more writers who use Google Docs than there are who use Git.
For example, we added collaborative editing on notebooks[0]. Multiple people can work simultaneously on the same notebook, see each other cursors and changes live. It's a much better experience. We also added multiple checkpoints behind the scenes[1].
I believe that as programmers, we have the opportunity to create better abstractions around workflows to simplify what we take for granted and do as second nature for others.
One issue is cognitive load. People having to remember to do things. We strive to make it automatic. One example is for experiment tracking: sure, they 'just' have to track experiments and add the necessary code, but we do it automatically so it happens without them remembering. Models are detected are saved, parameters and metrics are logged, etc. This considerably reduces the barrier to entry as even a beginner who knows nothing about tracking gets their experiments tracked.
This is what we're doing. Handle complexity on behalf of a user, and reduce the skills required to get things right. Same for deployment. Can they learn Docker? Sure. Why do they have to, though. So we automatically build model images for them, and they can deploy them themselves clicking a button. As I said, these are problems we've had in the past, for example a 'data scientist' having to sollicit the help of à colleague to deploy their model. This was a bottleneck for us.
What we have here is the: "I don't get it, why don't you read this 500 pages manual and type in those 10 commands to print your document" era of computing. People wondered why people had trouble "just" programming something.
That is not the job to be done. There are huge frictions and broken interfaces between roles: developers, engineers, data scientists, domain experts with everyone expecting the other to "just" do X.
If data scientists just wrote better code. If developers just were more familiar with machine learning. If ML engineers just could be more CICD aware. If domain experts just could get it right. This is not working.