[__mharrison__]

I just spent yesterday applying Kaparthy's autoresearch on an ML problem.

I teach ML for a living and was amazed with what the tokens gave back to me after many rounds of experiments. If Kaggle was still a thing, AI would generally beat it.

The challenge I've seen is that most data science/ml modeling work is quite weak. Folks don't even know the basic tools well. Not sure if giving AI to them will really open up many doors to them.

As always experts love minions of juniors doing their deeds. Non-experts get to wade through slop.

[twelfthnight]

I agree AI could probably do a decent job on Kaggle problems. Of course, almost no DS job is building models with well-defined objectives and perfect data. The DS and MLE folks I work with mostly spend their time reframing ill-posed product requests into ML systems that can be maintained and improved with feedback loops.

A _huge_ part of a DS is saying "No" to bad ideas posed by non-experts. The issue with LLMs is all they ever say is "Yes" and "Wow, that's such a great idea!"

[mscbuck]

Yeah, once you move onto legitimate business evaluation metrics (where Precision@k or Recall@k don't actually fit your business model without modification), GPTs just seem to suffer without context, and hey, knowing the context is part of what gives a data scientist his value.

[jimbokun]

The data scientist is like in house lawyers in that respect.

[whattheheckheck]

Karparthys autoresearch is just automated overfitting no?

[LaserPineapple]

Is Kaggle no longer a thing?