[xg15]

What irks me a bit at the way the Bitter Lesson is interpreted is that seemingly it didn't just throw out handcrafted model/feature generation, but also any attempt to interpret the learned models and features.

Like, in theory, this should be the absolute best time for people interested in analyzing unstructured data: Here there is this wealth of open-weight models, trained on half the internet that must have developed all kinds of absolutely insane feature detectors for all kinds of media: Programming languages, human-language prose, images, audio, video, whatever you want!

In practice, the models are mostly treated as black boxes and the weights as inscrutable. Which is why we now have the weird situation that our models are able to understand incredibly subtle and abstract semantic concepts in text - but the pre- and postprocessing is still on the level of regexes and string heuristics like 50 years ago. There doesn't seem to be any inbetween.