[gargablegar]

I studied using ML just over a decade ago. I actually compared MLPs to SVMs and had a similar thought to this. It does seem like there is a regression on understanding some of the fundamentals and older tools of the trade.

I guess everyone gets focused on the newer things.

Really does seem like people rediscovering older endpoints.

[uoaei]

There's been a huge flood of vanilla software engineers into ML, retconning it as "a subfield of computer science" (computability is a minor concern compared to the statistical underpinnings). They pretend to know the math because they can read the equations, then claim with utmost confidence that actually they're doing all the hard work in ML because they are experts in calling APIs and integrating into products, however useful or useless.

[consilient]

> "a subfield of computer science" (computability is a minor concern compared to the statistical underpinnings)

Computability theory is not all of computer science. It's just one subfield among many.

[uoaei]

I'm referring obliquely to a specific nitpick from select CS folks who argue that because the theoretical optimum is not computable in finite time/memory that the statistical basis for understanding ML is irrelevant.

[noduerme]

Really? Here I thought it was a flood of academics retconning neural net code into a "science" now that programmers had made it run in Python for them fast enough to be useful.

[uoaei]

As hacky as it ends up being in practice, there are some pretty solid theoretical fundamentals to the field of statistical learning.

The problem is the theory is constrained either to the micro-scale (individual layers/"simple" models, etc.) or to the supra-scale (optimization/learning theory, etc.).

Not much concrete can be said about the macro-scale (individual networks) in theoretical terms, only that empirically they seem tend toward the things the supra-scale theory says they should do.

The current controversy in the academia v engineers tussle is 1) what exactly do the empirical results imply and 2) how much does the theory really matter given the practical outcomes. The only thing the two sides broadly agree upon is that some amount of error will always exist because NNs can be broadly understood as lossy compression machines.

[visarga]

There's a similar trend of accusing LLMs of not really understanding, being just pattern machines. Funny that a whole group of people get the same treatment.

[jprete]

I'm not sure what your point is. Many software engineers blatantly pattern-match and copy-paste code to try to get their own stuff to work without understanding what's really going on. This is a long-standing complaint in the industry.

[Q6T46nT668w6i3m]

You’re comparing a subjective claim to an objective claim. Let’s be clear, transformers do not “understand.”

[famouswaffles]

They do in every practical, testable definition of the word.

"Transformers don't understand" is not an objective claim and in fact any attempt to objectify it leads to the opposite assertion.

[uoaei]

You will note the phrasing is tautological in nature and does not apply to all software engineers.

[jjtheblunt]

Retconning? What does that mean?

[chriswarbo]

It's a term from literature/fiction, where later installments try to explain something from earlier installments; usually in a way that's nothing like the original intent. Applying this to real history is more like "revisionism".

[ReactiveJelly]

"RETroactive CONtinuity", in addition to what the sibling comment said.

The Wikipedia top example is Sherlock Holmes dying in a fight with Moriarty and then coming back later when the author relented and decided to write more stories.

https://en.wikipedia.org/wiki/Retroactive_continuity

[jjtheblunt]

(thanks to the sibling comments for explaining...)

[Q6T46nT668w6i3m]

That’s science. You can’t expect everyone to know everything. It’s a preprint so this is the first opportunity to provide feedback.