[YeGoblynQueenne]

>> Inefficient is a whole lot better than can't even play the game, the story of GOFAI for the last few decades.

See e.g. my link above where GOFAI plays the game (Atari) very well indeed.

Also see Watson winning Jeopardy (a hybrid system, but mainly GOFAI - using frames and Prolog for knowledge extraction, encoding and retrieval).

And Deep Blue beating Kasparov. And MCTS still the SOTA search algo in Go etc.

And EURISCO playing Traveller as above.

And Pluribus playing Poker with expert game-playing knowledge.

And the recent neuro-symbolic DeepMind thingy that solves geometry problems from the maths olympiad.

etc. etc. [Gonna stop editing and adding more as they come to my mind here.]

And that's just playing games. As I say in my comment above planning and scheduling, SAT, constraints, verification, theorem proving- those are still dominated by classical systems and neural nets suck at them. Ask Yan LeCun: "Machine learning sucks". He means it sucks in all the things that classical AI does best and he means he wants to do them with neural nets, and of course he'll fail.

[adw]

> And MCTS still the SOTA search algo in Go etc

It's often forgotten that Rich Sutton said the two things which work are learning (the AlphaGo/Leela Zero policy network) and search (MCTS). (I think the most interesting research in ML is around the circumstances in which large models wind up performing implicit search.)

[YeGoblynQueenne]

Well, gradient optimisation is a form of search.

[famouswaffles]

That was a figure of speech. I didn't literally mean games (not that GOFAI performs better than NNs in those games anyway). I simply went off your own examples - Vision, Image generation, Translation etc.

>As I say in my comment above planning and scheduling, SAT, constraints, verification, theorem proving- those are still dominated by classical systems

You can use NNs for all these things. It wouldn't make a lot of sense because GOFAI would be perfect and the former would be inefficient but you certainly could which is again more than I can say for GOFAI and the domains you listed.

[YeGoblynQueenne]

I don't understand your comment. Clarify.

As it is, your comment seems to tell me that neural nets are good at neural net things and GOFAI is good at GOFAI things, which is obvious, and is what I'm saying: neural nets can make only very limited use of expert knowledge and so suck in all domains where domain knowledge is abundant and abundantly useful, which are the same domains where GOFAI dominates. GOFAI can make very good use of expert knowledge but is traditionally not as good in domains where only tacit knowledge is available, because we don't understand the domain well enough yet, like in anything to do with pattern recognition, which is the same domains where neural nets dominate. If explicit, expert knowledge was available for those domains, then GOFAI would dominate, and neural nets would fall behind, completely contrary to what Sutton thinks.

So, the bitter lesson is only bitter for those who are not interested in what classical AI systems can do best. For those of us who are, the lesson is sweet indeed: we're making progress, algorithmic progress, progress in understanding, scientific progress, and don't need to burn through thousands of credit to train on server farms to do anything of note. That's even a running joke in my team: hey, do you need any server time? Nah, I'll run the experiment on my laptop over lunch. And then beat the RL algo (PPO) that needs three days training on GPUs. To solve mazes badly.

[famouswaffles]

NNs can do the things GOFAI is good at a whole lot better than GOFAI can do the things NNs are good at.

[YeGoblynQueenne]

That's wishful thinking not supported by empirical results.

[YeGoblynQueenne]

Hey, og_kalu, I vouched for your comment but it stays dead. It's not you, it was me who was out of line, with my comment: "wishful thinking"; that's not a very polite thing to say. And my original comment was a bit prissy, too.

To be honest, I'm always a bit jumpy around your comments because I've noticed them all over the place and they're often grayed-out. You kind of tend to go for the jugular. I don't mean that as a good thing. I think others have noticed it too and you get more reaction than you should. That's a shame, because it's clear there's lots of interesting conversations to be had, given you have such strong views and you seem to have done quite a bit of reading; though only on one side of things.

Anyway sorry for starting it this time around and that you got dead'ed, I hope we get to disagree more in the future.

[YeGoblynQueenne]

Addendum:

>> Do you realize how much more data and compute it would take to train a Vanilla RNN to say GPT-3 level performance?

Oh, good point. And what would GPT-3 do with the typical amount of data used to train an LSTM? Rhetorical.