[dave1010uk]

> Explores what AI cannot

In other words, gradient descent isn't good at combinatorial optimisation. I'm sure the research is better but the hype in the blog post leaves a bad taste.

There must be a version of Rich Sutton’s Bitter Lesson that applies to alternative computing like this, along with all the other exciting specialised hardware we've seen come and go over the years, like expert systems, optical computing, neuromorphic computing, etc.

Something like:

    General purpose commodity silicon with rapidly evolving software generally beats specialised hardware.

Software is just so much faster to iterate and improve than hardware. AI is also improving it too (eg AlphaEvolve).

Specialized hardware may give a single, significant improvement that grabs headlines but in the long term, compounding small improvements win.

[bitwize]

> General purpose commodity silicon with rapidly evolving software generally beats specialised hardware.

All of the Amiga people are sighing right now, as they recall how their beautiful, elegant system synergistically designed with custom chips was outpaced by CPU/memory brute force in the early 90s.

[CamperBob2]

Likewise, the lesson Seymour Cray had to learn a couple of times because it didn't quite sink in the first time: "Never go up against CMOS when your company is on the line."

[thesz]

  > gradient descent isn't good at combinatorial optimisation.

If you convolve your problem with sufficiently wide Gaussian, you can use gradient descent. The approach is called Natural Evolution Strategies [1].

[1] https://en.wikipedia.org/wiki/Natural_evolution_strategy#Nat...

It requires O(N^4) evaluations to compute Fisher Information Matrix for N-dimensional parameterization of the problem in original formulation. But there are closed form solutions and more economical representations of covariance matrix (LoRA, hehe).

[geremiiah]

I don't think they are even referring to gradient descent here. I think they are referring to systems like AlphaEvolve where they use LLMs to give an informed/heuristical guess to try to tackle an otherwise insurmountable search space.

[anthk]

In hardware Prolog/Kanren/expert systems? That would be possible with libre microcode for Intel, and not this spyware corporate shithole we are living it.

We would be able to switch microcode at boot and set one for security, another one for C performance, others for Lisp performance and so on.

[tmzt]

Sounds a bit like Transmeta Crusoe.

[sixtyj]

“neuromorphic computer that combines quantum-tunnelling physics with a brain-inspired architecture to find solutions to hard mathematical problems”

I have Bruce Sterling’s Ascendaries: The Best of Bruce Sterling” and… the reality is somewhere here in his stories…

Or take Charles Stross and his Accelerando book.

Do you think that teams behind such projects are avid readers and just fulfill the sci-fi stories? :)

[nyeah]

I'm not sure whether these FPGA codes count as specialized hardware.