[benob]

Difflogic should lead to very efficient hardware implementations of inference for a wide range of problems. But is it going to allow to generate the logical chains that elude current LLMs?

--- edit ---

After reading the paper more thoroughly, I find the way they implement differentiable logic clever. They use continuous relaxations of 16 logic operators, run them in parallel and apply a softmax to select the most useful operator. At inference time, everything is binarized.

[gwern]

The original cited paper https://arxiv.org/abs/2210.08277 "Deep Differentiable Logic Gate Networks" struck me then as very clever, ultra-efficient/small, hardly any inductive bias or prior... But you have to wonder if it's able to scale reasonably well. Differentiating continuous versions of 16 discrete operations in parallel sounds expensive, especially since you presumably need a bunch of them chained in order to approximate a single neural primitive. Even distilling an existing LLM down might be too hard.

[pizza]

It's definitely slept on. I do think it ought to be very powerful given enough compute to throw at it, hopefully. I think short description length algorithms such as simple compression algorithms or instant-ngp could be interesting to play with through that paradigm.