[nico]

Thank you for the explanation, makes it a lot more digestible

Just nit picking a bit:

> Training any model to express some truth table is very hard

What kind of models are you including here? Truth tables can be modeled in regular code very easily and reliably. And I’m sure there are many deterministic models that could do the same. Are you talking about LLMs in particular or a certain category/type of models?

[thesz]

  > What kind of models are you including here?

Logistic regression, non-linear logistic regression, simple neural networks, etc. Not the code as in "bunch of ifs" or decision trees.

The truth tables? Predict zeroth bit in enwik9 from, say, bits of 64 previous bytes. Or multiply two 16 bit numbers and predict bit 16 of the multiplication result.