[Retric]

It doesn’t just polish the input. Tokenizing the output also significantly reduces the risk of gibberish especially if you do a grammar pass to ensure tense matches etc. It means a model with a much worse understanding of the language can preform better than something operating on raw characters.

[famouswaffles]

Fair, I didn't mean to dismiss the impact of tokenization as such.

But tokenization is still a process that's figured by another DL model. Human "insight" doesn't produce tokenization as it does. Another model trained on [insert language(s)] text figures out how best to break sentences into token parts.

That said, these things are a spectrum. I don't think, "no tips from biology whatsoever" or "no constraints at all" is really what Sutton had in mind. The less of it the better is the general idea.

[Retric]

Good point. I find it really reminiscent of how Alpha Zero ignored essentially all human knowledge about chess play, but still depended on insights into chess AI / search algorithms.

I think of deep neural networks as replicating long term memory/reflex rather than thought. I don’t know if that’s quite it, but they excel at a lot of very difficult AI problems when paired with just a tiny bit of handholding. Some of that might go away with even more compute, but I think approaching AGI is going to take more than just even more compute.