[michaelt]

The problem is some of the "gotchas" seem rather important in nontrivial applications.

Imagine a model that isn't sure if 9.11 is greater than 9.9 - which is difficult to reason about, because tokens.

Could such a model coach kids in math? Could it proofread a paper, or sense-check a business plan? Could it summarise a long document about carbon emissions? Could it generate a GUI? Could it spot mistakes in an OCRed document? Spot an off-by-one error or divide-by-zero in computer code?