[vidarh]

It wasn't that it didn't continue, but that this was unique to a branch of languages that largely were sidelined.

And the tokenization didn't prevent you from style differences anyway - as the article points out it e.g. keeps spaces etc. It only tokenized a few things, like keywords and line numbers.

(EDIT: in the late 90's I worked on a project written in Word BASIC.... It was also tokenized and that was used as an opportunity to translate the keywords in the localised versions of Word. But someone had managed to write a bunch of code in the Danish version and somehow exported it as text and imported it into the Norwegian version - the languages are similar enough that it was really hard to tell (no syntax highlighting, and they'd edited a bunch before realising and I had the fun job of untangling it... Yay...)

[BayesianDice]

I think I read about a tool for the Acorn BBC micros which you could apply to your program to remove unnecessary spaces etc. in the tokenised form and shave off a few bytes.

This had the side-effect that you could still display and (presumably with a bit more mental effort, read) the program listing fine, but re-entering a line as shown in that listing would fail because the computer depended on the spaces to do the parsing, even if they were redundant after the tokenisation happened.

[cvcount]

On the ZX Spectrum, numeric values were saved as both text, and in a five-byte floating point format. So making lines shorter often involved using keywords to avoid that: NOT PI, SGN PI, VAL "2" etc.

[kbelder]

Atari BASIC tokenized source a bit more thoroughly... eliminated spaces, parsed constants into their floating point representation, etc. It did enforce formatting to a degree.

I was always surprised that it did all that, but wasn't any faster than Commodore BASIC.