[asveikau]

> TI-BASIC programs are stored as tokens, not text: every command, function, and variable is a token of 1 or 2 bytes. The OS detokenizes (token→display string) to show a program and tokenizes (keypress/text→token) on entry; the parser walks tokens to execute.

From my memory of using a TI-83 in the late 90s, I would not be surprised if the keypad UI injects tokens directly based on your keypress, rather than "tokenizing the text". I seem to recall, for example, you could not position the cursor in the middle of a BASIC token, and if you managed to type out the tokens it would not work; you needed to find the right menu item to inject the correct token.

[duskwuff]

I can confirm that. On the TI-83, many of the TI-BASIC tokens contained lowercase characters which couldn't be typed at all - you could only type uppercase letters on the keyboard. (There were a few lowercase letters available as tokens for special purposes, but it wasn't a full set.)

Interestingly, you could print tokens in strings - e.g. you could Disp "Disp ".

[suburban_strike]

The 83+ let you type the full set of lowerchase chars as well, but they used 2x as many bytes per character for storage.

[7jjjjjjj]

There's actually a hidden lowercase feature, you can use an assembly program to enable it.

[jamesfinlayson]

Ah makes sense. I remember a younger me trying to open .8xp files back in the day and seeing gibberish, and eventually finding the TI IDE which... felt like it had been written a long time ago (the file select dialog capped the display of file names at 8.3 I think and used ~1 and ~2 etc as "the rest of the file name").

[siraben]

Yes, to type a TI-BASIC program you have to go through the calculator menus which directly insert the tokenized input into the buffer.

The weird thing about TI-BASIC is how seemingly innocent changes in the input can cause huge performance regressions e.g. https://siraben.github.io/ti84p-re/sub-tibasic-for-paren.htm...

  For(I,1,N
  If 0
  1
  End

is much slower than

  For(I,1,N)
  If 0
  1
  End

[asveikau]

The open paren being part of the tokens was always weird. I could imagine that doing strange things for the parser; when it sees a close paren it needs to know that several of the preceding tokens may have an open paren even without having a '(' token.