Hacker News new | ask | show | jobs
by refulgentis 757 days ago
> This is basically free to add, and there's no reason it shouldn't be made part of standard tokenization.

This is muchhhhh different from how tokenization works today. Adding tokens to the vocabulary is free, everything outside that (i.e. string -> tokens) is going to be a major pain in the ass. Doable but annoying and error prone
1 comments
Filligree 757 days ago
Doesn’t seem as complicated as, say, coding a lexer for C. And why shouldn’t tokenisation use lexers or an equivalent?
link
refulgentis 757 days ago
Good old software development. :( Recent case studies:

- llama.cpp wasn't tokenizing properly, and it came to a head with llama3. Essentially every local model before May 2024 is soft-deprecated, new ones have to indicate the proper tokenizer, and that currently only covers a small subset of popular models

- I recently had to review 41 Phi-3 and Llama 3 models, only 3 had the right tokenizer set

Not saying it's impossible, and we definitely should, and I bet it 100% happens, but...*shudders*

link
Filligree 756 days ago
Meanwhile, I just wrote a custom tokeniser for my fan control experiment.

It features such amusements as: - Tokens representing the current time of day and day of week, with half-hour granularity. [14:30][Monday], as the debugger reports. - An entirely separate set of numeric tokens for CPU usage and such, on a logarithmic scale. Also features tokens for digit position, measured from the right. - A hardcoded text tokeniser for executable paths. [/nix/store](..cut..)/bin/executable name. I didn't feel like using the usual approach, so I built a huffman compressor to generate the tokens for arbitrary text, because why not. - Tokens representing program state - "just started", "long-running", etc. - Tokens representing the fact that the following text is from `tail -f ~/.bash_history`. - Start-of-segment tokens for each of the above, and also for GPU and CPU core complex power usage.

It's not that many tokens in total, and the input is structured data, so why not represent it as such? I still had sixty-five thousand tokens for the text tokeniser.

link