[fastball]

Tokenization is <0.1% of the inference time for the first token in the same way it is <0.1% for the last.

[marcelroed]

Time to first token refers to the time until the model outputs one token, which includes the time to process the entire prompt (doing prefill). The GPU time per token is much lower when doing prefill, so the significance of tokenization is higher.

[dingdingdang]

Have you done preliminary numbers on replacing tokenizer on, say, llama-server?

[marcelroed]

Added numbers here: https://news.ycombinator.com/item?id=49015014

[marcelroed]

Running the numbers now