Gigatoken: Fastest Tokenizer

Y	Hacker News new \| ask \| show \| jobs

	Gigatoken: Fastest Tokenizer (twitter.com)
	26 points by convexstrictly 11 days ago

3 comments

convexstrictly 11 days ago

GitHub: https://github.com/marcelroed/gigatoken

link

Alifatisk 10 days ago

They even added a section ”AI Use Disclosure”, beautiful!

link

benj111 10 days ago

I kind of assumed the model would process the text 'directly', from what I understand, wouldn't this be biasing the input based on how you tokenise as it's lossy?

I assume this tradeoff is purely for speed/compression. Or am I missing what's going on here?

link

convexstrictly 10 days ago

Tokenization is done on the CPU. Models never see the raw characters. That's why you get trick questions like the number of r's in strawberry.

There are many research papers on models using characters directly. One challenge is that effective context length is smaller.

link

ramon156 10 days ago

Doesn't really explain why its faster. "Optimizing for every kind of CPU" is not really enough info.

Cool project nonetheless, I will go through the code later tomorrow

link