|
|
|
|
|
|
by minimaxir
6 days ago
|
|
|
Both the example libraries compared (tokenizers and tiktoken) are Rust-based with Python bindings. There's just a few levers in Rust that can speed it up even more particularly with LLM assistance as the AI Use Discloure here notes: > Final profiling stages and the last ~4x worth of performance from eliminating branching and improving the pretoken cache hierarchy |
|
|