Tiktoken-rs: Batteries-included Rust library for working with GPT-4's tokenizer

Y	Hacker News new \| ask \| show \| jobs

	Tiktoken-rs: Batteries-included Rust library for working with GPT-4's tokenizer (github.com)
	2 points by zurawiki 1192 days ago

1 comments

zurawiki 1192 days ago

In case others find this useful. Just updated for GPT-4, tiktoken-rs is a Rust crate you can use to tokenize text sent to and from OpenAI's LLM models

link