In the final stages of the project, AI was used to assist:
Implementing the user-facing API
Widening of compatibility, for instance generalizing and porting the pretokenizer implementations to support more tokenizers, less interesting features like padding/truncation/unicode normalization
Porting SIMD strategies between AVX512/AVX2/NEON
Final profiling stages and the last ~4x worth of performance from eliminating branching and improving the pretoken cache hierarchy
Refactoring and code reuse
In the final stages of the project, AI was used to assist:
Implementing the user-facing API Widening of compatibility, for instance generalizing and porting the pretokenizer implementations to support more tokenizers, less interesting features like padding/truncation/unicode normalization Porting SIMD strategies between AVX512/AVX2/NEON Final profiling stages and the last ~4x worth of performance from eliminating branching and improving the pretoken cache hierarchy Refactoring and code reuse