I initially had a Rust-based word cloud generator that generates word clouds in high resolution in ~100 milliseconds whereas it would take other generators a couple seconds to do the same thing. Does the world need a super-fast word cloud generator? No. Do I want a super-fast word cloud generator? Yes.
I later found enough optimizations to reduce the generation speed all the way down to ~16ms. Do I need a word cloud generator that fast? No. But if I have a word cloud generator it's going to be as fast as possible dammit.
1000x improvements unlock qualitatively new capabilities, even when only applied to subcomponents. Moreover, the fact that it's only 0.1% of the runtime is usually [0] an artifact of the entire project being treated that way -- why write this the right way when it won't move the needle on the composite project?
[0] Yes, for LLMs we're closer to the bound than 1000x. Even there though, basic pytorch operations are often 2x slower than simple rewrites, better scheduling algorithms have a history of closer to 5x-10x gains, and all of that supposes you don't have further architectural improvements over time. Moreover, who's to say that legitimately fast tokenization doesn't unlock additional capabilities elsewhere which people have ignored because it was never close to viable?
I don't think this was particularly trivial, but I do think that thanks to AI assisted coding there's more capacity for making improvements that "don't seem worth it" at first or when you look at it as a percentage of total.
But look at e.g. Biome; optimizing the formatter didn't seem worth it for a long time because it only took <1s to format most files. But it's <1s times millions of files, billions of times a day when you add up every developer that used Prettier for their code formatting - it adds up.
And I'm convinced Biome triggered or was part of a bigger effort to convert JS based tools to native code. This saved time and energy, which in turn allows for faster and / or more feedback loops, which in turn allows faster turnaround cycles for software development (bet it human or LLM assisted), etc. It's a compound effect.
I don't know enough about tokenization or whatever to judge this one, but if it's 1000x as fast as it used to be, there will be less need to try and avoid or minimize tokenization which may lead to new applications.
I later found enough optimizations to reduce the generation speed all the way down to ~16ms. Do I need a word cloud generator that fast? No. But if I have a word cloud generator it's going to be as fast as possible dammit.