Hacker News new | ask | show | jobs
by ericlewis 1021 days ago
I’m curious, did you actually run it through the tokenizer and see if it was less tokens vs uncompressed? I have seen a lot of people try these “compression” schemes and token usage can be higher.
1 comments

It's definitely less tokens at least in my contrived case. Looking at the compressed text, I can make out what is what, and see that it's just minimizing words to their root parts.

Typescript (22 tokens):

    export type IAssist = { id: string; prompt: string; promptResult: string[]; };
Story (26 tokens):

    IAssist contains: id which is a string; prompt which is a string; promptResult which is an array of strings.
Compressed (13 tokens):

    IAsst{id,prompt,promptR}
And again I'll just call this interesting, because is it really going to know promptResult is a string array in most cases? Definitely not unless it gets some help in the component description, maybe.