You could adjust tokens towards what's more statistically probable, and therefore more compressible (in your example, it'd be picking tomato instead of cheese)
I could see that as a plot point in a science fiction story: Intergalactic telegrams are prohibitively expensive, so before sending one you're offered various variants of your text that amount to the same thing but save data due to using more generic (per zeitgeist) language :)
Compare also with commercial code [1], a close historical analog, albeit with handcrafted, as opposed to ML-derived, compression tables. (There was a single code point for "twins, born alive and well, one boy and one girl", for example! [2])
Comes up as a minor plot point in Vernor Vinge's The Blabber (1988):
> “And from your standpoint, Hamid, there’s one big drawback. The mean bandwidth of this thing [an ansible, more or less] is just under six bits per minute.”
> “Huh? Ten seconds to send a single bit?”
> “Yup. Skandr left three protocols at the Lothlrimarre end: ASCII, a Hamming map to a subset of English, and an AI scheme that guesses what you’d say if you used more bits. The first is Skandr’s idea of a joke, and I wouldn’t trust the third more than wishful thinking.”
A bit of an aside- in one of the sequels to A Fire Upon the Deep, somebody has to interpret some very lossy audio and video into the most likely explanation, but they are stuck with a stupider than usual AI and it misinterprets the results (it's implied if they had their full AI it would have gotten the interpretation correct even with the ambiguity). This episode in the book completely changed how I think about imputation under uncertainty. Often, I don't want a single high confidence prediction, I want a probability distribution of the most likely predictions, rank ordered.
Fire has evocations, which are videos compressed down to something like just a description, then rendered at the receiving end in a way that hopefully has some resemblance to the original.
One viewer stumbles onto a key insight about the struggle taking place, but they only have evocations so they’re not sure. And they sound like a total kook so everyone ignores them.
I can't find an exact reference and I don't want to spoil too much, I think this was in Children of the Sky, at some point one of the wolf creatures is imprisoned and somebody uses the ship's reduced AI to spy on them.
Slight correction: the person with the spy system didn't believe its reports in the end -- the doubt/dismissal problem came before sharing with others, as I remember it. Agreed this was in Children of the Sky.
(aFutD also had a case of high-stakes suspicion of highly compressed messages during Zone turbulence, as I think the GP meant.)
I shudder to think how current LLMs would go with this. I guess we can currently do this easily for still images and audio. Kind of reminds me of Translation Party.
Yep. For lossy what could work even better is an encoder-decoder model, so that it is possible to just save the embedding, and later the embedding will be turned back into the meaning.
I've tried to build sort of model several times, but could never get it to work. The challenge is that small perturbations in encoder space lead to removing semantically important details (e.g. dates). You really want these to mess up syntax instead to get something more analogous to a lossy video encoder.
I built a lossy text compressor in the days before LLMs.
I used a word embedding to convert the text to a space where similar tokens had similar semantic meaning, then I modified an ordinary LZ encoder to choose cheaper tokens if they were 'close enough' according to some tunable loss parameter.
It "worked", but was better at producing amusing outputs than any other purpose. Perhaps you wouldn't have considered that working!
In terms of a modern implementation using an LLM, I would think that I could improve the retention of details like that by adapting the loss parameter based on the flatness of the model. E.g. for a date the model may be confident that the figures are numbers but pretty uniform among the numbers. Though I bet those details you want to preserve have a lot of the document's actual entropy.
Yep, makes sense... Something like 20 years ago I experimented with encoder/decoder models for lossy images compression and it worked very well, but it's a completely different domain indeed, where there aren't single local concentration of entropy that messes with the whole result.
Compare also with commercial code [1], a close historical analog, albeit with handcrafted, as opposed to ML-derived, compression tables. (There was a single code point for "twins, born alive and well, one boy and one girl", for example! [2])
[1] https://en.wikipedia.org/wiki/Commercial_code_(communication...
[2] https://archive.org/details/unicodeuniversa00unkngoog/