[highfreq]

So they are using the following GPT-4 prompt:

-- compress the following text in a way that fits in a tweet (ideally) and such that you (GPT-4) can reconstruct the intention of the human who wrote text as close as possible to the original intention. This is for yourself. It does not need to be human readable or understandable. Abuse of language mixing, abbreviations, symbols (unicode and emoji), or any other encodings or internal representations is all permissible, as long as it, if pasted in a new inference cycle, will yield near-identical results as the original text: --

There is no reason to think GPT-4 has any special knowledge about prompts, or how they should be effectively compressed so that it will treat it as equivalent to the original. It does an interesting job of faking it. But they are basically asking GPT-4 for a stylized version of "summarize the following:".

[sdenton4]

With access to the actual model you could try to do some real compression: encode a sequence, then search for the shortest possible text which gives an embedding near the embedding of the original text. This kind of 'optimize the input' is basically how Deep Dream worked...

[dragonwriter]

Yeah, what you'd need is something like the OpenAI embeddings API, but with a model compatible with the GPT model you are using. (Though it might be worth trying it with that API and the model it has—it won’t be perfect, but you don't need perfect to be valuable.)

[wingated]

Obligatory link to my own work - we did exactly this in a recent ACL paper: https://arxiv.org/pdf/2210.03162

We used soft prompts, not emojis.

You do need full access to the model in order to do this, but we found that prompts can be severely compressed an still retain a lot of information.

[staunton]

Indeed, LLM's seem to be much worse at introspection than humans. I wonder what would happen if one used reinforcement learning to train into it the ability to correctly predict and reason about it's capabilities and behavior.

[MacsHeadroom]

Then you would have designed https://github.com/Torantulino/Auto-GPT

(Uses recurrent langchain loops for introspection and learning about itself and its capabilities as they grow + vector databases like Pinecone for long term memory)