[visarga]

It's language modelling with search engine in-the-loop.

Instead of training GPT-3 with 178B weights, you train a 25x smaller model and allow it to retrieve useful snippets from a large text index as additional information.

This solves the problem of very large models and the problem of updating an already trained model, as you can swap the text corpus with a newer one. The model learns mostly syntax, burning less trivia in its weights than a regular LM as it can simply copy the relevant information from the index.

This development was bound to happen as large LMs are expensive to use and it was an obvious idea. We've had these semantic search text indices for a few years already[1], they just weren't combined with text generation.

[1] https://github.com/spotify/annoy

[alkonaut]

So the memory doesn't solve the context problem of e.g. "conversation context"? I.e. the storage isn't modified while the model is used? If I make an app that makes conversation using such a model model, then the storage isn't modified to insert knowledge about what the early parts of the conversation was about, and it's only bringing a database of fixed information into the conversation? (I have a friend who is just like that).

[tveita]

You could update your storage as you go, the indexing doesn't appear to be that expensive.

For many tasks it wouldn't be helpful because the input is small enough to be covered by the context already, and for summarizing and question answering tasks, you want it to repeat information from other documents, but not from earlier in its own output.

It might be interesting for a long-context task like "given the first parts of this book, complete the next chapter".

[TulliusCicero]

I know AI Dungeon and Novel AI both factor in several recent text inputs when generating new text, and also have a memory section where you can add things you want the AI to never 'forget' about the current story.

[jamesblonde]

Yes, the key technology here is a scalable embedding store. The leading players here are the indexes - faiss and scann. The open source platforms are opensearch, elasticsearch, featureform, milvius. Then there are saas products like pinecone.