[morkalork]

IMHO all these approaches are hacks on top of existing systems. The real solution is going to be when foundational models are given a mechanism that makes them capable of storing and retrieving their own internal representation of concepts/ideas.

[mr_toad]

Neural networks already have their own internal knowledge representations. They just aren’t capable of learning new knowledge (without expensive re-training or fine-tuning).

Inference is cheap, training is expensive. It’s a really difficult problem, but one that will probably need to be solved to approach true intelligence.

[morkalork]

In the way that they're trained to complete tasks from users, can they be trained to complete tasks that require usage of a memory storage and retrieval mechanism?

[dotancohen]

Where does fine-tuning sit in this? How easily are existing models able to be fine-tuned for new use cases, such as specifically legal or medical texts?