In humans, sleep is also required fo learning, so it's not fully continuous. An AI that occasionnaly retrains using the new knowledge would still be very interesting.
From what I understand, there are two different processes – a continuous process which runs while we are awake, building new connections between neurons (acquiring new "weights"); and a pruning process which runs while we are sleep, sorting through those new connections and deciding which to keep and which to discard. That's very different from an AI that occasionally retrains.
But what if you could run a new acquired neuron "wight" trough another LLM that would based on its provided context (guidelines) be able to determine the importance of this new neuron.
That loop could create more accurrate and higher quality data that could be used to re-train (fine-tune) new version of LLM.
Not quite the same though - if you teach me how to operate a drill press, I'll be able to do it today, I don't have to spend 8 hours in bed before the knowledge is available to me.