[dualityoftapirs]

They probably won't share how they did it, but there's been a lot of research over the past 6 months showing how you don't have to retrain the entire model to add in new sources. I know nothing about this stuff, but my limited understanding from blog posts is it's easier than anyone had thought to add in new data to a pre-existing model.

[skilled]

Do you by any chance have any of these blog posts available for my own reference? If not you maybe someone else does, I don't recall seeing it but it sounds interesting.

[circuit10]

I assume they mean fine tuning?

https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning)

https://platform.openai.com/docs/guides/fine-tuning

[rafaelero]

I think there was a paper from Google showing that if you included 5% of your original dataset together with the new data during the finetuning then catastrophical forgetting didn't occur. Perhaps it's that simple.