[tkgally]

Are separately trained models necessary for your case? As context windows get longer—Gemini 1.5 Pro now accepts up to two million tokens, and Google has talked of the goal of "infinite" context windows—couldn't a single base model be used with individualized contexts of sensitive data?

[threeseed]

> individualized contexts of sensitive data

My question is whether this capability even exists.

And if it does how robust it is to workarounds.

[tkgally]

That capability probably exists now if you are willing to accept cloud-based models and only moderately-sized contexts. With Claude 3.5 Pro, for example, one can put one’s reference data into a Project and query the model with that data in the context. In my testing, at least, it works quite well. The Projects can be shared among multiple users, too. The context size is only about one-tenth that of Gemini 1.5 Pro, though, and even the latter is probably much too small for most organizational purposes.

Of course, many organizations and regulators would not allow cloud-based models for sensitive data. A possible solution in that case might be multiple instances of an open-weight model hosted locally within the same secure environments as the sensitive data that the individual employees have access to. I don’t know how expensive that would be, whether current open-weight models are powerful enough, or whether context windows for open-weight models can be made big enough to be useful. But at least it suggests a potential path to a solution that doesn’t require training an LLM from scratch for each employee.