[fzliu]

Even with an incredibly long context window (say, 1M tokens), attention still suffers from a problem with long-term dependencies. This is probably why OpenAI hasn't publicly released their 32k token length model just yet.

[Der_Einzige]

I think they haven't released it because the capabilities it has are simply too powerful when combined with a vectorDB.

[sparker72678]

Maybe. It's also probably staggeringly expensive to run.

[nullsense]

Probably all of the above

[ChatGTP]

Curious what you mean by “too powerful”?