[jhoelzel]

That is in fact the elephant in the room: you will most likely never be able to actually use it for complex stuff because of the token limits required for ai.

Imagine analyzing a massive code base, sure it can tell you how you where solving function ex by translating it to natural language, but it still does not understand any of it.

As far as i know, training it on your dataset will not improve this.

[WithinReason]

Increasing the token limit is a solvable problem

[jhoelzel]

Sure we just need next level super computers for these large models and the patience of multiple days to wait for output

[throwaway3672]

Not necessarily - you just need hierarchical abstraction memory. I reckon my "token" limit when analysing code is around 7.

[WithinReason]

Increasing the token limit without needing more resources to run the network is a solvable problem

[jhoelzel]

but you do sure see the problem with a codebase right?

[WithinReason]

The current token limit comes from a O(N^2) memory requirements for N tokens, there is research that's trying to reduce this towards O(N), for example as the (downvoted) sibling comment suggests. This is not exactly straightforward but not impossible either. It's not a fundamental limitation of language models going forward.

[jhoelzel]

that still will only be enough to hold an extended cli application