|
My 2 cents on the long context (haven't used Pro mode, but older long context models): - With a statically typed language and a compiler, it's quite easy to automatically assemble a meaningful context with 1-2 nested calls of recursive 'Go To Definition' and including the source from that. You can use various heuristics (either from compile time or runtime). It's quite easy to implement, we've done this for older, non-AI stuff a while ago, for trying to figure out the impact of code changes. If you have a compiler running, I'm pretty sure you could do this in a couple days. This makes the long context not super necessary. - In my experience, long context models can't really use their contexts that well. They were trained to do well on 'needle-in-the-haystack' benchmarks, that is, to retrieve information that might be scattered anywhere in the context, which might be good enough here, but asking complex questions that require the understanding the entire context trips the models up. I tried some fiction writing with long context models, and I often found that they forgot things and messed up cause and effect. Not sure if this applies to current state of the art models, but I bet it does, since sequencing and theory-of-mind (it's established in the story that Alice is the killer, but Bob doesn't know that at that point, models often mess this up and assume he does) are still active research topics, and current models kinda suck at it. For writing fiction, I found that the sliding window of short-context models was much better, with long-context ones often bringing up irrelevant details, and ignoring newer, more relevant ones. Again, not sure how this affects the business of writing firmware code, but limitations do exist. |