[jerf]

The information theoretic barrier is in the information content of your prompt, not the ability of the AI to expand it.

But then I suppose I should learn from my own experiences and not try to make information theoretic arguments on HN, since it is in that most terrible state where everyone thinks they understand it because they use "bits" all the time, but in fact the average HN denizen knows less than nothing about it because even their definition of "bit" actively misleads them and that's about all they know.

[adastra22]

I have a CS theory background. If by prompt you mean the full context provided, then there isn’t an effective limit. Claude now has 1M token context windows. You are not going to fill that with just a task specification. You could easily fill it in a large repo with the accumulated design history and total codebase. However this is also largely static, and could be used for fine-tuning. With fine tuning you’re back to a 1M token task specification for the unique variation of this prompt, and recent changes. You are not going to easily fill that.