[gtirloni]

I'm on paid (rich, I know) and the performance is all over the place. Sometimes it'll spit out a whole paragraph almost instantly and other times it's like I'm back to my 2400bps modem.

I haven't noticed prompt size having an impact jut I'll test that.

[ravenstine]

This reflects my experience. Sometimes I'll provide a single sentence (to GPT-4 with the largest context window) and it will slowly type out 3 or so words every 5 seconds, and in other cases I'll give it a massive prompts and it returns data extremely fast. This is also true of smaller context window models. There seems to be no way to predict the performance.

[Kim_Bruning]

Oh hey... leep an eye on your CPU load. The problem might be on the near end. In my case on a slower machine it slows down if you're dealing with a very long chat.

(DO report this as a bug if so)

[gtirloni]

I think that's not the issue here but I do notice the browser going crazy after a while of chatting with ChatGPT. The tab seems to consume a baseline CPU while doing nothing. I just brush it off and close it... bad JavaScript maybe. I should look into this and report as a bug, thanks for the advice.

[entontoent]

This is basically how I respond to requests myself. Sometimes a single short sentence will cause me to slowly spit out a few words. Other times I can respond instantly to paragraphs of technical information with high accuracy and detailed explanations. There seems to be no way to predict my performance.

[wolpoli]

Early on, I noticed that if I ask ChatGPT an unique question that might not have been asked before, it'll split out a response slowly, but repeating the same question would result in a much quicker response.

Is it possible that you have a caching system too so that you are able to respond instantly with paragraphs of technical information to some types of requests that you have seen before?

[gtirloni]

Yes, search for LLM caching and semantic searches. They must be using something like that.

[clbrmbr]

I cannot tell if this comment was made in just or in earnest.

As far as I understand, the earlier GPT generations required a fixed amount of compute per token inferred.

But given the tremendous load on their systems, I wouldn’t be surprised if OpenAI is playing games with running a smaller model when they predict they can get away with it. (Is there evidence for this?)

[darkerside]

I'm guessing there are so many other impacts of own on the model that size of print probably gets lost. I can see a future where people are forecasting updates to ChatGPT like we do with the weather.

[gtirloni]

Yeah. It has so many moving parts that I doubt anyone can make a science out of it, but people will try for sure. Just like with most psycology/social experiments and SEO. I'm flooded with prompt engineering course spam these days.