[0cf8612b2e1e]

I saw someone’s excellent idea that if you have a slow system like this, you should communicate by email. It is no longer meant for realtime iteration, but more pointed questions for which there is more effort and time expected on both parties.

[rwz]

0.5t/s is still too slow even for email. For a moderately large inquiry (1MTok output, let's ignore the 4k context window limitation for now) it'll take the model around 23 days or uninterrupted execution to answer a single email.

Real world inquiries are gonna be much slower of course, but this setup is still too slow to do anything meaningfully useful I think.

[springtimesun]

I’m sure it’s possible, but I really struggle to think of an example that would result in a 1m token output.

[8note]

it would be 1M tokens worth of work, with some small report for the end

[0cf8612b2e1e]

Sure, you cannot do 1M, but there are plenty of useful questions you could ask that are far more modest. Simple Q+A, look at this function, how would you design X? All of those could have few paragraphs of outputs that would finish within a day.