[bachmeier]

> 70% complete, 10% indirect and 20% wrong

I haven't had too much problem with information in summaries being wrong, but there have been times the LLM will miss the most important details. Then when you tell it, the response is "Nice catch!" or something like that.

[ModernMech]

The key thing that makes a summary valuable is it retains the most important details while being shorter. Missing those details makes the summary wrong.

[cyanydeez]

normal human reading speed is 35 WPM. Here's what that looks like, assuming 1 word equals 1 token: https://mikeveerman.github.io/tokenspeed/?rate=35.7&mode=cod...

When you say "you haven't had much problem" one can only assume you're _not actually reading the output_. In fact, like most things in modern times, one has to assume you arn't actually reading the output. You're skimming it; you're finding what makes sense and extrapolating that. This is the 70%.

The problem with non-deterministic models is that the output can't be deterministically assessed. You're harboring a delusion that you're getting real good output.

Most likely you're doing the baby extrapolation: you make it do a small, tightly scoped project and it's does 99% right. Just like a baby doubles in size in a year. Extrapolating, that baby will double again; but it doesnt.

Your human compensation limits does not extrapolate to the size and knowledge that's fed into the model and the context it extrapolates.