[chelm]

ahahah, probably not. Looking at my own handwriting: Neither in writing nor in reading.

I find it interesting how the prompt changes the result.

If you let the model focus on the text, the open source got so good in the last year. That's remarkable. When you change to prompt to not only extract the text but also extract specific information, the pure text extraction result gets worse. For me, it worked to run two prompts on the same document to get both in a meaningful accruacy.

[nullc]

I had good luck with using a process where a supervisory pass looked at reasoning traces and improved the instructions. This had major improvements in performance and quality.

I thought about doing a pass with a non-vision-llm and and after having the vision llm give its result show it the non-vision and ask it if it needed to reconsider, but my test input was too easy to it to matter.

One thing I worried about was that my scans are big mostly empty pages, I thought this might not give enough attention to the receipt text and that I should do some preprocessing to crop the image. But, again, the results were too good without doing it that I don't think I could test if its an improvement or not.

I am also a little worried that reasoning about the content might make it hallucinate stuff, and worse have the hallucinations be credible enough to not get caught later. But the improvement it got from reasoning out genuinely unreadable parts made the risk worth it for my use.