[svcrunch]

Various frontier LLMs were evaluated on their ability to interpret handwritten proofreading marks in printed literary text, using a small benchmark based on Charles Dickens's "Little Dorrit". Results are modest at best, and surprisingly variable across repeated runs, even on the same pages, underscoring the challenge in building reliable, structured-document systems with current multimodal LLMs.

Curious to hear thoughts from others working on similar problems.

[ggm]

Can't answer this because not in the problem space but my partner is a (retired)professional proofreader and from discussions in the past I would observe interpreting proof marks is not solely about the mark, you scan the line independently (or semi independently) and so they derive meaning contextually as well as from "reading" so this demands both OCR and semantic understanding, as well as syntax.

I would be interested if LLM can intuit the corrections needed including at the level of M or N mark which is semantic, or the Oxford comma. Basically can they correct unproofed input to a similar or converging sense as a person would? Can they write the marks as well as read them?