[james-mxtech]

the eval agent you're running 8-10 wide is the part i'd want more detail on. silent chapter drops and dupes were the only extraction failures that ever bit me, and grading those automatically needs a ground-truth toc to diff against, which the epubs that lie about their spine never give you.

[flatline]

I currently parse the TOC for chapter markers, then walk the spine between markers to ensure everything is captured regardless of what the TOC says. I think it is in the spirit of what epub readers do and in line with the spec.

The eval agents perform the extraction and run a comparison of chapter/word count against the epub original. They also parse the extracted text to look for inconsistencies. The quality criteria have been built up iteratively and issues backed out into unit tests. The eval runs aren't part of my standard unit tests but a one-off, heavy lift process that I run periodically against the epub collection I already had and a bunch I built up for this project. I've also just done a ton of manual validation. There may still be edge cases.