Hacker News new | ask | show | jobs
by aiagenta2z 1 day ago
Hi I just read your methodology and I have a quick question about how the handbook are parsed and feed into the context window? The article mentioned that each handbook contains roughly 8K to 79K tokens of extracted text, and did the harness system use grep or search tools to find relevant chunks and feeds to the context, or did it just feeds all the pdf output to the model? There might by distribution bias between real world tasks e.g. Agents grep keywords from docs and only use the relevant chunks. So all the models of the overall pass@1 is relatively low compared to real world scenarios, that might not be the same precision that user experience when they actually handle the daily task? How did the benchmark bridge the gap?