[doug_durham]

I'd do it like a human would. If a human was reading the legal case they would have a notepad with them where they would note locations and summaries of key arguments, page by page. I'd code the LLM to look for something that looks like a core argument on each page (or other meaningful chunk of text) and then have it give a summary if one occurs. I may need to do some few shot prompting to give it understanding of what to look for. If you are looking for reliable structured output you need to formulate your approach to be more algorithmic and use the LLM for it's ability to work with chunks of text.

[araghuvanshi]

Totally agree there. And that's one of my points: you have to design around this flaw by doing things like what you proposed (or build an ontology like we did, which is also helpful). And the first step in this process is figuring out whether your task falls into a category like the ones I described.

The structured output element is really important too - subject for another post though!