[vagabund]

I've never had these retrieval demos work well. Either the LLM hallucinates content that could plausibly be in the document, or it's `reasoning` is so hamstrung by the constraint that it misunderstands the meaning of the specific quote it's referencing.

I just tried this one on the NBA CBA, which I would think is an ideal use case, and it didn't answer a single question correctly. Hopefully we find a more clever way to use LLMs in conjunction with a knowledge-base.

[freediver]

Can you share the link you used and a few questions/correct answers?

[vagabund]

PDF link: https://cosmic-s3.imgix.net/3c7a0a50-8e11-11e9-875d-3d44e94a...

"What's a mid-level exception?" (Initially stated the info wasn't in the provided context, but upon asking again, it got it correctly)

"How long does a team have to wait after receiving a player in a trade before trading the player again?" (It's 60 days under certain circumstances, it told me 6 months)

"What's a traded player exception?" (Claims isn't in the document)

"How are traded player exceptions created?" (Answers correctly)

So after playing with it a bit more I think I understand the sorts of questions it does and doesn't like to answer, but still, I would think fuzzy matches for section titles should work.