[dmezzetti]

Based on what you're looking to do, it sounds like Retrieval Augmented Generation (RAG) should help. This article has an example on how to do that with txtai: https://neuml.hashnode.dev/build-rag-pipelines-with-txtai

RAG sounds sophisticated but it's actually quite simple. For each question, a database (vector database, keyword, relational etc) is first searched. The top n results are then inserted into a prompt and that is what is run with the LLM.

Before fine-tuning, I'd try that out first. I'm planning to have another example notebook out soon building on this.

[ipsi]

Ah, that's very helpful, thanks! I'll have a dig into this at some point relatively soon.

An example of how I might provide references with page numbers or chapter names would be great (even if this means a more complex text-extraction pipeline). As would examples showing anything I can do to indicate differences that are obvious to me but that an LLM would be unlikely to pick up, such as the previously mentioned in-character vs out-of-character distinction. This is mostly relevant for asking questions about the setting, where in-character information might be suspect ("unreliable narrator"), while out-of-character information is generally fully accurate.

Tangentially, is this something that I could reasonably experiment with without a GPU? While I do have a 4090, it's in my Windows gaming machine, which isn't really set up for AI/LLM/etc development.

[dmezzetti]

Will do, I'll have the new notebooks published within the next couple weeks.

In terms of a no GPU setup, yes it's possible but it will be slow. As long as you're OK with slow response times, then it will eventually come back with answers.

[ipsi]

Thanks, I'd really appreciate it! The blog post you linked earlier was what finally made RAG "click" for me, making it very clear how it works, at least for the relatively simple tasks I want to do.

[dmezzetti]

Glad to hear it. It's really a simple concept.

[brimwats]

Where can we follow up on this when you're done--do you have a blog or social media?

[dmezzetti]

All the links for that are here - https://neuml.com