[Waterluvian]

> an explicit section in harry potter that states all of them at once?

Isn’t part of the point of GPT that it finds relationships without the training data having to be well-structured? So long as the text describes them in a way that a human, having read the books, could answer the question?

I don’t really understand how training works. This isn’t a jab.

[bealuga]

It's a great question!

Right now, I don't train anything, I've broken the text down to n characters and created embeddings for that subset of text -- then I search for the closest distance / relationships between the question asked. Then I add the text to the prompt, and tell gpt to use those paragraphs to answer the question (to ensure that it doesn't make anything up). This is one of the ways I can get around the token limit, but it comes at the cost of thinking it can only use the paragraphs I show it. I'm trying to improve the prompt to get more consistent results, and maybe 4 can help me give it larger bodies of text!

Hope that answers your question, let me know if you have any more!

[Waterluvian]

Oh! I think I understand.

So your software takes a prompt from me, does non-GPT work to find additional context from your source (the books, parsed and re-structured into word vectors or whatnot), and then asks GPT my prompt combined with the added context?

Like,

“What are the three foobars when considering these passages from a book <…> ?”

[bealuga]

Yeah more or less! I still use open ai for their embeddings (translating text into vector space)

- Your question -> vectors with open ai embeddings - Text you uploaded before -> vectors with open ai embeddings

Get the most similar above a certain threshold, and then add it to the prompt saying

"From these articles / paragraphs, answer the user's question: What are the three foobars"

So yep! I preprocess it

[Waterluvian]

I think that’s a really clever idea.

It feels like training is analogous to years of growing up and going school. And what you’ve done is taken that educated “mind” and said, “here’s a document. I’m going to quiz you on it.”

That seems really practical compared to sending an AI back to school to learn all about some specific topic.

I wonder if we will end up with a library of trained models, each of which went to different schools to obtain different focuses of knowledge. Maybe LiteratureGPT is better suited for this than General GPT or ProgrammerGPT.

Okay I think I’ve stretched the analogy far enough.

[just_boost_it]

The GPT models create sequences of words that are likely to be look correct. A side effect is that it sometimes happens to find a connection between ideas. The three unforgivable curses are something that it should be able to connect, as these are all probably in a similar conceptual space, but you would need a fair bit of data to push the model to see that. It's possible to uncover these connections with less data, but with LLMs you need to do things like prompt engineering to give you more leverage on your data if it's limited.

[bealuga]

Yeah, it looks like it actually got the documents correctly, but the prompt itself needed some reworking. I think there's a space out there for prompt testing.