[randkyp]

Does this do RAG over the character's chat history too? That's something SillyTavern can also do with extensions, but I figured since your project already uses Llamaindex, this feature can be something that's already baked in from the get-go.

[_akhe]

Yep it can do CoT for ongoing conversations or to get to the bottom of something through back-and-forth. And you nailed it regarding llamaindex, they provide framework options: https://docs.llamaindex.ai/en/latest/examples/chat_engine/ch... (perfect for HN with the Paul Graham example!)

They even dabble in custom personalities with prompt mixins (example: You can chat with a PDF that will respond like Shakespeare), and if this part was more robust I would delegate to it instead of what I created with ragdoll's prompt prefixes. Turns out the hard part is not converting third-person to first-person. For ragdoll, the heavy lifting is more in the configuration and management of different personas, its multi-modality (of models), the Node & React libraries so that developers can use them in realistic applications... where the value llamaindex brings is its incredible indexing capabilities combined with a conversational query engine (why I chose llamaindex over langchain for this). Ragdoll picks up where llamaindex leaves off regarding personas.

I love that SillyTavern says on their GitHub README: "On its own Tavern is useless, as it's just a user interface. You have to have access to an AI system backend that can act as the roleplay character." I want to avoid being a thin wrapper, and instead have that roleplay character aspect be central to what ragdoll does, so that it can be the de facto creative studio for any character-focused creative deliverable: A story, a film, music, games - so that a user can literally create films and music (and more) in this app like some kind of super Photoshop. I think to accomplish that, it cannot simply be a thin wrapper around an open model. It has to bring as much to the table as an ultra fine-tuned model would yet in seconds instead of years, and with the app- and community-level functionality needed (including being a free-to-use creator tool) to get people to actually build things with it.

[_akhe]

For anyone curious llamaindex's "prompt mixins", they're actually dead simple: https://github.com/run-llama/llama_index/blob/8a8324008764a7... - and maybe no longer supported.

I basically reinvented this wheel in ragdoll but made it more dynamic: https://github.com/bennyschmidt/ragdoll/blob/master/src/util...

[loa_in_]

I don't think a tavern is a place one goes to to make movies

[_akhe]

Not yet haha but even as a place to hang out and casually chat, it would be cool if the character occasionally rendered a cutscene to go along with narratives, or you could optionally enable music and sfx like an audiobook. Maybe the most interesting ones you could export (and distribute for others to experience).

Though I bet the transition from AI text chat to rich multimedia will be like silent films to talkies - where some characters just aren't as interesting with a voiceover or depicted in a video. For some types of characters (written storytellers, etc.) the best interactions might always be text-based.

I felt this with the Final Fantasy 7 Remake, though it's clearly improved from the 1997 version, something felt lost in the transition from the old pre-rendered scenes (drawings) and having to read the dialog in your head, to now having high-quality voiceovers in the best 3D scenes. Yet, if you take a Metal Gear Solid or a Madden - the richer the experience the better.

Ideally: You start out just wanting to go to the tavern and chat with a group of characters, but that interaction became so unexpectedly rich and entertaining you want to capture it, so you can watch it again or share it.

[loa_in_]

I see. That's an enchanting vision indeed! And with other human players in the same space it sounds unendingly wonderful.