[michaelt]

Depends on your application.

For example, if you're a game company and you want to use LLMs so your players can converse with nonplayer characters in natural language, replacing a multiple-choice conversation tree - you'd want that to be low latency, and you'd want it to be cheap.

[beepbooptheory]

But are people really going to do this? The cost here seems prohibitive unless you're doing a subscription type game (and even then I'm not sure). And the kinds of games that benefit from open ended dialogue attract players who just want to pay an upfront cost and have an adventure.

(All the sudden having nightmares of getting billed for the conversations I have in the single player game I happen to be enjoying...)

If there is a future with this idea, its gotta be just shipping the LLM with game right?

[Const-me]

> If there is a future with this idea, its gotta be just shipping the LLM with game right?

That might be a nice application for this library of mine: https://github.com/Const-me/Cgml/

That’s an open source Mistral ML model implementation which runs on GPUs (all of them, not just nVidia), takes 4.5GB on disk, uses under 6GB of VRAM, and optimized for interactive single-user use case. Probably fast enough for that application.

You wouldn’t want in-game dialogues with the original model though. Game developers would need to finetune, retrain and/or do something else with these weights and/or my implementation.

[sdwr]

It would probably look like an InfiniteCraft-style model, where conversation possibilities are saved, and new dialogue is computed as needed.

Small, bounded conversations, with problematic lines trimmed over time, striking a balance between possibility and self-contradiction.

I could see it working really well in a Mass Effect-type game.

[michaelt]

I understand there are games using LLMs for NPC dialog, yes [1]

> If there is a future with this idea, its gotta be just shipping the LLM with game right?

Depends how high you can let your GPU requirements get :)

[1] https://www.youtube.com/watch?v=Kw51fkRiKZU

[beepbooptheory]

FWIW to confused others, trying to extract something from that video, it looks like this game [1] is using this stuff. Based solely on the reviews and the game play videos (while definitely acknowledging its technically in development status), it kinda looks like long term profitability is the least of their concerns here...

EDIT: Watching the videos, I am more and more confused by why this is even desirable. The complexity of dialogue in a game, it seems, needs to match the complexity of the more general possibilities and actions you can undertake in the game itself. Without that, it all just feels like you are in a little chatbot sandbox within the game, even if the dialogue is perfectly "in character." It all seems to feel less immersive with the LLMs.

1. https://store.steampowered.com/app/2240920/Vaudeville/

[razodactyl]

Absolutely on the mark with this comment. LLMs aren't magical end-goal technology. We have a while to go it seems before they've settled into all the use-cases and we've established what does and doesn't work.