[andai]

I'd like to hear more about your experience using AI for game development.

I've had a strangely asymmetric experience where frontier models are failing apparently basic tasks like making changes to a Pong clone without breaking it, while the same models are successfully designing and implementing multiplayer servers with rollback netcode!

I think it has to do with what they can and cannot verify (i.e. they can't actually play pong to see if they broke it), but I'm not sure.

(Also happy to hear anyone else's experiences on this matter!)

[ksaun]

Overall, I've been very pleased, though I have seen inconsistency like you mention.

First, some background: At the start, I used Sonnet, but after a couple weeks, I'd switched to mostly Opus. I've used Fable when available. For any significant new feature or work, I use /brainstorming (superpowers plug-in). For Vestiges, I haven't written any design documentation per se. I'll start off a /brainstorming with a hefty prompt, maybe 5-15 lines. Throughout the brainstorming steps, I'll think of additional ideas/details about the feature, etc. I'll just add the change/addition to my answer to whatever question it had asked and it seems to handle that fine.

Sometimes I catch things in the subsequent design review process and it handles new guidance well there, too. Typically, I've left some details underspecified and Claude fills in the gaps -- often quite well, but I have to pay attention. The details that it decides later in the process (the implementation steps) are often solid, but when they're not, it's easy to iteratively fix later. Most of the time when Claude asks for clarity on a detail, its recommended option is what I wanted.

It's been rare for Claude to do something that just didn't work (but it has happened occasionally). I have found that it's not good at imagining the player's perspective; it doesn't consider the player's cognitive load for UX considerations, for example. That seems to be consistent with your observation about what they can and cannot verify. A recent instance was when I added a tutorial (which was much more scripted than regular gameplay, providing a more specific experience); I ultimately had to get very directly involved to get to the quality I wanted.

Vestiges doesn't have any gameplay involving timing like Pong does. I could imagine that sort of gameplay being harder for it to get right.

(I hope that covers some of what you were seeking!)

[andai]

>Vestiges doesn't have any gameplay involving timing like Pong does. I could imagine that sort of gameplay being harder for it to get right.

Thanks. Yeah I think that's the main thing. They can look at pictures (although I've had some amusing results with that, too...), so theoretically they could play a game frame by frame if they run it very slowly. That would just very slow (like 10 minutes per 1 second of gameplay) and very expensive.

But basically, they can't "see video" yet, so they're blind in that domain. They can only see static images, not movement.

I expect this to be solved eventually, either with some new architecture (Google is doing a lot of interesting work in this space), or -- and this is kinda funny -- just running them really fast. The labs are experimenting with specialized hardware (e.g. Cerebras) lately, which can do inference at thousands of tokens per second, and several of them have plans to offer it in the next few weeks.

In other words, I think we're going to get around the "time-blindness" issue, in the short term, by just... running them really fast. (Then it would only take 1 minute per second of gameplay, instead of 10...)

Unfortunately, specialized inference hardware massively increases the per-token costs, as well as the tokens per second, so... it's not going to be a practical option for a little while!

--

Re: Superpowers: Thanks for the tip, that sounds very cool.

[ksaun]

Re: superpowers/brainstorming: in an attempt to be more helpful, below are a couple examples of prompts I used. (I've been surprised at how infrequently people talking about genAI share their prompt text.)

I've found it's fine to start /brainstorming in a new session; Claude has been usually good at gathering up enough relevant context to directly dive in. I use high or xhigh for brainstorming; sometimes I ask Claudes opinion regarding level of effort (or whether using Fable would matter). In both cases below, I had more details in mind than I included in the prompt text; I try to provide enough context and direction to drive the discussion. I think I started the first one in a new session, with Opus, in parallel to another Opus whose work sparked my interest in this feature. The second prompt was during a Fable session that I felt had been going very well. It was working on a tutorial and I thought continuing the session with it having that additional context might be beneficial.

**

/brainstorming We're currently fixing up the Disrupted Memory panel, which leads into a Signal Lock match. Through that process, we decided we want to let opponents effectively have "items" like the player can. These would be authored specifically by opponent (i.e., Null has <item1> <item2> etc.). For passive items, the opponent would simply get the effect during the match. For Active items, the opponent would need to have their logic/strategy modified so that they can decide when to use it. I'm imagining the Signal Lock screen/UI shows/represents the opponent's abilities in a way that's symmetrical to what we're doing for the player's. It seems possible that we'll need other/new visual effects for when opponent active abilities are activated, to more clearly indicate what's happening. The design objective here is to help the opponents differ more significantly from each other, adding to the sense of them having personality, increasing difficulty, and influencing the player's own strategy. Please help me delve into the specific items for each opponent and then implement all aspects of this feature.

**

/brainstorming A tangent that's related to the tutorial: We want to make Mara more of a "character" in the game. "Mara" being the tool, or the AI agent within the tool. We use Mara more actively than we have been. For example, instead of just static system log messages, Mara "talks" to the player: when first starting the game, Mara says something like "Hi! It's been 2423 days since I was last activated. You could not be Erik, so I assume you are a new analyst interested in accessing digitized memories. Since you are presumably new here, would you like me to give you an overview of how all of this works?" etc. Going this route would involve a few things: 1) In elevating Mara's presence, we'd probably want to define her voice differently, probably give it a little more personality. 2) We'd want to adjust the tutorial as if it is being delivered by Mara. Both using this new personality and maybe adjusting some other details to make it feel more like what the tool itself would want to explain. 3) We'd want to audit the system log and see what messages it contains would be better presented by a Mara message (either instead of or both). 4) We'd want to revist the Ask Mara content of the Console, maybe adding more questions or modifying some of the answers. 5) We'd reposition Signal Lock's place in the fiction. (E.g., "Restoring disrupted memories is very complex and poorly suited for human consumption. I understand that many humans excel at card games, so I've translated the process into a card game.") 6) A writing pass on all of the text that's supposed to be in Mara's voice to better fit the new voice. 7) Probably we'd want other places/opportunities for Mara to say something to the player. 8) We should re-envision the UI a bit -- maybe we stop the map from reaching all the way to the bottom, and repurpose the screen to the right of the system log for Mara (like the Tutorial does). Maybe we want to allow the player to "talk" with Mara, at least in some situations. Like... Mara might ask the player a question and the player can answer and there's reactivity to these choices later. I see two routes we could take: a) fix up the things in the list above (1-7) and let that help us to see how to better leverage Mara and then revisit 8 and iterate; b) Design out 8 and implement this, then figure out how to cascade the implications of 8 through the rest of the game (i.e., 1-7 + more?).

[arsenide]

I've been having a bunch of fun working on a couple games on a new platform - everything (platform and games) developed with AI.

quackwave.com

Hasn't really been stress tested yet (this is the first time I'm mentioning it publicly), but it's been fun playing with friends & family and iterating.

Party game platform, think "jackbox but no host screen" - just need a web browser open on phone/tablet/PC.

Your comment matches my experience: it has to do what can and cannot be verified. As an example, it was much easier to have AI write a large e2e test using Playwright (then add test cases and expand) than to assume it'll correctly fix bugs without guidelines like this. Also, the human loop is still important in things like screenshot verification - but the frontier models are getting even better here so I'm not sure for how much longer this will be true. The ratio of test code to production code is a bit over 2:1 right now.

[andai]

I have an amusing update to report, I've checked one of the major new features that was added a few days ago. It turns out it was implemented backwards, and in a way which defeats the entire purpose.

Nevertheless, several thousand lines of test code were added, and all tests pass!

I found this particularly amusing because, I've been obsessed with correctness and verification lately, thinking about rewriting the game in Rust, and adding formal verification to the game.

What I realized here is that, this wouldn't have helped at all. The AI would have just written the wrong code in Rust, and then written a mathematical proof that the wrong code is correct...

[arsenide]

Agree. The tests are great! Playtesting reveals that gameplay breaks my assumptions on how the system should work. Where to go from here?

Telling AI to "do TDD" is a shortcut that gets me over 80% there by itself in aggregate with little effort. The last 20% is painful. More painful than the usual "last 20%" in game design and development. I think this is part of the human vs. LLM gap.

If you revert the code to before the feature was implemented, then called out the way not to do it (the failed version you're describing) - what would happen? Maybe that "80% there" turns into "90% there", or maybe it's a bit better. Or maybe a bit worse. Working with LLMs vs. humans, does the gap shrink further if you gave other people the same requirements you gave the LLM? I think it depends on who you're working with - other people have their own opinions and don't necessarily just do exactly what they're told without taking their experiences into account.

In a way I see it like playing a game of telephone. I have my understanding of the system, I translate it into words, the LLM interprets it and does what it does based on that. For some reason, it feels like this is a telephone game with 3 people: A to B to C where the translation layer of my thoughts into words is an inserted "B". In contrast, if instead I'm expressing my thoughts to another person it's A to B. This extra connection here is lossy in the same way a game of telephone is lossy from player A to player C.