[jschomay]

Thank you. You have a great suggestion. I didn't do that, but I did consider it and I think it can be very powerful. I had 2 example use cases that having an actual AI felt good for, first validating a new feature based on the spec, and second, finding unexpected bugs (like trying to enter a locked room through the back wall). It didn't do so well on the latter, but did great on the former. Having a million simulated games could probably catch those, but how would you track the reports after? Perhaps using an LLM to read the logs/reports could be a good use. Your set up sounds awesome, nice work.

[purple-leafy]

You’re welcome :) for you I’d recommend try get 10 games running / simulated first, and manually analyse the reports yourself to see if the report data is useful. Try and get the report data into a useful shape, and have it as either a json array or an excel. Then you can feed it into an llm to analyse.

For example for me my reports will basically be data points per AI archetype - like how often they collide with a wall, how often they perform certain actions, how often they get blocked or go idle. Straight numbers or booleans. This plus an ELO type system to rate the AI against one another so I can have an AI tier list. Then I can get an LLM to ingest the data and pick out issues / outliers etc.

My game is kinda like chess so this all makes sense for my game.

And thanks for the insights I will try a similar llm setup for manually playing my game, it’s definitely possible and it’s inspiring from your blog