[montebicyclelo]

Very cool! As a further variation on this idea, I'm imagining training a reinforcement learning agent on atari games / super mario, but with an additional music-based reward/input, to try to get a "musical" looking playthrough... (Not sure how good it would look / whether it would be worth it though...)

[dberst]

I'm a novice at machine learning, but Open AI made a python library for reinforcement learning in video games, and a fork of it is still actively maintained [1]. It's been a few years, but I remember being able to get it up and running in a day or two (maybe a weekend). It used the Retroarch emulator, which is compatible with a huge number of emulators and consoles.

https://github.com/Farama-Foundation/Gymnasium

There also SethBling's excellent video on YouTube about machine learning specifically with Super Mario World:

https://www.youtube.com/watch?v=qv6UVOQ0F44

I encourage you to give it a try! I feel that video games are a bit underrated by current AI buzz, I think there's lots of potential for a machine to learn a skill through playing a game. And lots of potential for a game being selected or even created with the goal of teaching a specific skill. However at that point maybe it's better to forego the audio and visuals and speak to the machine in text or with pure data.

On the other hand, I have seen a video about convolutional neural networks that feed on each pixel of an image. So perhaps training with sound data, or with the pixels of a spectrogram, could have some positive results. And certainly I would be amused to see a game played in time with music, or possibly even dancing with a songs melody, harmony, and story as well.

Anything that's ever been created by humans, existed first in the imagination of a human brain. And you've got one of those. A mental vision pursued and brought from the mind into physical reality is a beautiful thing, a gift for all of humanity in my eyes. I think it's quite worthwhile. But that's just my perspective. Thank you for sharing your imagination. Have a nice day

[vunderba]

Crypt of the NecroDancer explores this idea of rhythmically timing your character's movement to get bonuses in game.

[Sirizarry]

Don’t forget the legend of Zelda spinoff “cadence of hyrule” which I’m pretty sure was made by the same guys as crypt of the necrodancer.

[jerf]

Bit.Trip Beat is a game based on almost exactly the concept in the linked page: https://youtu.be/LHbg-sNqe4w?t=47 , as well as its later sequel Flux.

[jncfhnb]

Not very good. Mario doesn’t have enough rhythm to evoke a lot of musicality.

The original game’s sound was tied to the frame rate so this vaguely happened by default. Later ports to PAL broke this because it ran at a slower frame rate.