[sillysaurusx]

Cool! How'd you make your video dance to a beat? Is the code available anywhere?

@jonathanfly has been posting some interesting machine learning effects: https://twitter.com/jonathanfly/status/1185843103271444480

I guess one way to make these effects dance to music would be to make a Mel spectrogram of the audio, then somehow use the shapes in the spectrogram to apply deltas to the rendered frames.

[FpUser]

The music animation code worked something like this:

1) Each of my pixel shaders was driven by let's say 32 parameters (do not remember the exact value)

2) The code would generate first set of said parameters and the second set (values were random) and start transitioning ( lerp ) between 2 with the length of transition of about 60 seconds.

3) Upon completion of the transition the first set would be replaced by second set and the second set would be replaced by freshly generated third set and an infinum.

4) Seps 2 and 3 allowed for non stop fluid motion.

5) Lerp value the degree of transition between sets for each parameter would be modulated by sound (one FFT band for each also passed through synth like attack / decay.

6) Finally there was beat detection part which upon detecting a beat would invert lerp direction

There were more steps and various tricks to make it more interesting and non repetitive but I am not writing article here ;)

The end result was quite artistic. The visualizer was part of much bigger enterprise grade media playback / management / delivery /scheduling platform I've developed for hospitality industry