|
This is cool, I had some questions because I was trying to understand this more - not the diffusion model training part, because that I've actually seen before- but the actual way you bundled this into a web application and the choices made... Did you publish the actual trained model anywhere? I see how in the code there is python for how individual samples can be generated, but the model checkpoint pulldown comes from a directory that... I don't see. I then went through the code of how this runs on the web and- I'm not a web dev guy- so I'm pretty confused at all the bits bolted together to make this into a web app. It seems like there is a WASM bit that is compiled from a typical C++ audio plugin that is doing the stuff like conv reverb and limiting and distortion in the web app - all that is oldschool, non-generative AI, DSP being applied to the samples. Then the samples are just... a few default generated samples, to start- where are they pulled from, physically? And you have a login requirement to spool up the actual generative AI part to generate new samples to run into the DSP (because that needs a GPU on the backend to do, so, a login to help rate limit this) How big is the actual generative model? Did you ever think about building the generation engine into the WASM bundle, using maybe WebGPU in the WASM to accelerate in a platform agnostic way, so that the entire app would run offline in someone's browser window? I'm having fun just playing with the kick program without a login, which, again, am I right in saying in that mode there is no gen AI of sampled happening server side, it is just playing starting with some pre-made samples? |
It was a lot of work to get a good DSP to work on the web hahah. Yes, I am writing the DSP in C++ and compiling to WASM. Im using multithreading so the audio work is done in the AudioWorklet while the UI is run in the main thread. I was thinking of writing another article on this because it's pretty interesting and a bit complex.
I sound designed some of the kicks myself and some of them are from sample packs. I just renamed them all to have German names.
If I wanted the model to run completely in the frontend I would need the user to download 300MB of weights and it would probably be tricky making sure it works on everyone's hardware. So I though about this but it didn't seem like the best option. I'm pretty sure it is possible though.
And yes I put rate limiting so no one goes crazy on my credits.
Im glad you're enjoying it!