Show HN: Demon – open-source real-time music diffusion engine, 25Hz local GPU

[ryanontheinside]

YO,

I’m Ryan, lead author. I’ve been contributing open source generative audio stuff for a while now, audio reactive Comfy nodes, extended ACEstep support in Comfy, etc.. I just opened-sourced a new audio project that I've been working on for a few months and I want to tell y'all about it.

WHAT IS IS DEMON: Diffusion Engine for Musical Orchestrated Noise

This is StreamDiffusion but with audio instead of images, and ACEStep 1.5 instead of Stable Diffusion. It’s responsive enough that you can play it like an instrument, and remix in near real-time.

I also distilled the ACEStep VAE: it’s faster at the expense of some quality.

I also trained something like 200 lora/dora for ACEStep 1.5 and 1.5XL: I will release these in batches of 5 or 10 or something

WHY IT IS Two reasons: 1) Making music is an inherently real-time activity 2) Why not bro

SOME RUNTIME CAPABILITIES -Real-time remixing of songs -Denoise, structure, timbre strength adjustment -Reference track swapping -Prompt blending, parameter scheduling with curves -LoRA hotswapping, runtime strength adjustment -Latent channel (research preview) -Feedback -Vocal stem cutting/pasting with melformer (s/o u/BuffMcBigHuge) -XL support (its less stable, working out VRAM pressure issues and whatnot) -Lyrics/vocals SOON -Spectral quality SOON -Other stuff

SOME LIMITATIONS -ACEStep (correctly) ‘begins’ and ‘ends’ the song. This system is optimized for remixing either an entire song, or continuously remixing a loop. The loop works fine, but this is not pure, continuous music. Autogression wins here. -Many others, for a more exhaustive list, please see the full writeup via the project page -Please let us know if you find any, I would love to try and address them if possible

LINKS My YouTube (DEMON tutorial): https://youtu.be/FBv1b5gmjcE Github: https://github.com/daydreamlive/DEMON Project page: https://daydreamlive.github.io/DEMON LoRA: https://civitai.com/models/2416425/acestep-loras DreamVAE: https://huggingface.co/daydreamlive/DreamVAE Try it w/o installing: https://music.daydream.live

[cmuir]

Let's talk more about why it is reason number 2.

[ryanontheinside]

lets talk philosophy, bro

[byMARCO]

Wow this is so cool! Do we have VST support?

[hthillman]

can you explain performance differences vs Magenta / Lyria? that's the most similar thing I can think of

[ryanontheinside]

Good question. First and foremost, Magenta is autoregressive, and this is not. Practical differences: THIS: - has more controls; output can be highly custom - has very fast response time for updating controls; this approaches real time - runs on consumer cards - is optimized for loops, and full songs, not continuous infinite music

MAGENTA: - can generate music forever, it plainly wins here - has fewer controls - has high parameter update latency (chunks of 2 seconds) - I did get magenta to fit on a 5090, but just barely

[djfobbz]

Mind blown! This is absolutely sick.

[tritone1]

incredible piece of work

[ryanontheinside]

thank you kindly

[viborcip]

this is sick and why not bro!!

what was the biggest unexpected unlock you got while building this? like was there something that just worked out unexpectedly easy compared to the rest?

also, another question, how much time it took you to build this thing all together?

[ryanontheinside]

biggest unexpected unlock i would have to say is the shared mutable curves that bypass the ringbuffer drain. i wouldnt have thought of that without the throughput that preceded it

[viborcip]

very cool!

[ryanontheinside]

oh and as for time - this is several months of work, but is the culmination of several years of audio adjacent genai work

[viborcip]

nice to see it all coming together. curious about the VST soon too!