| I adapted VACE to work with real-time autoregressive video generation. Here's what it can do right now in real time: - Depth, pose, optical flow, scribble, edge maps — all the v2v control stuff
- First frame animation / last frame lead-in / keyframe interpolation
- Inpainting with static or dynamic masks
- Stacking stuff together (e.g. depth + LoRA, inpainting + reference images)
- Reference-to-video is in there too but honestly quality isn't great yet compared to batch Getting ~20 fps for most control modes on a 5090 at 368x640 with the 1.3B models. Image-to-video hits ~28 fps. Works with 14b models as well, but doesnt fit on 5090 with VACE. This is all part of Daydream Scope (https://github.com/daydreamlive/scope), which is an open source tool for running real-time interactive video generation pipelines. The demo was created in scope, and is a combination of Longlive, VACE+Scribble, Custom LoRA. There's also a very early WIP ComfyUI node pack wrapping scope: https://github.com/daydreamlive/ComfyUI-Daydream-Scope Curious what people think. |