[vunderba]

Yep. Additionally having experimented pretty heavily in this space, I'll state that even with state-of-the-art models (like NB Pro or GPT-Image-2), good luck getting them to generate sprite sheets with consistent frames for animation particularly if you're shooting for a pixel art aesthetic.

I've made a few sprite-based animations with AI assistance but it still required a fair amount of manual photoshop (or in my case Aseprite) post-processing.

[nananana9]

If you're not going for pixel art, use the video to video models (the character swap ones that take in a video reference of the animation and a still of the character (the tiktok deepfake models)).

You can e.g. download a run cycle from Mixamo, use Blender to render a video from whatever angle you wish, and feed that into the model along with your sprite on a green background, it's pretty much perfect.

[vunderba]

Thanks for the info! I'll give this a shot. Most of my knowledge is from over a year ago, when I would set up OpenPose-style positioning with a controlnet at a high strength to guide an image-to-image workflow in comfyui. But it was pretty tedious and prone to errors.

[throwthrowuknow]

It’s a tough problem but Retro Diffusion is making some incredible progress

https://retrodiffusion.ai/

[vunderba]

Nice. I need to go check them out again. I really liked their original Retro Diffusion models but this a while back in the SDXL days. It felt like it was trained on more natural language, so prompting it was quite nice for that era.

But if they’ve been working on it specifically for animation, that’s pretty neat.

[throwthrowuknow]

They have a few different specialized models now and have put a lot of work into generating sprite sheet animations. The latest thing is a pipeline for ensuring pixel perfect results that also look good stylistically.