[sorenjan]

Related to this, is there a name for the effect when you stitch together video frames into a static background while keeping the moving objects moving? The best example I can think of is this Bigfoot video[0, 1], where the shaky footage has been combined into a bigger canvas with "Bigfoot" moving through it. It's a combination of video stabilization and image panorama, but with some smarts to only keep one version of the moving object in each finished frame.

[0] https://www.youtube.com/watch?v=Q60mSMmhTZU [1] https://x.com/rowancheung/status/1641519493447819268

[iamjackg]

A long time ago I did some work to do exactly this in an automated fashion using ffmpeg. It wasn't perfect, but it was better than nothing. I tried going back through my bash history, and the last related entry was this command line:

    ffmpeg -i C0119.MP4 -vf vidstabtransform=interpol=no:crop=black:optzoom=0:zoom=0:smoothing=0:debug=1:input="weirdzoom.trf",unsharp=5:5:0.8:3:3:0.4 kittens-stabilized.mp4

I think the trick was to set all the stabilization parameters to 0 and crop=black to force ffmpeg to move the image around as much as necessary and zoom everything out.

EDIT: nevermind, it was more complicated than that. I actually wrote a Python script that modified the motion tracking information generated by ffmpeg to reduce the zoom amount and fit everything within a 1920x1080 frame. Man, I wish I'd added comments to this.

The https://www.reddit.com/r/ImageStabilization/ subreddit has a lot of posts in that style, but from the research I did it seems like it's mostly done manually by lining up each frame as a separate layer and then rendering an animation that adds one layer per frame.

[chompychop]

I believe the term in research literature for this is "Dynamic Video Synopsis". Check this CVPR 2006 paper for instance: https://www.cs.huji.ac.il/~peleg/papers/cvpr06-synopsis.pdf