[phreeza]

I tried to build a prototype of something like this with opencv once, didn't get it to work reliably though. I have a feeling there should be a relatively simple signal processing version of this, basically a spatiotemporal Fourier transform, that should solve it.

[schobi]

I would go with something like sift features (or a non patented variant thereof if you plan commercial usage). You analyze a first picture and extract the center part. You run a feature detector on each image and the can cheaply match (scale and rotation invariant) against each other - this gives rotation and scaling.

[phreeza]

Yea that's precisely what I ended up doing, but the frame extraction was quite finicky.