Another dubious optimization you could do is finding an optimal transport solution to move the apples the least amount of distance, something similar to the Earth Mover's Distance.
I actually tried using TSP to optimize the order of apples moved for a single frame, but what you sent seems like a much better fit (I'll bookmark it for the future). Either way, conveying all the moving information on a single reference picture is hard.