I assume if your weights are resting on the floor, and the pivot point of the swing arm is resting on the floor, then the floor deflecting may cause the arm to swing even in the absence of an attraction to the weights.
That's true, but the details of the placement of the ladder doesn't matter at all. The only thing that matters is that the ladder doesn't move or wobble around during the experiment.
But it isn't. It's suspended from a support by a thin filament. That's the whole point of the experiment.