[michael_dorfman]

I thought that part of the video was bullshit. I thought the same when Atwood wrote his "Groundhog Day" piece, which the video seems to accept at face value.

A/B testing lets you test two competing hypotheses. It does not in any way require that the two hypotheses be very similar to each other. You can test radically different designs against each other just as easily as you can test minor differences.

So, the real question you are asking is: how do you generate new hypotheses? And the answer is: the same way as always. You can see what other are doing. You can think "out of the box." You can use your best judgment, and whatever creativity you have access to.