[mbesto]

Totally agree. The OP wrote:

"But we can still be fairly confident that the image was the main factor since this test had a much higher change in click through than what we typically see when we just test text vs text."

How can you be fairly confident? They broke the cardinal rule of the scientific method.

[skrebbel]

> How can you be fairly confident?

Come on. It's a puppy. A puppy!

[mhartl]

In other words, we can be confident that the puppy influenced someone—we just don't know if it's the subjects or the experimenters.

[bobfunk]

Well, it's worth noting that while we're interested in the general science of puppies and conversions, our end goal with this test was not to determine the exact value of the puppy picture, but just generally finding the best way to get people to click our call to action button...

[LargeWu]

Except you have 2 variables - puppy/no puppy; and copy A/copy B, and therefore 4 combinations. You only tested two combinations, so you have no way of knowing if the puppy or the alternate copy was the reason for the increase.

[herbig]

But how do you determine the best way by introducing multiple variables into your experiment?

The best way may be with the puppy or without. From this you have no idea.