Hacker News new | ask | show | jobs
by gjm11 5185 days ago
Yes, test everything. But what exactly do you mean by "get a feel"?

Suppose your CTR is known to be either 0.09% or 0.11%, you've had 10,000 impressions, and you've got 11 clickthroughs. (This is of course the most likely number if your CTR is actually 0.11%.) The likelihood ratio between the two possibilities is about 0.81. So if you thought those two possibilities were equally probable before, you should now think it's about 55% likely that the CTR is 0.11% and about 45% likely that it's 0.09%.

So if by "get an idea" you mean something much stronger than 55%:45% then I fear you may be fooling yourself, no matter how much you've spent on advertising over the last few years. (Whether that means you should reconsider "test everything" depends on the costs of testing -- the actual cost of doing it, and the cost in running something other than the currently-believed-best version.)

And with 1000 impressions? Forget it. You expect to see 0.9 clickthroughs on average with a 0.09% CTR and 1.1 on average with a 0.11% CTR. You can probably get some extra information from (e.g.) when that expected single clickthrough happens, but it's not going to take you near to that 55:45 ratio. (I might believe 52:48.)