[austincheney]

The biggest technical challenge we came across is that when I had to hire my replacement we couldn’t find competent developer talent. Everybody NEEDED everything to be jQuery and querySelectors but those were far too slow and buggy (jQuery was buggy cross browser). A good A/B test must not look like a test. It has to feel and look like the real thing. Some of our tests were extremely complex spanning multiple pages performing varieties of interactions. We couldn’t be dicking around with basic code literacy and fumbling through entry level beginner defects.

I was the team developer and not the team analyst so I cannot speak to business assumption variance. The business didn’t seem to care about this since the release cycle is slow and defects were common. They were more concerned with the inverse proportions of cheap tests bringing stellar business wins.