| This is a complicated question. I know of a direct-response guy who had horribly ugly sales pages that literally included misspellings. He tells this story about how all kinds of marketing consultants and designers would come in and say, "Your sales page is so ugly! You need a redesign." He would laugh and say, "Everyone says that. But I have data from the last 5 years, and this is the single best-performing page of all." He made $25m last year. Sophisticated direct marketers are not to be trifled with. However, if you're building a company that depends on other non-direct factors like brand, you can't really A/B test that. You just have to make a decision. A few years back, I met a guy whose girlfriend worked on the optimization team at Amazon. Now THEY are sophisticated. They knew how A/B tweaks would affect users several months later. But even still, Amazon often makes strategic decisions that can't always be justified with data. Bottom line: It's important to know the advantages and disadvantages of various types of marketing, including direct/brand/etc. But you will have to make some tough choices. Personally, on my sites, I've seen lots of interesting results with testing designs...but I finally told my staff to stop micro-testing since we'd optimized the hell out of some of our stuff and were only getting incremental gains (e.g., going from 35% opt-in rate to 39% is prohibitively difficult). The biggest results I've gotten have been from strategic changes like changing the offer, adding a new product, pricing changes, back-end offers, deeper customer research, etc. |
In this age of analytics, I think a lot of small startups are overlooking the gut decisions they need to make and focusing too much on optimization (e.g. this HN question).