[earthmancash]

This post is right up my alley, as I'm a.) the CEO of an AB experimentation platform www.geteppo.com and b.) I'm an Airbnb alum where we rolled out an analogous feature that labeled a listing as "this is a rare find!"

And the funny thing is, I agree with this article. Both the content and the heading of this hackernews article:

1. Notification/scare spam can have long term retention ramifications. The previous generation of experiment platforms made long term metrics literally impossible to read. But now companies can use holdout groups and long term metrics like retention to give more clarity.

2. Even if you can read long term metrics that include retention, the scare/notification spam could lead to less word of mouth growth. For travel, I am guessing that you will be swayed to drive WoM growth more by differentiated inventory, reliable service, and cost, so maybe it's just merely annoying but not a risk to the business.

3. Notice that Airbnb's UI is very, very different from Booking, Expedia. We made a conscious choice to always make sure Airbnb came across as a "sincerely helpful friend" as a booking platform. An AB experiment showing that metrics improve doesn't mean that you have to launch it. You can look at those results and say, "this metric lift isn't worth how ugly it's making my site", and that's a completely valid choice. (a choice we made often at Airbnb)