[Aperocky]

> Increasing the number of bookings by 2% here, 3% there.

I've always felt that there is a dilemma in this space. Any number of small UI features can claim a small increase that are often just noise and curated data (assuming no malicious baking), the cumulative increase isn't even close what the claims stacks up to.

The only fundamental way for a business to make more money is to invent actual, new feature and business models. Shifting a box here and there, enlarging an icon just creates a generation of glorified engineers and PMs who are placing their attention entirely on the wrong thing. Yet the success reporting often rewards this behavior and eventually drown the business in fruitless 'noise' endeavors that favors short term gains over actual innovation.

[greatpostman]

Maybe. Without outing myself, I have work on a choke point widget on the front page of a top 10 USA website. We had 30 engineers running these tests all day long. Was common to get 10 million+ increases to the bottom line on very minor changes. This was over a sample size of 100 million page views.

[nosianu]

> on a choke point widget

So a single data point? I didn't see it claimed that there is not a single small thing that can have a big effect. It's about large numbers. I think individual examples obfuscate more than they help us understand, when it is given in lieu of wider view instead of just as a support for one.

In this context, I would also like to see considerations of a larger picture. Sure, it helps any single company make more money. What about beyond that? Because money is a proximate goal, even if it's the ultimate one for a business, from a human society point of view that introduced this tool for a specific reason and not for its own sake.

For example, what does it mean to have some ultra-optimized large companies dominating their space? Would society really lose something without some micro-optimizations, even if they raise profit? The people who end up not making a purchase on a website because they are not as much caught in the optimized patterns, is it actually bad for them, or maybe they are actually better off and the optimizations tricked them into making an actually unfavorable (even if slightly) purchase? I think there is more to consider than just the view from the respective company or even the department. Are the means used to win actually good from a wider point of view that is not centered on that winner alone?

If someone does not make a purchase because something is slightly off, are they actually worse off, neutral or better in the end? It can't have been all that important to them to make that transaction, no?

[yjftsjthsd-h]

Did you ever test doing nothing? I assume it's been studied, but I can imagine ways for 10% to actually be noise or seasonal/weekly/whatever variations.

[greatpostman]

Yeah that’s called the control. We have very sophisticated automated systems for measuring all of this

[hnov]

This is called A/A testing and is used to vet the setup.

[hiq]

I thought A/B testing was the state of the art in that field, which would address your concern?

[EnKopVand]

> The only fundamental way for a business to make more money is to invent actual, new feature and business models.

I’m not sure this is fair for services like Airbnb where what they sell is conveniences.

Their entire product existed before Airbnb. It even existed before the internet. What Airbnb did so successful was that they branded themselves with a good product that disrupted the market by being easier to use.

The thing about disrupting markets by selling easy of use, is that your competition is already well versed in the market you disrupted. Some of them may do the dinosaur but others will take your ideas and try to make them better. If you don’t continuously improve the ease or use of your product, then people are going to download eurobooking or whatever other options there are now instead.

You know this, because you’ve likely done it yourself a million times with various applications that do the same thing, but some find a way to be easier to use.

Obviously these companies are going to want to find (or invent) new products to get included. But that is typically more of a non-software quest that eventually leads to some of the engineers being tasked with building what the marketing/sales/business process people find.

[square_usual]

You shouldn't be downvoted for this - you're not entirely wrong. In my workplace, this kind of "bps gaming" has been a real thing the leadership has has had to take a stance on and fix with better processes and tech to measure cumulative impact.

On your second point, though, if all you focus on is building new features, you are losing out on tons of unrealised gains you could've had by making an old feature work better. At the scale a bigco works in, small percentages add up to quite a bit.

[wging]

> Any number of small UI features can claim a small increase that are often just noise and curated data

The answer to this is better measurement and statistical analysis, not abandoning the effort entirely. It does take more than "haha, trend line go up" to prove a change had a positive impact, but it can be done.

[MarkSweep]

One way to measure whether a team’s cumulative effect is positive is to have a long term holdout of users who never see their experiments. You can check for a statically significant difference between this holdout group and the group of users who see the features the team shipped.

[1123581321]

You raise a point companies should take seriously (making sure the improvements are substantial) and are wrong that the only way is new features.

From a company’s perspective, the PMs and devs good at optimizing existing features already exist. Why not hire them to work alongside and behind the devs and PMs good at developing new capabilities?