Launch HN: [name-redacted] (YC [year-redacted]) – Price A/B Testing for Shopify

[_8lib]

Hi all! I'm [name-redacted] and I recently founded [name-redacted] ([url-redacted]). [name-redacted] helps Shopify sellers A/B test their product pricing so they can make more money.

Let’s say you sell shoes on Shopify for $100 and want to test $110. You’d 1-click install the [name-redacted] Shopify App and go to a dashboard where you select and start a test for $110. We then handle the heavy lifting of showing price A to 50% of your visitors and price B to the other 50%. As the test is running, [name-redacted] keeps track of revenue, conversions, and visitors at each price point. You’d repeat this process as much as you can since the optimal price for a product is always changing.

Up until now, most people have priced off competitors, but that doesn't cut it anymore with big companies getting more and more sophisticated. A prime example of this is Amazon accelerating their development of in-house brands (~640 in late 2019) and using their dataset to outprice other online sellers (2.5m price changes per day in 2013 and more today).

One recent test of a best selling product for a Shopify brand doing 1M+ monthly revenue resulted in a data significant ~10% revenue lift from a -7% price. This better price for the best selling product translates to a ~30k+ monthly revenue increase (~180k+ revenue over 6 months).

I chose to work on pricing because I love optimization. Past optimization includes dropping out of college after 1 year to be a growth (A/B testing) engineer at Pinterest for 3 years and building an app for discounts + promotions at Hulu. In high school, I also optimized enough to be a top 200 player in League of Legends.

Would love to answer any questions or comments below!

PS. If you know anyone on Shopify, I'd love to help them nail down their pricing. :D

[psadri]

I remember people used to hate finding out they were shown a different price from other people for the same product. Has this changed?

[Kip-Kasper]

I think people will change their minds as this gets more adoption, or if they don't tools to track price changes over time & notify will become more popular and price-sensitive people will buy when their ideal price target is reached.

On a separate note, I assume from the post and the copy on the landing page (that I see) that this tool doesn't help you price discriminate on an individual basis but instead changes the price across a cohort of n segments (A/B/C...) in a round-robin way, doing this is very different from predicting someone's preferences and real-time adjusting the price to a predicted max they would pay. I think people would be right to get really pissed at that, but this seems to just figure out what aggregate pricing should be based on demand (or am I wrong about this OP?)

[_8lib]

Addressed this a bit in another comment, but if you think about it, non-public/targeted discounts are pretty much unorganized price a/b testing. Also, the alternative is having sub-optimal pricing which isn't going to cut it pretty soon with Amazon (+ other big companies) dynamic pricing tech + in-house brand expansion.

[misterbwong]

I'd argue there's a qualitative difference between A/B testing regular pricing vs. testing via targeted discounts.

If your friend finds out you got a "private sale" price, this doesn't seem to violate the "fairness" principle because you were a part of something or got targeted for a reason. It might actually end up increasing the friends FOMO/jealousy and get them to find ways to join in on the discount via mailing lists, insider clubs, etc.

A/B testing pricing might come off as arbitrary and not easily reasoned away thus violating a sense of fairness (at least during testing).

I don't see this as a huge issue as A/B testing is usually limited time/scope but something to think about, I guess.

[neonate]

Could that problem be solved by changing the price at different times, rather than for different people in the same time interval? I wonder if that's why Amazon is making 2.5m price changes a day like the OP says above.

[_8lib]

Unfortunately this wouldn't be a proper a/b test since it'd be comparing 1 day of 1 price to another day of 1 price. Only by splitting traffic 50/50 do you properly isolate the price variable.

[neonate]

Why can't the time slices be much shorter, say 15 minutes rather than a whole day?

[dividuum]

Most likely, although that solves a different pricing aspect. I would guess most of those are the result of automated pricing optimizations to influence your offers position among other sellers for the same product. There are services that automatically do this for amazon and other price comparison services. It can also result in a book priced at $23,698,655: http://www.michaeleisen.org/blog/?p=358 :-)

[hn_throwaway_99]

I recall seeing this blog post a while back arguing you should never A/B test your pricing strategy (IIRC it was posted on HN, can't remember): https://www.priceintelligently.com/blog/bid/180676/why-you-s... . I'm curious about your thoughts on the points in this post.

[_8lib]

Re point 1: in many cases people actually DO have the sample size if the magnitude of the revenue increase is large enough (e.g. double price and increase conversions - lots of stories about this).

Re point 2: in the end increased revenue from better pricing means you're capturing more of the value you're delivering to customers. but yes, there's a brand decision to be made here between 1 price that you always keep the same vs. a changing price that is more optimal.

Re point 3: companies should always be testing their pricing since external factors don't stay the same.

PS: this is also on the blog of a dynamic pricing company so they don't want the people on the fence about handing over pricing control thinking about a/b testing price haha.

[mritchie712]

ProfitWell is targeted at B2B SaaS, very different from ecommerce (Shopify).

[outloudvi]

"Keep in mind that Amazon does dynamic pricing already so it's already commonplace to not always get the same price."

(from [url-redacted] )

"Amazon also does something" doesn't mean that's right. As one of the biggest cloud providers, AWS is widely recognized for its complicated pricing model. Does it make a complicated price model good, welcomed, encouraged or well reputed?

(Why I'm still standing to object this even when this thing seems to make sense? Because I'm a customer.)

[mtnGoat]

Can you easily opt traffic out of testing based on traffic source? for instance Google Shopping(Microsoft Shopping as well) is pretty strict about the price in the ad page, matching the price on the product page.

do you plan to offer this product for other platforms? the number of Shopify merchants with traffic and sales large enough to get good significance in a timely manor is rather small, IME.

[cdipaolo]

1. How are you handling statistical significance questions / ensuring the A/B test has enough statistical power to be meaningful?

2. Same for seasonality.

Are you offloading much of the statistical concerns to the user or are you hoping to answer these for them?

Have to say, I don't really understand the hate on this thread having worked in this area. This seems like an awesome product that can really help sellers.

[dyeje]

Do most Shopify stores have enough traffic for this to produce real results? Seems like the vast majority of Shopify sites are small and are just going to get random results due to a small sample size.

[_8lib]

Depends on the magnitude of the revenue increase.

One thing that inspired me to build [name-redacted] were the stories of sellers doubling their price and actually increasing their number of conversions. I haven't had a customer like that yet, but I had one person who reduced their price 30% and increased their conversions 5x+. In that case, it's pretty clear there's a winner.

[dustin]

30% might work in software, but it's is a very deep discount for most physical retailers.

If the product has a 50% margin (sells for $100, costs $50 to source) then your retailer makes $100 for 5 orders instead of $50 for 1 order.

Except they're probably keeping their own inventory, so they now have to keep $250 of inventory for that $100 in profit, rather than $100. This is problematic because a lot of online retailers are limited by their working capital.

This is a drastically simplified example, but the bottom line is that physical retailers think very differently than software sellers.

Great idea for an app either way. I wish you the best of luck with it as you learn the territory. For something similar in the Amazon space (a platform that lends itself much less to split testing) check out splitly.com

[pgodzin]

Few questions:

1. How do you determine who fits into the A cohort and who fits into the B cohort? If I refresh could I fit into a different cohort and thus see a different price? If I go Incognito?

2. Instead of a 50/50 A/B test, can you multi-arm bandit a handful of different price points and shift a higher proportion to the "best" one? And thus be able to change over time as well?

[_8lib]

1. Using random id + localstorage, so yes, incognito could result in a different price.

2. I'd love to implement MAB, but focused on nailing core product for now.

[kipply]

How do you avoid customers changing their behaviours by observing pricing changes? Or when they share it with a friend and go "cute shirt for 10 dollars" and their friend is like "wat it's 15 dollars"

[cdipaolo]

Could cohort based on geo-location instead of completely at random. Makes analysis a bit more challenging though and doesn't completely solve the problem of course.

[wackget]

Advertising prices like those in your example screenshots would be illegal in the EU.

You can't imply that the price "was" $49 and is now $39 when it's also being sold at $29 to another group of customers.

In fact, the whole thing is legally questionable according to EU price fixing laws, not to mention ethically unsound.

[srg0]

Would advertising $49 as the base price and a time limited offer/discount for a group of customers be acceptable?

[desdiv]

How do you deal with search engines? The vast majority of stores have the the vast majority of their customers come in via search engines.

1. Showing one page to the search engine spider and another to users is a black hat SEO tactic known as bait & switch. Most search engines heavily penalizes against this. How do you deal with your customers possibly losing their search engine reputation that they've built up over the years?

2. I don't know about other people, but when I see a price, X, on the search engine result page, then get a different price, Y, in the actual store, I immediately lose trust in the store. If the merchant is willing to bait & switch me on the price, then I don't trust it not to bait & switch me on their actual product or services.

[niqmk]

Is your pricing [url-redacted] also A/B testing? :D

[davidajackson]

This looks like a great infrastructure as a service company. Is there a way to build an AB testing tool on Shopify that finds the right price by zeroing in on it, rather than having the customer input several? Did you consider that approach?

[_8lib]

Yep, automatic testing will be coming soon! :)

[newfeatureok]

If this were to become popular what would stop someone from building something to do the reverse of this, e.g. see if the item could be cheaper if you bought it as another customer/from another device/computer/location?

[_8lib]

Yeah, someone could totally build Honey for [name-redacted] if they really wanted to here haha. If you think about it, discounts that don't apply to everyone (what Honey targets) are pretty much unorganized price a/b testing.

[asattarmd]

That's probably a good problem to have! Nonetheless, customers are still buying from the same store, so, it's a win-win. Breaking customer trust is probably the only thing that's bad here.

[_8lib]

Yeah, it's a tradeoff for sure. I would say discounts that don't apply to everyone are a less organized way of a/b testing price. So a brand that does discounts all the time on certain cohorts of their users is going to have an easier time swapping over to using [name-redacted] vs. a brand that has 1 price, no discounts ever, never have changed their price.

[usaphp]

I just want to point out that I like a way you describe what your product does. Simple, clear examples and result. I like when people do that instead of asking for your email just to see what your product is even about.

[_8lib]

Thanks! Keeping things simple + self-explanatory is a big focus with [name-redacted].

[jasoncui]

+1 to this

[alphagrep12345]

How does it compare to other A/B testing apps? Eg: https://apps.shopify.com/neat-ab

[_8lib]

Neat A/B is a 100% daily traffic swap app. This approach isn't a valid a/b test since it isn't properly isolating the variable (price).

[neatab]

Neat A/B Testing here! First off, congratulations on the launch. I would argue showing different prices to simultaneous visitors is an ethical gray area - not to say it's not something that shops do, or wish to do. But that's why we choose to cycle pricing. In our experience, with proper consideration of cycle timing, test length and traffic consistency, it's possible to get solid results this way.

[_8lib]

Hey there! Thanks for the congrats. I see your point, but external factors + different cohorts visting throughout the test is a bit of a pickle. Nonetheless, I wish you the best!

[itsdavemartin]

One observation: Pricing at $199, $499, $899 per month for a single feature seems interesting considering Shopify itself charges $29 $79, $299 per month.

[gustaf]

Considering the value of [name-redacted] I'm surprised it's not higher price.

[ketzo]

Yeah, this is totally a "charge more" kinda scenario. A/B testing is, like, THE first thing you do once your ecommerce business gets a little established, right?

[dontwaitesforme]

Upsetting to see the negative reception to this. I know as a consumer this sort of experimentation might leave a bad taste in your mouth, but on the flip side I don't think you can blame a seller for trying to systematically find the fairest price for their product in a market. Overall, I think this is a cool idea with a simple but effective premise.

[shafyy]

Nothing against you personally, but I'm the only one who thinks it's weird that a Shopify A/B testing app got into YC?

[greenbay20]

I'm incredibly disappointed this was the top comment. Instead of asking whatever it is you want to know in an optimistic way, you just threw a random comment that with decent chance just makes the OP feel worse.

For starters, a really simple question you could have asked that may update your belief of how weird it is they got into YC: "How much money are they making and how much are they growing per week?"

[AVTizzle]

I appreciate the compassionate nature of your comment, but anecdotally I can very much attest to this thought being the first thing through my mind when I read the post title.

This question sitting at the top further validates that other people are wondering it, so disappointing as it may be - people are curious!

[shafyy]

I do think that an A/B testing platform can be a great business, and making any business work is for sure an achievement.

I just had the impression that YC tries to back more "ambitious" or "important" businesses, but may be I'm wrong about that.

[kipply]

ambition or "importance" don't always manifest in the early stages of a startup (they probably usually don't). Shopify started as a snowboard selling company and I'm sure that the founders here pitched YC a larger vision that was appreciated.

[break_the_bank]

Yep, that's the first thing that came to mind. Seems like a great bootstrap / indie venture but they raised VC money, I don't know why.

The founder does say that they want to change all of pricing. Even then I'd argue that launching making recurring revenue would have got them more favourable terms with a VC in the future.

[extesy]

I'm actually surprised this is not already built into Shopify by default. This seems like a very basic functionality which can be added at any moment so what would that do with a startup?

[kipply]

Former Shopify employee, but speaking based on public knowledge;

1. Liability. A result of "raising prices by $10 is 5% more revenue" is not confirmation that raising prices by $10 is a good idea. I expect a non-trivial error rate, especially since most merchants aren't statistics savvy (forming experiments, doing risk analysis and interpreting data is hard). If Shopify causes this, it's a story. Startups will have room to develop features / legal to adjust to the liability concerns

2. Channels. Shopify is a very multi-channel platform and many stores actually rely on Facebook or Instagram integrations and physical stores rather than their online store. To be able to control prices on all these channels is an engineering challenge, and will also further dilute the value of the data. It'll also amplify concerns about pricing-inconsistencies.

3. Just a high-risk idea. Customers don't like seeing different prices. Merchants don't like running experiments that could cause them to lose money. Merchants also like being "good" to their customers by some measure, and some merchants would consider this unfair.

[lancesells]

I agree with all of this. If there was any store I shopped at and realized there were two different prices I would think they have some savvy marketers but would also stop being their customer. I don't want amazon or airline fluctuations in price when I'm shopping.

Best of luck though. I can see many stores thinking this is the way to unlock profitability.

[_8lib]

All great points. Data significance + education on how it works is super important here (and something I've been personally explaining to many brands and merchants).

Another thing I'd like to add: most of the comments in this thread are assuming price is going to be increased when in reality a good number of merchants are going to actually end up lowering their prices (and serving more customers). In the end, it's all about delivering + capturing as much value as possible.

[kirillzubovsky]

It may appear that way, but it's not surprising at all.

First of all, it's a great business. Shopify is exploding like crypto kittens, and with millions of sellers, even if you secure just the top sellers, let's say 1% of the Shopify market, and charge $10,000 per year, that's a 100-million-dollar-business. That's incredible.

From there, you have a choice to either expand further in the market, or start offering more advanced tools to your best customers. Given that Shopify is currently at 100B market cap, yours could be 1B in just a few years, if not sooner.

It's brilliant!

[senojretep356]

Yeah, I feel like people don't understand how big shopify is and how quickly it's growing. If you look at Elliot who just failed trying to be a competitor to shopify, you will realise that this is a much better route to go.

[naeemtee]

YC is a crapshoot. We had the most impressive metrics in our interviewing class and killer recommendations and didn’t get in. The ones from our interview group that did were astoundingly dumb.

There wasn’t a single successful entrepreneur amongst the “partners” who interviewed us.

Here we are a year or so later flying at 7 figure revenues thankful we didn’t give up nearly 10% of our business to an incubator that’s a shell of its former self.

[shafyy]

I've said this here before (I think) and I'll say it again. With almost all initial YC partners leaving and YC getting bigger, they also get more risk-averse, just like any growing business.

Therefore I'm afraid that we will see less and less interesting and "out there" YC companies but more confirming ones.

I'm aware that growing has other benefits for YC and its companies, but I'm personally just a bit sad about this...

Edit: I feel like Seibel is the last interesting full-time partner and I wouldn't be surprised if he leaves within 2 years.

[neonate]

Wow. What's the business?

[quadrature]

Thats what the implementation is today. the larger idea/problem is e-commerce optimization which is a big problem worth taking a bet on.

[pryelluw]

This might be their Apple 1 and not their iPhone.

[mritchie712]

they could have gotten in with something more ambitious, then pivoted to this when they realized it wasn't going to work. Not uncommon to launch something different than what you were accepted with (see Brex)

[_8lib]

This is what I applied with.

[shafyy]

Good point

[justinzollars]

Not surprised one bit. This is brilliant.

[_xgw]

Maybe there's potential to port the app to other platforms!

[_8lib]

Yep, going to do all online pricing.

[gleb]

Small suggestion: include how contribution margin changed in that example. That’s the Northstar metric for a typical ecommerce company.

Including it would sound more credible, and would prevent a typical customer from doing the math in their head of profit margin where this change becomes profitable.

[_8lib]

Thanks for the suggestion! Contribution margin/profit is another metric we're going to track in the future for sure.

[mgh2]

Conversion rate optimization (CRO) specialist working on Shopify platform here: Why can't I just duplicate a product page, change the price and do a SPLIT instead of an A/B test? Some reviews will have to be ported/duplicated, but it is totally doable.

[_8lib]

It's better having 1 page from an SEO perspective.

[kkt262]

Why do you think you got accepted into YC? I'd love to check out your application.

[_8lib]

tldr; Pricing is painful. Most companies leave it to chance and there's a real opportunity here to change that.

[purple_ferret]

Do privacy addons block this and if so, does your price tracking account for it?

[zatel]

This seems like a great shopify app I will certainly be adding to my next store. Do you have any go to optimization resources to share? (just general stuff on how to think about optimizing things in ones own life)

[_8lib]

Awesome, feel free to email me at [email-redacted] when you get started and I'll personally onboard you if you'd like! And in terms of optimizing, I think the biggest shift for me was keeping everything I want to do in a list (Google Doc ha) and building momentum by doing the easy things.

[aresant]

Handful of good links on price elasticity testing

1 - https://vwo.com/blog/ab-testing-price-testing/

2 - http://www.conversionvoodoo.com/blog/2012/12/why-price-elast...

3 - https://prisync.com/blog/price-testing-how-to-test-different...

[kipply]

Is there an AI(specifically deep learning) element to optimally distribute the A/B testing ratios or to suggest products/prices that may be good to A/B test? (Referencing the .ai url)

[_8lib]

Yep, automatic testing is definitely on the roadmap!

[akbar-saliev]

If this becomes popular what would stop Shopify from building this?

[dahdum]

I'd guess public outcry - people despise this type of practice and Shopify endorsing it would hurt their brand.

[pkiv]

Congrats on the launch! Have you thought about exposing the API instead of tightly coupling to Shopify? Similar to Split.io but for product pricing?

[_8lib]

Yep, an API is definitely on the roadmap as well. :)

[lukasfischer]

This is evil... I would immediately stop using such a shop, if I know such practices are used. Your solution is harming trust.

[greenbay20]

This is dope. As close as one can be to the money, niched down and in a growing market. Good job! Keep us updated =D

[itisit]

Are there any good browser extensions you recommend that help detect a service like yours?

[jasoncui]

Awesome product and love the landing page. Super simple and clear.

[_8lib]

Thanks Jason!

[gormanphil]

Hi Keenan, want to share you product by Presenting at E-Commerce Day REMOTE on oct 1? Shopify our lead sponsor, Rebecca Minkoff is keynote, 3,000 attendees expected. You’d be a great [e-mail]

[break_the_bank]

Congratulations on the launch! Curious why you went with venture funding over indiehacking/bootstrapping?

Heard this podcast a while ago[0] where they discussed main dishes(who new platforms) and side dishes(github apps / shopify apps). At the moment [name-redacted] looks like a side dish, which seems well served by the bootstrapping community.

Edit: Add link to podcast

[0]https://www.indiehackers.com/podcast/152-tyler-tringas-and-j...

[thrownaway954]

did anyone else do a double take and think the name was Jira?

[lazydon]

May be it's just me, but the first thing reading that name reminds me of is the Zika virus. I wonder why would you choose a name for the startup that's so close to something with horribly negative connotations. Especially, in today's world scared to hell by viruses.

[shafyy]

> In high school, I also optimized enough to be a top 200 player in League of Legends.

Sorry but that's super obnoxious. No one in their right mind would use the verb "optimize" in this context.

Feel free to downvote, but sometimes you need to call people on their bs.

[ketzo]

I play the game, so maybe I'm biased, but that's legitimately a huge accomplishment. Use whatever verb you want I suppose, but that's literally top 0.001% in a game renowned for being incredibly difficult/competitive.

[shafyy]

I play esports myself and of course it's a great achievement. But the word "optimize" just triggers me when used in this context.

[GonzaloQuero]

I see the value on this. As a very, very bad player, but with some knowledge, it´s a game that rewards heavily the ability to strategize, optimize builds for different, very quick situations, and shows dedication.

Unfortunately, I agree that it´s a poor thing to say in a presentation, as it requires people to actually know the game to understand why that would be good. People who don´t play might just think "what do I care that he's good at a game?"

[frakkingcylons]

That’s a big achievement to be honest, so who cares about the word they happened to use there. Takes a lot of focus and work to be that good at strategic games like League, Dota, or Starcraft.

Edit: Actually I think optimize is appropriate in that context too. To be that good, you have to min/max every game mechanic you can.

[MperorM]

I agree that it requires an enormous amount of focus and work. At least for Starcraft, I don't think most pro's are great because of explicitly min/maxing things.

Rather they are great because they subconsciously pick up on minute details to learn from in every match they watch and play. Ask a top player why they did some maneuver, and you'll often get an answer like: "Because it's good". There really hasn't been that much system 2 thought put into it.

Being great at a video game is mostly about cultivating an instinct that always makes the right decisions.

For what it's worth, I was grandmaster in starcraft and made a decent salary coaching lower rank players during my highschool years, so I feel my opinion carries some weight at least :)

[neolog]

I don't see the problem.

[tmpz22]

This definitely originated from the Chris Sacca anecdote about Travis Kalanick being one of the top Wii Tennis players - it either came out in a start-up podcast or some shark tank episode.

Anecdotally watching and following top e-sports players (in Overwatch in particular) I would see high video game performance as at best a neutral indicator... those archetypes of people are typically very good at the one activity they do all day and very poor at taking care of themselves or sticking to things that don't immediately excite them.

Finally, for many video games are a coping mechanism and may be a better red flag indicator for past trauma, depression, etc.