[doctorpangloss]

Meta, TikTok and Google have all the data. Not even a sampling. They can exactly answer what is the most effective copy for possibly anything that anyone wants to do. No A/B test required.

So why don't they offer that?

I know it's not because the data is proprietary or private, because basically all the information you need is visible on Facebook Ad Library, more than enough to answer most questions about authoring copy by sheer mimicry.

You emphasize the UX here a lot. I don't know. I think Meta's UX is fine, end of story.

This isn't meant to be a takedown. It just seems intellectually dishonest. Anyone who has operated these optimization systems at scale knows that the generalized stuff you are saying isn't true. You're emphasizing a story about engineers versus product managers that is sort of fictional, like the right answer is the one that most companies are already taking which is to not A/B test at all, because it doesn't matter, and when you do see results they are kind of fictional.

And anyway, it belies the biggest issue of all, and this is actually symptomatic of Twitter and why things were so dysfunctional there for so long, long before they were taking private: you are saying the very Twitter esque theory that "Every idea has been had, and it's just a matter of testing each one, one by one, and picking the one that performs best." That was the core of their product and engineering theory, and it couldn't be more wrong. I mean if you don't have any experience or knowledge or opinions, why the hell are you working there?

> However, coming up with the copy, testing it, and optimizing different variants across users would take us forever—sometimes months.

The right answer is right in front of you! Don't do it! Don't optimize variants, it's a colossal waste of time!

[altdataseller]

I think even if you have all the data, it’s not always a science too because what works for one audience will not work in another.

HN is a good example of this. Headlines that are too outrageous or catchy do not get upvoted that much here but something simple like “I created a rust debugging toolkit” will likely get upvoted like crazy, while something like “I got laid off a day after I got pregnant. Here’s what happened” probably will get buzz on TikTok.

[apsurd]

The danger is it's all local optima. A/b test shows lift on "i created a rust copilot" vs "rust debugging toolkit". Ultimately.. it's one big "so what" outcome. These are distractions. A product either has market fit or it doesn't.

The big companies are especially susceptible to these distractions because they have the budget to blow chasing micro funnel optimizations. It sounds reasonable, but in my experience i agree it's a waste of resources.

It's too hard to prove causality. Entire orgs are set up to run rigid experimentation analysis, and prove incrementality so we can trust the data. But that should be warning of just how complicated it is. and we can't 100% trust it. There's external factors. hence a button color and a line of text shouldn't make the cut list of priorities. it's not that significant.

[altdataseller]

Yes but just because you have product market fit doesn’t mean you shouldn’t try to optimize copy so that it reaches as big an audience as possible

[nehagetschoice]

Yeah - two themes here -

1) Is copy even important? I think it is. If this post was titled, "Auto-tune experimentation for short-form content optimization", it might make half the audience confused about the product. In fact, the 1-liner we use for HN is very different from when we talk to VCs, because the audience is different with different goals & backgrounds. I guess the point I am making is that messaging has to be contextualized, depending on users, platform, and goals.

2) PMF vs copy - I agree that the two are orthogonal. Copy is not going to solve for the lack of a PMF (and it shouldn't). Exactly the point above - the goal is to help more and more users comprehend what you do, hopefully in a way that's more personalized to them.

[apsurd]

PMF isn't orthogonal to copy if you're experimenting with copy to drive an outcome. what is the outcome then? how do you measure it? isn't the state of the art conversion?

That's the challenge: conversion funnel is complex with many factors. and largest one of them, in simple terms, is PMF.

if we measure downstream like clicks or inbound leads etc, that's more aligned with "discovery of PMF" and that's good. But it should be stacked ranked as so, it's not driving the needle. it's exploratory.

[apsurd]

More companies think they have PMF than really do. So the risk is they get funding, without fit, and can afford to set up data science orgs to prove out experiments and use non trivial resources running copy tests.

if justwords can make this trivial then at least it's minimizing the distraction. that's a win, and fwiw i think b2b wants this product, so the company can do well. i just don't think micro content optimization, after doing it for unicorn for 8 years, really moves the needle like people believe the data shows. People use PMF products in spite of their UX! (for example)

[lowercased]

> I think even if you have all the data, it’s not always a science too because what works for one audience will not work in another.

Wouldn't 'all the data' by definition have the data for various audiences at least calculable?

[nehagetschoice]

Appreciate the candid comments and opinions here. I'll break it down and go over them - 1. Having access to data is not the problem we're after (most companies have the first-party data in-house). The key challenges are around having a platform that fundamentally separates strings (copies) from code, and lets you update them effortlessly, based on inferences from that data. So, I am not sure I understand why this is a Google/Meta product? 2. UX is not the value add from this product - agree with you on it (even if it appeared to be the emphasis). The ability to make scientific edits without re-deployments and accelerating continuous iterations based on user feedback, is what we are going after. 3. Curious why you think A/B test results are fictional? Getting stat sig results is probably the surest way to conclude results. Perhaps there's a different angle you are talking about here? 4. RE: don't A/B test at all. Given the number of users that get exposed to every change a consumer company as large as Twitter makes, not testing can be disastrous, which brings up another great point - Large companies are struggling to use all the (generic) gen AI content today, because it needs to be performance tested before it can be placed in front of millions of users, and that's not scalable today. 5. You may be alluding to another good point - copy is as much as art, as it is science, and writing it well takes context, quality, and expertise. That's something we hold a strong opinion on, and we don't see this or any other tool eliminating that expertise. The goal is very much to streamline and augment those workflows.

[doctorpangloss]

> So, I am not sure I understand why this is a Google/Meta product

Which audiences am I optimizing copy for? Where do they come from? Some Google, Meta TikTok or Apple owned channel right?

Google has indexed every website. Meta has every ad. Can't they just tell me what copy to use? Why don't they? I mean, they know! They know what copy works best, for pretty much everything. They can sort by clickthrough rate, revenue due to the purchase data they have, they have everything! You talk about SMB - they know every SMB! They know your margin and your COGS and whatever because they in aggregate they observe rational spending where all the cost is eaten by marketing; they know your potential market, etc. They know all this. They don't need to run tests. They can look at very recent, weeks old, historic data, and they have way more than enough samples to answer these questions to more or less the same degree of certainty and scientific rigor that any SMB doing it themselves as a first party can do.

I mean if they wanted to, they could run the A/B tests for you! Google could "just" serve a different web page with slightly altered copy. And see if more people "click" or "convert" or whatever. They have better technology, 1,000,000x more data... Why don't they do this? You wouldn't even need UX. It could just happen, you would check a box, and they would do this.

> fundamentally separates strings (copies) from code... and lets you update them effortlessly... The ability to make scientific edits without re-deployments and accelerating continuous iterations based on user feedback...

You keep talking about UX for developers and product managers. These are UX things. It doesn't actually matter. The existence or non-existence of what you're talking about doesn't correlate to higher or lower conversions, it isn't a scientific opinion on the practice of optimization, it is just a bunch of UX patterns to achieve it, but it could be achieved in many ways, perhaps with even better UX. Like in the example I gave, where Google "just" does this for you, which is the best UX because there is no UX, you don't need to separate strings from code, and you don't need to update them, because you don't need to do anything. Google could just do this. They own the channel, they see everything, they have the technology.

So why don't Google and Meta and Apple offer an automatic optimization product? You ought to have an opinion, it can't just be, "I don't know." I mean the sort of obvious answer is that "optimization doesn't really work" instead of "three paragraphs of bullshit."

> Curious why you think A/B test results are fictional? Getting stat sig results is probably the surest way to conclude results. Perhaps there's a different angle you are talking about here.

Well one reason I am very confident they are fictional is because the people who own the channels for a decade haven't offered a tool to do this.

I mean maybe they will. Maybe it was a technology problem, but I don't believe so. You could have Markov Chained your way through 5 word long taglines and whatever. They didn't need to way for generative AI to create valid test strings for people's websites. Indeed they could just let you copy the best performing taglines they see in their systems. Why. Don't. They?!

> Given the number of users that get exposed to every change a consumer company as large as Twitter makes

Another POV is that every change they made was bad. They thought they were a product organization, and they are really a backend engineering organization, where the best decisions are all based on first principles or executives' opinions, not on some unknowable measurement about audiences.