[pg]

All successful startups are lucky, but they're never just lucky. If they were, we could save ourselves a lot of work reading applications and doing interviews. We could just pick randomly instead. If we did that for one batch, you'd really be able to tell the difference, believe me.

[nwhitehead]

I would absolutely love to see a double-blind randomized experiment with controls for YC applications. In one batch, randomly and secretly assign CONTROL or EXPERIMENT to each application (in whatever proportion you are comfortable with). Of the CONTROL batch, do regular reviews. For the EXPERIMENT batch, randomly accept the same percentage of groups as the control. Then have people outside of the reviewers try to label each group correctly at different stages and see how they do. These people could be: YC alums, YC staff not on the interview loop, VCs making investments, outside experienced CEOs, successful startup founders.

For full double-blindness you could even do interviews with the EXPERIMENT group, then use a program to automatically either use the interview result or not. This might be aggravating for reviewers, though, seeing their work wasted by random choices.

[fragsworth]

How can you really do any experiments on this kind of shit, when the real winning investments exist in front of a zipf curve?

You get one or two huge winners out of hundreds, and they are so huge that they are bigger than all the others combined. The investors care most about getting these rare winners, and if YC does any kind of controlled study on who becomes these winners, I don't believe there will be any statistical significance to speak of.

[fleitz]

If there is no statistical significance then it means that there is no difference in picking randomly and doing interviews for the rare winners.

Since the winners are what they are after they can omit the interview process.

[dllthomas]

That's not what "no statistical significance" means.

[fleitz]

Sorry, statistically speaking the difference in results of interviewing and not interviewing are insignificant.

I have a feeling that eventually there will be a Vangaurd of startups that will massively outperform traditional VC/incubators due to reduced management fees similar to index funds vs. regular funds.

[vasilipupkin]

On the one hand, it's probably true that experts overweight their own ability to pick winners. My guess is, PG is likely overestimating the ability of YC staff to actually identify good patterns simply because it's easy to draw conclusions from just a few datapoints but it's hard to draw statistically significant conclusions that hold up out of sample. However, I doubt that a startup index fund will outperform the best VCs and incubators simply because good deal flow is key, whereas in publicly traded markets, everyone has access to all deals. Having said that, I have no doubt that a VC index fund would outperform many of the VCs that are not that good

[powera]

So pg's years of experience getting better than random results is irrelevant and you'll only believe him if he does a long experiment (that would cost him lots of money if it's not just luck) to prove something he already believes?

[scarmig]

It's unclear whether he's getting better than random results. You're assuming the consequent.

It's well-known that the most successful startup in a group of startups is likely to earn more in profit than all the rest combined. We're here on HN talking about PG because YC has been the most successful of all incubators. However, there's always going to be one incubator that significantly outperforms all the others as a result of the distribution of startup success.

I don't know enough about how well PG's investments have done to make informed judgments, but only a very select few people are likely to have that info. I suspect those people have also done their due diligence to figure out if YC is a value-add compared to other incubators (it should be possible to do this without a convoluted double blind experiment), but anyone who's not them really can't say for sure that that's the case.

[baddox]

Generally, and especially with science, the way things work is that you prove something before believing it.

[temujin]

This is oversimplified. In many cases, it is the conjunction of a reasonably strong prior belief and the absence of preexisting "proof" that motivates a scientist (or mathematician) to try to prove something in the first place.

The key is to perform proper Bayesian updates in the face of evidence in either direction; if you do, as long as your prior wasn't totally insane it doesn't really matter where you started.

[epsylon]

Thankfully for us, business isn't science.

[acemtp]

Launching a successful startup is luck BUT selecting entrepreneurs/startups who have luck is no random. You can definitively find some metrics to evaluate the current luck of a startup and expect that the founder has enough skill to continue to maximize luck in the future.

[jbooth]

It'd be unfair to great startups in the experiment batch -- not that it's pg's job to make life fair for people, but it'd put a big dent in YC's reputation. Why spend time flying out there if there's a chance your interview doesn't even matter?

[hotpockets]

You can still do a pretty similar experiment. For instance, startups 1-50 get into yc like normal. Startups 51-100, together with 50 randomly selected startups get grouped together and tagged as "the bubble group". Meaning that they were on the bubble of getting into yc. Pretty high honors. However, they don't get any access, consultation, etc. to yc itself. Then after a few months you unblind which were highly selected and which were random, to see who does better.

[nwhitehead]

You could also accept groups with good interviews in the experiment batch, but not include them in the experiment.

[jbooth]

But then they're effectively excluding the best-interviewing groups from the random experiment.

[jayd16]

Can't you pick the best interviews, then hold your experiment with random groups as long as you mix in a the data from the interview groups in such a way to build a representative sample?

[baddox]

We would only be able to conclude that it's unfair for "great start ups" if the experiment showed that random acceptances performed worse than the interview process.

[brudgers]

If it were just luck then the companies rejected by YC should statistically perform as well as those selected for interviews which should perform as well as those offered spots in a given batch.

[jxcole]

Sorry, not an expert and I have never been an entrepreneur so forgive me if I am misunderstanding, but doesn't YC give them money? In order for this statement to be true either (1) having startup seed money has no bearing on success or (2) YC gives the same money to people it rejects as to people it accepts.

[brudgers]

The amount of capital provided by YC is trivial...a rounding error relative to the value of a successful company. But more relevant is that YC almost certainly has some meaningful data on the companies it rejects.

So PG's claim could very well be supported by proprietary information.

[nitrogen]

The amount of capital provided by YC is trivial.

Never been accepted to YC, but I gather that the value of access to YC's demo day and a strong network of investors and alumni is far from trivial.

[frobozz]

Trivial once the company is successful, but isn't the purpose of the money to ensure that the founders don't have to give up and get "proper" jobs before the company succeeds?

[argonaut]

But then you're acknowledging that startup success is not solely dependent on luck...

[baddox]

I think you're interpreting the idea of luck incorrectly. Luck in this context is simply the phenomenon of being one of the start ups with high success. To say that success depends solely on luck means that the success of any startup is simply rand(). That doesn't mean all start ups are equally successful.

[indrax]

Giving selected startups money confounds two factors: the selection criteria and the extra funding. This means that any differences in the success rate can't simply be assigned to the selectors choosing wisely.

[dangoldin]

Unless being a part of YC adds value which I suspect it does. Impossible to really test this unless YC would provide the same level of support and resources to other companies.

[icambron]

Depends on what "luck" entails. Success is measured by something--for the sake of argument, let's sake acquisition price. But there have to be proximal factors; to be a bit reductive, no one is arguing that large companies pick randomly for the world of startups and them for millions of dollars. Perhaps the most immediate proximal factor is number of users or revenue or...whatever. You wouldn't say, "if it was just luck, companies would preform just as well whether they had users or not".

So the question with luck is how far down the causal ladder you start considering things inputs (i.e. things which may or may not influence the outcome) instead of outputs (i.e. things which might be the result of those inputs or of luck). So with YC, is that getting or not getting into YC is an input variable that we could check against success, or is getting a YC spot one of those proximal factors, and the luck was just in getting that, at least in part?

So the "is it pure luck?" question isn't very well defined, but the GP's argument might still be useful in answering it. Moreover--larger questions of luck aside--it would be helpful in determining whether the YC selection process is actually useful. But probably only if they didn't tell anyone about it.

[boomzilla]

I think the OP's point is after filtering out obviously non-successful startups, the chance of success is potentially non-correlated with other "traits", "characteristics", or "metrics" etc. that people often associate with successful startups/founders.

It would be an interesting experiment if YC lower the acceptance threshold to say twice the original batch, and randomly pick from that bigger pool, and see if there's any difference between the ones that didn't make the original cut and those did. (A/B in its finest :)

[pesenti]

In order for us to "believe you", you would need to test your hypothesis at least once - for example, by selecting a small number the startups you invest in at random and see how they fare compared to the other startups. Keep in mind that most stock pickers would respond exactly like you even though their choices are rarely (if ever) better than chance.

[vacri]

If I was to walk for the first time into your kitchen and tell you that your hotplate was turned on, would you not believe me until I'd also experienced your hotplate turned off? You would discount my previous experiences with hotplates, because maybe, just maybe, your hotplate is hot enough to boil water when it's unpowered?

The idea that the null hypothesis is the only source of information needs to die.

[baddox]

But on the other hand, there are well-known cognitive biases and counterintuitive facts that can cause people (including extremely intelligent people and even people aware of the biases and facts) to believe things that scientific experiment can show to be false.

[pesenti]

Figuring out that a hotplate is turned on can be done though a very simple and direct observation, so most likely I would take your word for it.

Figuring out what makes people successful at starting companies is extremely complex. And without objective evidence, no level of expertise, be it PG or other VCs, will convince me to take someone's word for it.

[acemtp]

It's not luck for you, it's luck for entrepreneur.

Your job is to select people who were lucky enough to meet the right people, meet the market, had luck to have enough traction, lived in a country where he could spend time learning and so on...

But when you are entrepreneur, you have to maximize your luck by maximizing number of opportunities.

[yarou]

Spot on. I think it's important to mention the context of Taleb's philosophy. You have to keep in mind that in certain jobs, like trading, it is fairly plausible that successful traders are plain lucky because all traders eventually blow up. In the startup world, this analogy doesn't hold for a number of reasons. If you are genuinely skilled, you will succeed. Perhaps not on the first try, but you will succeed eventually.

[socillion]

> If you are genuinely skilled, you will succeed.

Some skilled people succeed, therefore all skilled people succeed eventually?

Alternatively, it's a tautological statement that defines "genuinely skilled" people as those who succeeded, therefore unsuccessful people must not be skilled.

[paulbaumgart]

The crucial difference might be that the finance world is zero-sum while the startup world (wealth creation) is positive-sum.

[rbanffy]

I was discussing a somewhat related subject yesterday, on an in-company product development workshop.

For a DropBox-like product, feeling safe you won't lose your files is the baseline. If you cannot reliably ensure the files won't get randomly deleted, you don't have a product. Once you have a product, the value you add gets increasingly more subtle - better metadata capture, nicer ways to organize your files, an outstanding slideshow or music app - and what determines your ultimate success may feel completely random and totally unrelated to the original way you defined your product.

It would be interesting if we could measure luck based on previous YC batches.

[lifeformed]

Basically, the random choice is pulled from the pool of all the talented and productive people. So the best we can do is try our best to continuously be talented and productive, and hope we get lucky one of the times.

[wellboy]

You just need to let in exclusively the crazy startups, the insane ones. That's the only metric.

90% failure rates, but the remaining 10% (5 startups) is all Dropboxes and AirBNBs --> Average YC startup valuation up 10-fold.

[ForHackernews]

Why not try it? http://www.investorhome.com/darts.htm

[acangiano]

That would actually be a rather interesting, if costly, experiment for you guys to try.

[halfcat]

All you stat nerds keep on talking about statistical significance. While you are busy waving your hands, PG will keep raking in millions.

"Your best? Losers always whine about their best. Winners go home and fuck the prom queen."

[minikomi]

This makes me wonder if there's some kind of fantasy football for startups played behind the scenes by some people.

[diminish]

It's best if founders are all up to get some fun, and investors are in to get lucky, not vice versa.

[kevando]

So you're saying my app that helps hamsters connect via foursquare has a chance?

[loceng]

What about if given enough time and access to enough resources?

[man_bear_pig]

More interesting than double-blind test is if each of the yc partners play fantasy startup.

a) each pick his/her top 5 yc companies per batch that they think will be most successful (and main drivers why) b) each pick his/her bottom 5 that they think will be least successful (and main drivers why) c) each assign the value of each of the 5 most successful companies 5 years from now. (and build a bridge as to how that value will be achieved).

and then run some fun numbers on your predictive capabilities.

when i ran "c" for my private equity firm on all investments made over 20 years in various vintages and industries, i found my answer as to why my job of building complex financial models was pretty much worthless number painting. very low predictive ability in value drivers and financial projections being met (these are later stage businesses, too). also, i realized that i should get out when every newer vintage had a larger % of the driver of value stemming from leverage (i.e. financial engineering) than ebitda outperformance.

[mdda]

NB: There is a lot of hard-earned wisdom distilled in man_bear_pig's comment.