[rurp]

I had complaints about 538, especially the early days, but don't understand this critique at all. A 30% chance hitting is completely unremarkable, and it was a perfectly reasonable reading of the evidence at the time. Nate isn't wrong that conventional wisdom was way off, with even supposedly statistical models giving Hillary a 99% chance of winning.

Elections, like many things, have some inherent uncertainty. A several point polling error is normal, so a candidate who is down a couple points on election day has a decent shot of winning.

[ngriffiths]

Discussion of stats models is always complicated by the fact that a lot of people will read "30%" as a "no" prediction and claim your model is wrong if the thing happens. On the one hand, one strategy is to "hide" the numbers a bit behind a blaring headline that says "we are not sure!!" It's a bit of an art to decide when to be "sure" or not. On the other hand, in research for example you can just say screw it, I care if the correct people are correct, not if a bunch of wrong people are wrong.

I feel like the correct strategy for 538 when it was actually niche was to be precise, but then it went viral and maybe should've hit the IDK button much harder and more often after that.

[gh02t]

The real caveat is that 538 was a Monte Carlo model, and is only as good as its inputs. "Here's what the current spread in polling numbers is *given our model and the current polling and their reported uncertainties.*" Polling uncertainties are themselves computed under certain models, and those models are subject to errors. I don't think 538 hid this, but it's a difficult caveat for people to reason about because the sorts of modeling errors that have the most influence usually represent "unknown unknowns".

[this_user]

Building a model for predicting the ultimate winner of a US presidential election is particularly difficult, because you are dealing with noisy input data and nonlinear effects, i.e. just a few thousand votes in a few key states can completely flip the outcome. If you then have poorly calibrated polls with a large margin of error, there is really nothing much you can do.

On the other hand, it does raise the question how valuable the 538 models for something like this really are if the outcome is a coin flip anyway.

[ngriffiths]

Exactly, and correlated errors, where a polling error in one state predicts similar errors across the board.

I disagree that it's all pointless though. Most basically it's smart for campaigns to have a good model and let that inform strategy where appropriate. Since the president is a big deal other people's decisions are also impacted, and in the long run it pays to have good predictions of those chances. Also, the outcome sometimes is fairly certain and that isn't always easy to see.

[gh02t]

I agree, it's far from pointless. The 538 model is arguably close to the best you can do considering how difficult the task is, but it's important to understand it as purely a reflection of the polling data (and 538's reliability scores for polls), and that polling data is inherently flawed. After all, there are only 2 ways to perform a perfectly accurate poll: either know the outcome a priori, or run the election. We shouldn't be too surprised when models like 538 fail to correctly predict the outcome, because that's not what they represent. It's an analytical tool for understanding the current state of polling.

[bryanlarsen]

Regularly referring to that ~30% spread as "one polling error" made this a lot more understandable than most statistics for many people.

[FireBeyond]

> Discussion of stats models is always complicated by the fact that a lot of people will read "30%" as a "no" prediction and claim your model is wrong if the thing happens.

I've even heard things like "70% chance of Hillary winning means she gets 70% of the votes!" (and tangentially, my far-too-long argument with someone on Reddit who insisted "there is no way in hell 50% of the people in this town make above the median income"...)

[bsimpson]

That's a core mechanic in games like Dispatch.

People don't like seeing a 95% chance of winning and then losing. The game tweaks the odds, so certain thresholds become gimmes (something like "if the displayed odds are better than 75%, treat them as 100%").

[tantalor]

That's stupid. That would piss me off.

[greycol]

Seems weird that it would piss you off, if you were really that invested in the cold hard stats you'd know that if it was fair rng you could still have been the 1 in 100,000 player that got lucky on 75% 40 times in a row.

[lmm]

Conversely weather forecasters report a 40% chance of rain when the actual chance is 10% or similar.

So I have a bit of sympathy for people who don't have a good intuition for probabilities, given that the world is constantly gaslighting them.

[a_t48]

Fire Emblem does something complex with averaging random numbers to do the same thing - a 95% chance to hit becomes 99.5, and the reverse for low percentages.

[lacewing]

I don't understand why this is surprising. People didn't go to FiveThirtyEight to marvel the science behind it. The science was just supposed to give you what you came there for: the actual election results.

In the end, it turned out that predicting elections is still very hard, and that for all the fanfare, FiveThirtyEight performed only slightly better than what you could find in any other reputable newspaper, so it kinda lost its appeal.

[akio]

> FiveThirtyEight performed only slightly better than what you could find in any other reputable newspaper

FiveThirtyEight gave Trump double the odds of the next highest reputable prediction, which was The New York Times Upshot (15%). Princeton Election Consortium gave Trump less than 1%.

That is not "only slightly better" to anyone who's statistically literate.

A FiveThirtyEight reader in 2016 was significantly better calibrated regarding Clinton’s chances than a reader of other reputable newspapers.

[Bratmon]

This embodies what 538 and its defenders miss about 538's appeal:

People didn't come to 538 for explanations on subtle points of statistical literacy (although those were provided). They came because, for whatever reason, they wanted to know who would win the election.

People not trained in statistics treated like the scoreboard at a football game- it's always better to be winning, but score is a near perfect predictor in the last minute.

Once 538 stopped delivering perfect predictions and started delivering "Actually the difference between 1% and 30% are way bigger than you think" lectures, the appeal disappeared. There are better places to learn math from.

[akio]

Speak for yourself. That's not why I read FiveThirtyEight.

The purpose of FiveThirtyEight was never to be an oracle for the average person. It was always a deliberately wonky site for a wonky audience. They were very clear about that in the articles they published and topics they covered.

[bombcar]

If we’re brutally honest the vast majority of 538 readers read it to be assured that the right outcome was outcoming.

[akio]

They went to the wrong place then.

----

Nov. 1, 2016 — Election Update: Yes, Donald Trump Has A Path To Victory — https://archive.is/kwdab

> Tuesday was another pretty good day of polling for Donald Trump.

> Trump remains an underdog, but no longer really a longshot: His Electoral College chances are 29 percent in our polls-only model — his highest probability since Oct. 2 — and 30 percent in polls-plus.

> This isn’t a secure map for Clinton at all. In a race where the popular vote is roughly tied nationally, Colorado and New Hampshire are toss-ups, and Clinton’s chances are only 60 to 65 percent in Wisconsin, Michigan and Pennsylvania.

> If you want to debate a campaign’s geographic planning, Hillary Clinton spending time in Arizona is a much worse decision than Trump hanging out in Michigan or Wisconsin.

----

Sept. 16, 2016 — How Trump Could Win The White House While Losing The Popular Vote — https://archive.is/rxP5l

> Using a prototype of a demographic election calculator that FiveThirtyEight will be unveiling in the next few weeks, I decided to simulate a few election scenarios.

> The result? Clinton would carry the popular vote by 1.5 percentage points. However, Trump would win the Electoral College with 280 votes by holding all 24 Romney states and flipping Florida, Ohio, Pennsylvania, Iowa and Maine’s 2nd Congressional District from blue to red.

----

Jun 29, 2016 — Donald Trump Has A 20 Percent Chance Of Becoming President — https://archive.ph/ryIkP

> A 20 percent or 25 percent chance of Trump winning is an awfully long way from 2 percent, or 0.02 percent. It’s a real chance: about the same chance that the visiting team has when it trails by a run in the top of the eighth inning in a Major League Baseball game. If you’ve been following politics or sports over the past couple of years, I hope it’s been imprinted onto your brain that those purported long shots — sometimes much longer shots than Trump — sometimes come through.

----

FiveThirtyEight was probably the worst reputable source to read if you were looking for maximum assurances that Clinton would win.

[hungryhobbit]

538 was never about magically making polls more reliable, and only people that don't understand what polls are could think that (caveat: lots of people don't understand how polls work).

538 was about analyzing and communicating the information from those polls in an easily accessible form. If you came to the site for that, you weren't mad that they "predicted poorly something that was impossible to predict from the data sources they used" ... you were just mad at Trump for winning (despite polls suggesting otherwise).

[lacewing]

Again, I don't think any of this matters. People were not coming there to have "information communicated to them". They were coming there for the satisfaction of knowing the results before everyone else. And FiveThirtyEight couldn't realistically deliver on that.

[rurp]

That makes as much sense as visiting ESPN and expecting them to tell you who will definitely win the Super Bowl next year. Anyone expecting that is going to be disappointed often no matter what.

I thought it went without saying but a good analyst can't predict the future in politics, sports, or anything else. What they can do is make good probabilistic estimates of what is likely to happen. 538 wasn't pretending to do anything more than that.

If people want magic predictions there are plenty of touts and scammers willing to offer them, they don't need to waste time with charts and numbers though.

[lmm]

> a good analyst can't predict the future in politics, sports, or anything else. What they can do is make good probabilistic estimates of what is likely to happen. 538 wasn't pretending to do anything more than that.

Well, sure, but how big is the market for that, really? Particularly for a binary outcome like an election, knowing who's going to win is fun, reading a pundit telling you who's going to win can be fun, but ultimately the man in the street is going to take whatever the pundit said and reduce it to candidate X or candidate Y, and you can only do so much better than replacement level at that.

[anon7000]

I think anyone who actually operates that way is very misguided, but it’s a fair point. But either way, 538 was such a nice site for just looking at the data in a fresh way at the time, and it’s a shame that went away.

If people are expecting anyone to have a magic prediction algorithm for things like this… I mean there’s only so much one can say. It’s not realistic.

[nomel]

I'm very curious to see how polymarket fairs, compared to the news agencies. I suspect prediction markets will be the norm, going forward. Polls can't fully capture the element of anonymity that's required for an accurate poll of something controversial.

[bombcar]

My experience was that prediction markets were lagging indicators and basically followed something akin to an aggregate opinion of polls.

This is especially viewable if you watch them during the 2020 election.

[ghostbrainalpha]

Polls became much less interesting as an Entertainment category once we all had experience with how unreliable they are.

[thwarted]

> it turned out that predicting elections is still very hard

So maybe we shouldn't be doing it. The value of predicting an election in the large out in public seems kind of dubious, and it's more like gambling than actually being useful. A candidate only runs, and continues running, if they think they can win. All predictions like these do is confuse voters leading up to election day and while they are voting. It keep candidates from making strong cases for their platform, keeps the voters from listening to the candidates' platforms, and encourages team-based partisan politics.

[Barbing]

Predictions are like exit polls, aren't they, in that they're both able to provide a check on official election results?

Think that's outweighed by the negatives?

[mulmen]

No. They can’t be used to verify election results. There’s too many confounding factors.

[Sparkle-san]

I can't find the source anymore since 538 is no more, and I recall Nate even describing what could (and did) happen, which was that one swing state moving to the right had a high likelihood of them all moving to the right.

[rurp]

Yeah, Nate has talked a number of times about polling errors being correlated across states. In fact that's probably one of the most common mistakes models can make, treating correlated inputs as independent. There's a long history of that mistake in financial markets as well.

In 2024 the single most likely outcome his model had was trump winning all 7 swing states. The second most likely was Harris winning all 7.

[Retric]

Not just that, predictions also impact voter participation.

[munchler]

I think this is all true, but it dodges the bigger issue. A presidential election has a binary outcome: yes/no, win/lose. If your statistical model doesn’t contain this single bit in its output, then it doesn’t meet the minimum requirement for being a prediction.

Now you might say that it was on me as a consumer to understand this in 2016, but I remember the look of total shock on Nate Silver’s face when he called the winner on live TV that night, so clearly he didn’t really understand it either. Lesson learned for all of us, I guess.

[causalmodels]

> If your statistical model doesn’t contain this single bit in its output, then it doesn’t meet the minimum requirement for being a prediction.

I would be fascinated to know how you model a coin flip.

[munchler]

Well, that’s my point. You can model a coin flip, but you can’t predict it. A model and a prediction are two different things.

[FergusArgyll]

It was 30% in the end, before the nomination it famously gave him a 2% chance of getting nominated. All the talk about 30% is disingenuous

https://en.wikipedia.org/wiki/FiveThirtyEight#2016_U.S._elec...

[lmm]

They had him at 30% at a time when most reputable media had him in single digits at best. You might not remember now, but the result of that election was a real shock.

[Brendinooo]

From the article:

> Being dismissive of Trump’s chances early on in the Republican primary was the biggest analytical mistake of my career, and I think I deserved a lot of blame for that.

He went on to write a whole retrospective, which is exactly what you'd want to see:

https://web.archive.org/web/20160527093831/https://fivethirt...

So unless your standard is "pollsters must be perfect at all times", I dunno what else should be asked of Silver here.

[FergusArgyll]

I'm fine with him[0]. I don't like the apologists who explain patronizingly that 30% odds happen 3/10 times. He gave Trump much much lower odds early on and only put it at 30 in the last few weeks.

[0] I'd say the better defense of him is: the whole model is built on polls, the pollsters stink, people should complain about pollsters, not their aggregators.