[woodchuck64]

Not just the left and right's echo-chambers, but those who attempt to actually listen to reality got it wrong, too: FiveThirtyEight for example has been using proven, reality-based statistics for a while and should have predicted this result. (As it was, Silver was lambasted for being too generous on Trump's chances.) At issue in this election is apparently a new idea: people who hold positions that are being publically shamed tend to lie to pollsters.

[downandout]

Huffington Post was one of Nate Silver's largest critics because his math disagreed with their political desires. This graphic [1] - laughably claiming to be scientific - was on their home page on Tuesday afternoon. 538 had Trump at about 28% at the exact moment I took this screenshot, and HuffPo's "scientists" had him at 1.7%. If anyone wonders whether HuffPo and its ilk are "biased but reliable," wonder no more. They're just biased.

[1] http://imgur.com/CSzc5Rb

[wskinner]

Not really a new idea. This effect is called preference falsification: https://en.wikipedia.org/wiki/Preference_falsification

For those who don't want to follow the link: "individuals convey preferences that differ from what they genuinely want"

[maratd]

No, the idea of being persecuted for what you believe and then in turn, lying in public about it is not a new idea.

It doesn't need a Wikipedia article nor a term for it. It has been around since the beginning of time.

What is new is that this sort of environment is taking hold in the US, which is unfortunate. And this forum is as guilty as any other. People lambasted Peter Thiel for speaking his mind and voting his wallet. They demanded his head and spoke of repercussions.

That's what you get when you behave that way. When people are threatened when they speak out, they just wont speak in public, but it doesn't change what they believe. Our society is worse off for silencing their voices.

[hga]

No, the idea of being persecuted for what you believe and then in turn, lying in public about it is not a new idea.

However, formalizing the concept this way allowed the further step of the preference cascade, and that, while also not exactly new in practice, was a more enlightening concept.

[internaut]

> What is new is that this sort of environment is taking hold in the US, which is unfortunate. And this forum is as guilty as any other. People lambasted Peter Thiel for speaking his mind and voting his wallet. They demanded his head and spoke of repercussions.

Very true unfortunately. That's not all of HN of course. Many of us supported Thiel and his ideas, or else thought it was worth having an opposing voice around even if they disagreed with it.

I wrote an essay on Peter Thiel's ideas and (partly about) how the Media misunderstood him here:

https://medium.com/@internaut_48577/peter-and-the-wolfe-b8de...

It was composed from a chain of HN comments before the election went to Trump, so I can credibly state that I saw a media malfunction with reality on the cards.

I was watching his press conference from a few days ago at the National Press Club event, and I really did Laugh Out Loud when he said (paraphrasing), "You know, this is pretty much the first time I'm not making a contrarian stance, I mean literally how can I be a contrarian when half the population agree with me, and for this I get the flak? Unbelievable!".

Together with the investment in Seasteading, this has been incredibly high impact for comparatively minor sums of money. Thiel has managed managed to make himself the only venture capitalist in Silicon Valley who appeals to the majority of the working class. That is power. He'll be reaping dividends from that for the rest of his life.

This is because many serious inventions and innovations were created by the working class, not the middle class as they appear to think today. In fact historically (and I would argue currently too) the middle classes and elites looked down on entrepreneurship. Entreprenurship is seen as being unprofessional. Most engineering was created by working class with the aristocrats holding up the exploration of science. The water mills, the steam engines, these were not created by the kind of people who studied at universities. We no longer understand our own past.

This may sound like an attempt at provocation, but I swear it is not. I really do think many of the best innovators and inventors have a working class way of looking at the world.

Let us take something that isn't directly in Silicon Valley's ballpark. Tiny Houses.

All the non-cosmetic innovations I've seen in Tiny Houses have been created by working class people.

The majority of Tiny Houses are created by young aspirational middle class or those who want to retire without worrying. It is especially gratifying to see many young women picking up the hammer to forge something for themselves. I saw one who built a VOC-free house recently and that is impressive if you know anything about building.

Despite this, the only major differences between most builds are cosmetic. People really are afraid of being different despite being adventurous enough to live differently to most. There is a finite amount of adventurousness people have and it is easily exhausted.

The names in italics are Youtube channels if you wanted to inspect what I'm talking about.

Esket Tiny House is built by a carpenter, and he has built a magnificent curved roof when everybody else went with a standard gable roof. If you're perceptive you'll notice that a Tiny House only has 13 ft 6 inches to play with. So logically changing the roof shape would hugely improve the living space in the loft area. Yet they choose not to change it.

Tiny House Customs is another working class Joe (framer). He has cleverly hidden the trailer's wheels and tow bars so it doesn't even appear to be a potentially mobile structure.

Life in a Box (electrician) has introduced the concept of a rain screen, despite being in Arizona! Everybody's builds (in the more rainy regions) are going to have the siding rot off in five years! This is smart.

The Not So Tiny House (power plant operator but definitely working class) has very cleverly hidden his trailer using a faux rock skirt. It now just looks like a regular house on a foundation. It is a rather brilliant solution.

The reason why I mention these is because there are very few working class people building Tiny Houses. I think that is a shame. However the thing to notice here, is that when they do build, they build them very differently to the cookie-cutter approach most middle class people do.

In retrospect, all their innovations are extremely obvious. That is quite curious.

I think this says something pretty profound. There is something about being middle class that could damage your ability to integrate innovations into real life. I realize this is anecdotal but in retrospect I have the feeling I see this pattern everywhere in life.

One explanation could be that: once your comfort increases, your comfort 'zone' decreases. I think this could be a powerful explanation for why so few Silicon Valley companies are truly inventive despite being a Mecca for all kinds of talent and enterprise.

As Peter Thiel points out: all kinds of clever young minds go to Harvard, and despite that they keep electing to join industries just as bubbles are bursting. That is not an accident, it is culture.

tldr; When you think of diversity, maybe think of the working class also. Maybe they have something to add nobody else does.

[Uehreka]

A couple notes on FiveThirtyEight:

First of all, yes, they got it wrong, in that they gave Clinton a higher chance of winning than Trump. They didn't give him a 0 chance, and a model that has 67%ish certainty should be "wrong" about 1 in 3 times. They were better than the other major forecasts, insofar as they gave Trump the highest chance of winning.

I respect Nate Silver for his opposition to non-data-driven media narratives. In 2012, when the media was calling the race super close, Nate called them out for making a media circus when the race was actually looking really good for Obama. In this race, he did the opposite, constantly pointing out how likely a 30% chance was and how big of a polling error there could be (and also how many undecided voters there were). He also called out herding (the tendency of pollsters to not publish outliers) when he saw a ton Clinton+3 and Clinton+4 leads in the week before the election. His forecast may have been wrong, but his instincts are pretty good.

Regarding the shy-Trump effect, we didn't see this play out in the primaries (Trump often underperformed his polls), which makes me skeptical. It might be that with all the reluctant republicans felt a need to vote for him in the general election, the effect got played up. Personally, I think there are 2 big problems in polling as we do it today:

- Not enough pollsters are polling in Spanish. From the polling we do have, we know that Spanish speaking latinos vote very differently from English speaking latinos.

- Our methods for finding people are a mess. Phone polls often use land lines (Cell numbers are often unlisted and may belong to people who live in other states despite their area code) which prejudice them towards the demographic of "people who still have/answer a land line". Internet polls suffer self-selection bias. We need a way to randomly sample people without them making the first move to join a poll. This problem needs to be solved.

Lastly, I want to point out that even Trump's campaign didn't see this coming. No one (except Bill Mitchell's yard sign and Halloween mask model) saw this coming. We have a lot of work to do to fix polling.

[basseq]

  They were better than the other major forecasts, 
  insofar as they gave Trump the highest chance of 
  winning.

You might be interested in the LA Times / USC Daybreak poll, which projected a Trump win for some time. (And similarly, "caused dismay — even outrage — among some readers, especially Democrats, who have denounced it and often criticized The Times for running it".)

http://www.latimes.com/politics/la-na-pol-usc-latimes-poll-2...

[mylesm]

Nate Silver actually wrote a post about the LA Times poll. I don't think it makes sense to compare individual polls with poll aggregates. A single data point being accurate/inaccurate can be due more to luck than methodology, while an inaccuracy in a probabilistic model like Silver's indicates more of a systemic bias across all polls.

http://fivethirtyeight.com/features/election-update-leave-th...

[Declanomous]

I spent 5 years living in Wisconsin, and I've been telling everyone that Trump would definitely win there. The week before the election, I actually went so far as to say that I expected Hillary to win the popular vote, and Trump to win the electoral college.

I wasn't predicting a Trump win out of bias. I'm very liberal, and my friends are too. However, none of my friends believed me. I'm not going to say I was confident in my prediction, but I thought the race was much closer than it appeared. I talk to a lot of people in my job, and I could just tell that a lot of people would be far more okay with Trump as a president than they were willing to overtly say.

I'm not saying I'm some election analysis god, but I felt like people were galvanizing Trump support by refusing to acknowledge Trump as a legitimate candidate, which I think resonated with a lot of people who felt that their concerns were not being taken legitimately. The HRC campaign focused on the negative aspects of his campaign, but his support came from the fact that people felt that Trump cared about their concerns.

[protomyth]

Yeah, I deal with a lot of people all over the spectrum and there was an undercurrent of something going on. A lot of those leaked e-mails really offended people and they got passed around a lot. Its one thing to believe a group looks down on you, but its a whole different thing when you read the e-mails. I was amazed at the number of people that were going to that site and sending links.

It also seemed like the celebrity endorsements had the opposite effect. Some of the people who I've met who spoke ill of the celebrities used language that I don't think they normally use. It has been quite odd.

[MaxfordAndSons]

I agree with your analysis, but it's sort of an orthogonal point to the issue at hand. The issue is that the personalized media bubble effect prevented more people from attaining your insight, because we were simply not exposed to the same media as those who were galvanized by Trump, and on the other side of the coin, it allowed Trump supporters to avoid exposure to valid critiques of their candidate and instead be hyper focused on (sometimes fabricated, often exaggerated) content that was critical of Clinton.

[Declanomous]

That's fair. I also go out of my way to seek out people's thoughts on divisive but relatively popular topics, because people have some reason for believing what they believe. It probably helps that I don't spend much time on Facebook because I think most people who share my ideology are at least as closed-minded as people whose ideology conflicts with mine.

That being said, I saw a lot of arguments on Facebook about Trump. I don't think the echo chamber is as complete as claimed. I think a lot of the issue is not that social media provides an echo chamber, but that social mores prevent people from voicing controversial views in large groups of people who disagree with you. So, assuming the people in your friends list are from a similar socioeconomic background, a substantial majority of them probably support one candidate over the other. So the only posts you'll see about the other person are from the least "well behaved" members of your social circle.

I think the larger issue is that the quality of the conversation on social networks is just terrible. A five-minute conversation with a stranger has more depth to it than a status with 100 comments on it. Online discussions are basically won by whichever side's memes appeal to the broader audience.

[woodchuck64]

Makes sense. Now, how can Silver correct his methodology to capture the above in future elections, I wonder.

[chaosbutters314]

I think his methodology was fine and pollsters were shit. If you look carefully at the map, he basically called all the swing states hilary and trump took from one another. His model saw the holes in the pollsters data. I think to swing trump to winnning, he'd probably just have to adjust a weighting factor by fractions of a percent. Look at how close michigan, florida and wisconsin were. The tiniest of margins won it for trump in the end. Literally a handful of factories that had closed in the past decade between those states determined this race.

The scarier thing is that for the next 10-20 years, boomers in the rust belt are going to swing republican unless dems can get a Bernie. To me, whatever democrat wins the michigan primary, is their candidate above all else.

[maxxxxx]

He should call it quits together with all other pollsters. All that predicting is at best a totally useless activity or even worse, it influences voter behavior. Bet on football games instead. At least that doesn't do damage to the country.

[551199]

The main point of polls is to influence and give social proof to selected candidate.

Don't believe me? When you look how they are constructed it becomes clear. Too bad for Clinton campaign she was highly unlikable and even the most expensive campaign in history couldn't help her.

[maxxxxx]

Agreed. All polling and media was totally skewed against Sanders (and then Trump).

[Jemaclus]

Technically, they did predict it: 538 had 35% chance of Trump winning and emphasized the uncertainty factor, while other outlets had Clinton's chances as high as 99%.

[alva]

Just like they predicted Trump's chance of winning the primary as 2%.

Hilarious defence of their work in recent tweet

"Clinton came within 2 points of 307 electoral votes, in which case polls would have been right in 49 of 50 states. "

ie. If the results had been different, our prediction would have been correct.

[zzalpha]

If the results had been different, our prediction would have been correct.

First, that comment is about the polls, not their model.

Second, it's called a polling error. It's actually a thing.

This election happened to be extremely sensitive to polling errors due to the nature of the electoral college and the fact that a number of key races were extremely tight.

When an average poll has a 3% margin of error, and you factor in state-level correlations (and the fact that the state level polls tend to have more variable quality than national polls), a slight error results in a huge swing.

Which is the entire point they're making, if you're willing to set aside the snark and read what they wrote (since it's actually, you know, interesting and informative).

[mplewis]

Sounds like you didn't read the part where they reviewed their research and looked at where they went wrong.

http://fivethirtyeight.com/features/how-i-acted-like-a-pundi...

[jtsnow]

Nassim Taleb has rigorously shown that FiveThirtyEight did not accurately account for uncertainty in their models.

[xapata]

How does one accurately account for uncertainty?

If you're talking about Taleb's "black swan" style uncertainty, that's specifically the kind that cannot be taken into account.

If you're talking about events being correlated instead of independent, that's what set 538 apart from other poll-aggregators: Nate gave Trump a relatively high chance of winning, because a polling error in one state would likely indicate the same polling error in every other state.

[jtsnow]

As I understand it, FiveThirtyEight's models reflect current and past uncertainty, but do not properly take into account future uncertainty. This greatly undermine's their models for use as a prediction, a probability, or a forecast.

At one point, FiveThirtyEight was forecasting/predicting that Clinton had a 66% chance of winning the election. If there were infinite parallel universes that veered off from that moment in time, their implication is that she would win the election in two out of three times. Taleb is saying that is nonsense, that there is more variability than that.

Of course, Taleb's model wouldn't drive page views or get people excited. FiveThirtyEight gives voters a reason to check-in daily on how their candidate is doing in polls, but it is misleading as a forecasting tool.

[xapata]

Eh? Taleb was criticizing the variability of 538's estimates and said they should not have been making such drastic changes to their estimates. Taleb waffles between Bayesian and Frequentist and in this case he's being Bayesian.

He didn't criticize them for underestimating uncertainty, but for being too uncertain.

[DubiousPusher]

Donald Trump barely won. They regularly gave him a %25+ chance of winning and regularly pointed out that that was a good chance.

[pfortuny]

The thing is, these surprising results tend to happen the first (and only) time... which is rather unforgiving with the percentages.

[AznHisoka]

he won 2 extra states he had no business winning. thats not barely winning.

[DubiousPusher]

In Michigan, a state where 41/2 million votes were cast, he won by ~10,000 votes. That's nearly the definition of barely winning.

[rokosbasilisk]

except he did it multiple times in the rust belt.

[maverick_iceman]

https://en.wikipedia.org/wiki/Shy_Tory_Factor

[protomyth]

Particularly nasty this time around with the amount of name calling and threats of violence[1]. I'm pretty sure any tech person who was for Trump shut the heck up after the whole Peter Thiel narrative. The amount of people who would rather act out than understand what is going on is a bit mind blowing.

Of course, this is just a weird year when a third-party had a shot at the 5% and all of them sabotaged themselves by picking people who were uniquely unlikely to get votes. Could have been the setup for a 2020 three party run.

1) you would have to be insane to run a business and have a Trump sign in your window unless you were hoping for an insurance settlement.

[pluma]

I'm not in the US and I was a bit taken aback when people on my Twitter timeline started calling for boycotts of a certain online sticker service entirely because the CEO came out on Twitter as supporting Trump a few weeks ago.

I've only seen this form of complete isolation as part of feminist "no platform"-ing (basically: actively excluding speakers because of their political views, regardless of the topic of the conference or occasion) before. But this time it wasn't simply about fringe extremists (e.g. racial supremacists) but about all supporters of the final candidate of one of the two major parties.

I hope this is the end of this practice rather than the start of something worse. Demonising half (or a third, depending on how you measure) the population is not how you fix social issues, especially if it desensitises people to slurs you will still need to label the real extremists.

[smogcutter]

It's easy to underestimate how deeply polarized the US really is.

Trump in particular is so far out of the bounds of normal politics that to many people, support for Trump appears no different than support for, say, Mussolini. It may only be fringe extremists playing the racial supremacy song, but half the country is marching along to the beat. At that point it doesn't matter what your private reasons for supporting a Mussolini or Franco or Trump are - you're still a collaborator. The justification for boycotts etc comes from that point of view.

It's a difficult problem - the left (such as it is) needs to find some kind of productive path forward, but I'm not convinced that pandering to the sensitivities of people who have no intention to reciprocate is at all useful.

[jtsnow]

> proven, reality-based statistics

Nassim Taleb has been calling BS on FiveThirtyEight's methods and models for months: https://twitter.com/nntaleb/status/762032883414556674

He's now gone as far as calling FiveThirtyEight "clueless" and posted further comparisons of his model with theirs: https://dl.dropboxusercontent.com/u/50282823/binary%20foreca...

[xapata]

Did you glance at his charts? He's saying a "rigorous updater" would have estimated Trump's chances at almost nothing all year long, until dramatically changing the estimate just before the conclusion. In contrast, 538 estimated Trump's chances as much higher all year.

He has a point that the chances of winning shouldn't shift so rapidly, but a rapidly moving gauge is a decent indicator of uncertainty.

[renaudg]

At issue in this election is apparently a new idea: people who hold positions that are being publically shamed tend to lie to pollsters

Definitely not that new. For example, pollsters in France have for decades been applying a "coefficient of shame" to account for Le Pen voters' underreporting.

[harveywi]

> proven, reality-based statistics

Ex falso quod libet. The statistical mechanics can be rock solid, but if the data is garbage or the model is garbage then the prediction will likely be garbage. Hiding behind mechanical minutiae is no excuse for the overtly bold and wrong predictions.

[pfortuny]

So, why did they not call the data garbage instead of giving us a number?

As with credit agencies... "We do not give advice on investments", yeah but everybody invests based on your ratings.

[woodchuck64]

> if the data is garbage or the model is garbage

There's no better reality-based approach than aggregating polls across the political spectrum. But who would have thought that all polls would be wrong in the same direction?

[Pigo]

Is it uncommon that I can't ever recall being asked who I'd likely vote for? I suppose there's been a website once or twice that I've clicked a button on, but no one has ever asked me, or called me, or even handed me a survey that I can think of. What is the most common to actually make the inquiries.

[neves]

And you probably don't know anyone who has been asked. This is the power of statistics.

[pmoriarty]

"people who hold positions that are being publically shamed tend to lie to pollsters"

Alternate theory: People did not lie to the pollsters -- but there was voting fraud, and Trump didn't win.

[pluma]

After HRC supporters have gone to such great lengths to explain how Trump's warnings about fraud were delusional and unrealistic, I would really love to hear a justification why Trump could have pulled this off when supposed Clinton couldn't have.

[pmoriarty]

You know, after the debacle that was the Bush vs Kerry election, there were widespread allegations by Democrats of voting fraud on the part of the Republicans -- especially in respect to electronic voting machines. For some months after the election, the Democrats made a lot of noise about those electronic voting machines, trying to get them removed. After a few months, they apparently seemed to forget all about it.

When the next election rolled around, when Obama won, they didn't make a peep about the electronic voting machines, which were still in use, nor about the possibility of voting fraud. Neither did the Republicans. Nor did anyone mention these issues in the following election, when Obama won again. It was only Trump speaking up about it which brought the issue back in to the news. But the possibility of massive voting fraud has been there all along, and will continue to be there for the following elections, as long as electronic voting machines are still in use.

Now, why does the mainstream of both the Democratic and Republican parties permit these machines to continue to be used? That's a very good question. The consipracy minded might speculate that they might have an agreement between them, and that US elections have become simply political theater, with the real decisions as to who gets to hold office being made through other means, and by people other than the voters.

[logfromblammo]

Of course there was fraud. We know better than most how easily a box can get pwned when the attacker has unsupervised physical access for a few minutes, and the manufacturer gives short shrift to security and practices security through obscurity--such as by keeping the source tightly closed.

The question is whether Trump supporters were better at cheating than Clinton supporters, or better at covering their tracks. Even if nobody suspected fraud, we should still be doing randomized checks for it, in a transparent, publicly-auditable way.

My hypothesis is that voting fraud is endemic, but that it only rarely influences the results of any election. I suspect that most of it is done by crackers for hire or by people already heavily invested in politics who happen to have the necessary skills.

It just seems like such a simple, boring, and yet insanely high-risk hack that no one would bother doing it just for the giggles of getting Ivanna Tinkle elected as county dogcatcher. Also, in order to pull off a significant advantage, you would need a conspiracy of multiple actors in several different counties, and the more participants you have, the less likely it is you will be able to keep it secret.

Tyler Durden (of Fight Club) could silently steal an election with a vote fraud conspiracy, but no actual, living person could--not until all the votes are cast on network-connected machines, anyway.

[pmoriarty]

"no actual, living person could"

It doesn't have to be one person. It could be multiple teams of people.

They don't have to intervene all over the country, either. They could make a huge difference by affecting a relatively small number of votes in a handful of battleground states where the race is very close.

In the case of Bush v Gore, it would have been sufficient to make a difference in only one state.

[logfromblammo]

Tyler Durden had multiple Fight Clubs, all of whom were completely loyal, anonymous, and committed to operational security. That's why I said he'd be the only one able to do it.

For real people, the more people you add for operations, the more you have to add for security and cleanup. Beyond a certain point, you just can't keep everybody quiet without extreme measures, which are themselves likely to be noticed.

Secret conspiracies have to be small, otherwise someone eventually gets disgruntled or has an attack of conscience and spills the beans.

Even large, public conspiracies, like the classified documents protection system, eventually develop leaks, and it is already really expensive to operate before accounting for cleanup up after spies or whistleblowers.

So if we ever reach the point where five people or less could remotely rig the vote for every county in Florida, I have to not only assume that it is being done every election, but also that multiple groups may be stepping on each other's toes while doing it. Previously, the traditional ways of rigging the vote are right out in the open. You get people likely to vote against you stricken from the voter register. You enter fake ballots in the name of someone not likely to vote, such as the recently deceased. You sabotage polling places in urban areas such that voter throughput is reduced, and lines grow around the block. You get local cops out on the streets, giving out traffic citations to selected people that may be on their way to vote. I have even seen sudden construction activity on election day obstructing the sole entrance to a polling place.

Those are nasty, but at least people can seek redress for the misbehavior that they can see.

[551199]

FiveThirtyEight and Nate Silver's predictions have been hilariously wrong about Trump in every way possible.

Reading this honestly makes me question why I read HN.

[zzalpha]

Well, for me, it was usually because of the high quality commentary, wherein folks had rational discussions about the content of a piece rather than resorting to Reddit-style content-free snark that neither educates nor illuminates.

Funny how times change, eh?

[551199]

All the dissident views about polls being outright wrong were met with disbelief when it was clear to anyone who looked that they were blatantly off. Mainly to create social proof to vote Clinton.

[woodchuck64]

What reality-based approach to predicting Trump's win would you have used? Note; anecdotes are not a reliable approach.

[551199]

How about one that isn't used only to influence voters?

Read Talebs tweets:

https://mishtalk.com/2016/08/07/nassim-telab-blasts-nate-sil...

https://twitter.com/nntaleb/status/794608806072352769