Saturday, November 12, 2016

Pols and Polls - The good, the bad, and the ugly

John Cassidy (New Yorker) frames the big question: How did the polls, and the press, get this election so wrong? Let’s start with Cassidy’s takes.

Since Tuesday night, there has been a lot of handwringing about how the media, with all its fancy analytics, failed to foresee Donald Trump’s victory. The Times alone has published three articles on this theme, one of which ran under the headline “How Data Failed Us in Calling an Election.” On social media, Trump supporters have been mercilessly haranguing the press for getting it wrong.

Clearly, this was a real issue. It’s safe to say that most journalists, myself included, were surprised by Tuesday’s outcome. That fact should be acknowledged. But journalists weren’t the only ones who were shocked. As late as Tuesday evening, even a senior adviser to Trump was telling the press that “it will take a miracle for us to win.”

It also shouldn’t be forgotten that, in terms of the popular vote, Clinton didn’t lose on Tuesday. As of 6:30 p.m. Eastern Time on Friday, a tally by CNN showed that Hillary Clinton had received 60,617,062 votes, while Trump got 60,118,567. The margin in her favor—now at 498,495—is likely to grow as the remaining votes are counted in California. At the end of the day, Clinton may end up ahead by two per cent of the total votes cast. If the United States had a direct system of voting, Clinton would have been the one at the White House on Thursday meeting with President Obama. But, of course, Trump won the Electoral College. If the final count in Michigan remains in his favor, Trump will end up with three hundred and six Electoral College votes, to Clinton’s two hundred and twenty-six.

Hand-wringing about the electoral college is not likely to be productive. The system is enshrined in the constitution (see the Wiki entry, for example) and it has resisted change over the years.

To the extent that there was a failure, it was a failure of analysis, rather than of observation and reporting. And when you talk about how the media analyzed this election, you can’t avoid the polls, the forecasting models, and the organizing frames—particularly demographics—that people used to interpret the incoming data.

Some analysts did suggest that there might be. Immediately after the 2012 election, Sean Trende, of Real Clear Politics, pointed out that one of the main reasons for Mitt Romney’s defeat was that millions of white voters stayed home. Earlier this year, during the Republican primaries, Trende returned to the same theme, writing, “The candidate who actually fits the profile of a ‘missing white voter’ candidate is Donald Trump.”

The Times’ Nate Cohn was another who took Trump’s strategy seriously. In June, pointing to a new analysis of Census Bureau data and voter-registration files, Cohn wrote, “a growing body of evidence suggests that there is still a path, albeit a narrow one, for Mr. Trump to win without gains among nonwhite voters.” As recently as Sunday, Cohn repeated this point, noting that Trump’s “strength among the white working class gives him a real chance at victory, a possibility that many discounted as recently as the summer.”

There was a straightforward reason for all the skepticism about Trump’s chances: when you looked at the state-level polling, it looked like Clinton’s “blue wall” was holding. Take Wisconsin, which turned out to be a state that Trump won. The Huffington Post’s polling database lists the results of more than thirty polls that were taken in the Badger State since June: Trump didn’t lead in any of them. Three of the final four surveys showed Clinton ahead by six points or more, and the Huffpollster poll average put her lead at 6.3 percentage points. Trump carried the state by one point. In other key states, the pattern was similar. The final Huffington Post poll averages showed Trump losing by nearly six points in Michigan, and by four points in Pennsylvania.

… a lot of Trump voters refused to answer the pollsters’ calls in the first place, because they regarded them as part of the same media-political establishment that Trump was out railing against on the campaign trail. Something like this appears to have happened in Britain earlier this year, during the run-up to the Brexit referendum. Turnout wound up being considerably higher than expected among lower-income voters in the north of England, particularly elderly ones, and that swung the result.

The problem with models is that they rely so much on the polls. Essentially, they aggregate poll numbers and use some simulation software to covert them into unidimensional probabilistic forecasts. The details are complicated, and each model is different, but the bottom line is straightforward: when the polls are fairly accurate—as they were in 2008 and 2012—the models look good. When the polls are off, so are the models.

And the polls were off. Here are selected snippets from Nate Silver’s email explaining why that was so. Basically the “missing white voter” theory is correct, Scriber thinks.

Undecideds and late deciders broke for Trump

The single most important reason that our model gave Trump a better chance than others is because of our assumption that polling errors are correlated. No matter how many polls you have in a state, it’s often the case that all or most of them miss in the same direction. Furthermore, if the polls miss in one direction in one state, they often also miss in the same direction in other states, especially if those states are similar demographically.

There were some other factors too, however, that helped Trump’s chances in our forecast. One is that our model considers the number of undecided and third-party voters when evaluating the uncertainty in the race. There were far more of these voters than in recent, past elections: About 12 percent of the electorate wasn’t committed to either Trump or Clinton in final national polls, as compared with just 3 percent in 2012. That’s a big part of the reason our model was quite confident about Obama’s chances in 2012, but not all that confident about Clinton’s chances this year.

Indeed, late-deciding voters broke toward Trump, according to exit polls of most swing states. Or at least, that was the case in states where Trump outperformed his polls, such as in Pennsylvania and Wisconsin. …

Pollsters simply can’t do much about voters who make up their minds only after the survey is completed. (And making inferences is a guessing game: It’s sometimes said that undecideds tend to break to the challenger, but the empirical evidence on this is mixed. Obama won late-deciding votes in 2012, for example.) But modelers can do something about it, by allowing for more uncertainty in the forecast when there are more undecideds. If only 3 percent of the electorate is undecided, then winning undecideds 3–2 — as Trump did in several swing states — will shift the overall outcome by less than 1 percentage point. But if 12 percent of the electorate is undecided, winning them by that ratio will produce a net swing of 2 to 3 points toward a candidate, potentially letting him overtake the front-runner.

A failure of conventional wisdom more than a failure of polling

… We strongly disagree with the idea that there was a massive polling error. Instead, there was a modest polling error, well in line with historical polling errors, but even a modest error was enough to provide for plenty of paths to victory for Trump. We think people should have been better prepared for it. There was widespread complacency about Clinton’s chances in a way that wasn’t justified by a careful analysis of the data and the uncertainties surrounding it.

No comments:

Post a Comment