Two Million Votes, Zero Clue: The Polling Disasters That Changed America
Photo: State Library of Queensland, Australia, No restrictions, via Wikimedia Commons
If you wanted to predict the 1936 presidential election, you could hardly ask more Americans. The Literary Digest, a fat weekly magazine with a reputation for calling elections correctly, mailed out roughly ten million ballots. Over two million came back. Alfred Landon, the Republican challenger, led Franklin Roosevelt by a comfortable 57 to 43 percent in the returns.
Roosevelt won the actual election with 60.8 percent of the vote — one of the biggest landslides in American history. The Digest, having polled more people than anyone before or since, had missed by nearly twenty points. Within two years the magazine was gone.
The lesson, which America keeps having to relearn, is that sample size is not the same thing as sample quality. Here are five moments when the country asked the wrong people — and what each one teaches about the sneaky ways surveys fail.
1. The Literary Digest and the Price of a Stamp (Coverage Error)
How do you get ten million addresses in 1936? You don't knock on doors. You buy lists — of your own subscribers, automobile registrations, and telephone directories.
The catch: it was the Depression, and cars and phones weren't evenly distributed. A family with a telephone was meaningfully wealthier than one without, and wealthier Americans leaned Republican. The Digest had accidentally built a poll of the affluent and called it a poll of America. The fancy name is coverage error: when the list of people you can ask doesn't match the list of people you need to ask.
Meanwhile, a young pollster named George Gallup surveyed a few thousand carefully chosen respondents using quota sampling and correctly called the landslide. His sample was roughly a thousand times smaller. It was just aimed better.
2. The 1948 Polls That Stopped Trying (Nonresponse Bias)
Fast forward to 1948, when the major pollsters — including Gallup and Elmo Roper — agreed the Thomas Dewey–Harry Truman race was basically over. The Chicago Daily Tribune printed its infamous "DEWEY DEFEATS TRUMAN" front page. Truman held up the paper with a grin; the photo became the most famous correction in journalism.
The polls had quit too early, and the people they did reach didn't answer honestly about turnout. Here's the mechanism: surveys are answered by the people willing to answer. In 1948, the respondents who had actually voted in past elections skewed toward Republicans, while Truman's working-class coalition was underrepresented among the cooperative. That's nonresponse bias — the error isn't in who you ask, it's in who bothers to answer. Truman's voters were harder to reach and less interested in talking to pollsters, and the numbers quietly bent around that absence.
3. The 1984 and 1988 'Shy Tory' Cousin (Social Desirability)
This one isn't a single American election but a recurring American problem with a British nickname. In the UK, pollsters noticed voters hesitating to admit support for Margaret Thatcher, so they lied to pollsters — the "Shy Tory effect." The American cousin shows up whenever a candidate is socially awkward to support: people tell a pollster one thing and the voting booth another.
The 2016 pre-election polls offer a modern example. National polling was actually decent — the popular vote margin landed close to the final averages. But state-level polls, particularly in the Midwest, systematically underweighted the voters who ultimately tipped the Electoral College. Some of that was weighting; some was that certain respondents were simply absent from the sample in the first place.
4. Weighting Gone Wrong: Fixing One Error, Causing Another
Modern pollsters don't just report raw numbers. They weight — adjusting the sample so it matches the population's known demographics. Too many college graduates? Turn their answers down. Not enough rural respondents? Turn them up.
Done well, weighting is the quiet hero of survey research. Done badly, it's a Rube Goldberg machine of guesswork. The 2020 election cycle produced the widest polling errors in about forty years, especially in states like Wisconsin and Iowa, and post-election analysis pointed partly at weighting choices: many polls over-corrected for education after 2016, assuming college graduates were more Democratic than they turned out to be in key states. The pollsters fixed last cycle's error and, in some states, manufactured this cycle's.
The uncomfortable truth is that weighting requires knowing how the country will behave — which is the very thing you're trying to measure.
5. The Mode Effect: How You Ask Changes the Answer
The most modern failure mode is also the least intuitive: the way you ask changes what people say. Live phone calls, automated robocalls, online panels, and text surveys all produce systematically different results from the same population — a problem called mode effect.
People answer a live human differently than they answer a robocall. Online panels reach the digitally comfortable. And as fewer Americans answer unknown phone numbers at all, pollsters lean harder on the people who do — often older, more settled, more likely to pick up. Every survey is now a negotiation with who's reachable, and the negotiation leaves fingerprints on the results.
The Moral of the List
Notice the pattern. In none of these disasters did anyone need more respondents. The Digest had millions. The 1948 pollsters had decades of track record. The 2020 pollsters had the most sophisticated weighting software ever built. What they lacked was a representative sample — a genuine slice of America rather than a distorted one.
A big sample of the wrong people isn't reassurance. It's just a bigger, more confident mistake. Which is worth remembering the next time a poll crosses your feed with a margin of error of plus or minus three points: the margin of error only counts the error we can measure. The Literary Digest's 2.4 million respondents came with a margin of error too. It just didn't say so.
