Millions of Lives, a Handful of Data Points: Five Small Samples That Shook America

We like to imagine that big decisions rest on big evidence. History disagrees — repeatedly. Some of the most consequential calls in American life were made on samples so small you could fit the entire dataset in a shoebox. Here are five, each with a lesson the data world had to learn the hard way.

1. The Thalidomide Near-Miss

In the early 1960s, a sedative called thalidomide was widely prescribed across Europe — often to pregnant women dealing with morning sickness. The evidence for its safety in pregnancy rested on remarkably thin data; the drug had never been systematically tested in expectant mothers.

The story could have ended in tragedy for American families too. It didn't, largely because of one FDA reviewer named Frances Kelsey, who refused to approve the drug, citing insufficient evidence of safety. Her skepticism — essentially a demand for better data — was vindicated when thalidomide was linked to thousands of severe birth defects worldwide.

The lesson: Absence of evidence isn't evidence of safety. When nobody has looked, the sample size is effectively zero — and zero is the most dangerous number of all. Kelsey's stand helped push through stronger drug-approval requirements, transforming how evidence gets weighed in the U.S. to this day.

2. The 1936 Literary Digest Poll: A Million Data Points, All the Wrong Ones

Here's a twist: sometimes the problem isn't too little data — it's the wrong data. The Literary Digest mailed out around ten million ballots before the 1936 presidential election and collected 2.4 million responses, confidently predicting Alf Landon would crush Franklin D. Roosevelt.

Roosevelt won 46 of 48 states.

The Digest's sample was enormous but badly skewed — drawn largely from telephone directories and automobile registration lists during the Depression, when owning a phone or car meant you were comfortably well-off. George Gallup, meanwhile, predicted the outcome with a much smaller but carefully balanced sample of a few thousand.

The lesson: A biased sample doesn't get honest by growing. Two million skewed responses will lead you further from the truth than two thousand representative ones. Quantity can't launder quality.

3. The Poll That Called Truman's Loss Before It Happened

The 1948 election produced one of the most famous wrong calls in American journalism: "DEWEY DEFEATS TRUMAN," splashed across the Chicago Daily Tribune front page. Several major pollsters had stopped surveying weeks before Election Day, confident the race was over.

But their late-summer samples froze in time. Undecided voters broke for Truman, farmers shifted late, and the polls — built on earlier snapshots — never saw it coming.

The lesson: A sample is a photograph, not a movie. It captures a moment, and moments expire. Modern pollsters weight, track, and re-sample constantly, because yesterday's data answers yesterday's question.

4. The Pill Studies That Moved Too Fast

The history of American medicine includes a rogues' gallery of treatments blessed on the strength of a handful of patients — and later reversed. Early bloodletting-era reasoning aside, even 20th-century practice saw therapies adopted after small, uncontrolled series of cases, only for rigorous randomized trials to reveal they didn't work, or worse, caused harm. The multi-decade effort to require randomized controlled trials with pre-planned sample sizes exists precisely because expert opinion built on a dozen cases kept embarrassing itself.

The lesson: Anecdotes aren't data, and a dozen anecdotes aren't a trial. The human brain is a pattern-finding machine that will find patterns in pure noise — which is why we invented the control group: to give the noise a chance to announce itself.

5. The Intelligence Estimate Problem

American intelligence analysis has had its own reckoning with thin data. In the lead-up to the 2003 Iraq war, assessments about weapons programs rested partly on a small number of sources — later shown to be unreliable — that were treated as more conclusive than they deserved. Post-mortem reviews, including work by the commission that examined the intelligence failures, pointed to over-reliance on limited, uncorroborated reporting.

The lesson: A single source repeated often starts to feel like many sources. Statisticians call the fix independent verification: evidence only counts when it comes from separate, unconnected streams. Corroboration by repetition isn't corroboration at all.

What Ties These Stories Together

Notice the pattern. In every case, the failure wasn't stupidity — it was the very human tendency to treat a small, convenient, or flattering dataset as if it were the whole truth. The antidote is the core idea behind this site: n equals many. Before trusting a conclusion, ask how many observations stand behind it, where they came from, and whether they'd still hold up if you gathered them all over again.

Small samples built some of history's biggest mistakes. The next big one is avoidable — if we remember to count before we conclude.