Why Polls Fail: Attitude, Intention, Behavior

Since it’s election day, let’s discuss the polls. Specifically, why their track record has been so poor. I love this topic because it closely tracks prediction errors in market and economic forecasts.

In 2024, I discussed a major source of polling error: the pools pollsters use. Today, I want to discuss a more challenging failure: what is termed “respondent error,” but you are probably more familiar with as Human psychology.

The key respondent errors are Attitude, Intention, and Behavior. These three phenomena affect our ability to predict our own future decision-making and/or behavior.

I have made various arguments over the past twenty years; rather than reiterate these, consider three questions that confront pollsters:

1. What do people really believe?

2. How well do they know their own actual intentions?

3. How likely are they to act on those?

That is the challenging psychology of polling voters, and it has plenty of interesting wrinkles.

Note that I am not addressing whether pollsters reach a representative audience (they don’t) or whether they use unbiased polling questions (it varies). Instead, I suggest that the better pollsters understand the psychology, the more accurate their numbers will be.

Let’s discuss a few details.

1. What do people really believe?

A surprisingly challenging question. We assume people know what they believe; after all, we know our religion, partisan affiliation, favorite sports teams, hobbies, etc. But with so many complex, emotional, and conflicted issues, what people say and what they truly believe often don’t match.

We also assume they answer honestly; ask who they think will win, and you get a biased response based on who they prefer.

This tension creates problems for pollsters. People hold “non-attitudes” on many (most?) issues — ask about a policy they’ve never considered, and they’ll create a response on the spot that’s unstable.1  Answers can reflect weak opinions, conflicting values, and are impacted by the way a question is presented.

Ambivalent voters who like one candidate’s personality and the other’s policies will respond to pollster questions with a squishy answer they’ll likely revise. This is especially problematic for issue polling.2

Then there is expressive answering: some respondents use a poll to send a message rather than report voting preferences.

2. How well do they know their own actual intentions?

Humans declare their intentions to do a specific act with high confidence. After all, we wouldn’t say it if we didn’t think we’d do it (would we?).

What tends to happen is that we say what we either want to believe or what we think the listener wants to hear. This again creates a gap between stated intentions and future behavior. Expressing support costs the poll respondent nothing; their commitments to those responses may be conditional or aspirational.

Then there is the bandwagon effect: Some state their preference based on who they perceive will win. We don’t want to back a loser, and we often claim support for who we expect will win. 

3. How likely are we to act on these?

This is the big one: Self-reported turnout intention overstates actual turnout by a large margin. Many people say they’ll vote, but don’t.

Pollsters try to model this from election to election with various degrees of success. But turnout shifts in ways past models can’t anticipate. The 2022 midterm “red wave” never materialized; 2024 showed the reverse. The low-propensity voters who show up only for a specific candidate (e.g., a Trump-only voter) are exactly the people a likely-voter model often excludes. The results of these model adjustments have been decidedly mixed.

People can state what they want to do, and then the world intervenes: bad weather on Election Day; breaking news that changes how we perceive one candidate or the other; geopolitics, the economy, markets, and inflation. The electorate consists of people who vote but whose composition is uncertain beforehand. Voting history and other evidence make turnout only partly predictable.

~~~

Pollsters mistakenly believe their core work is data analytics; history suggests it’s more similar to the work psychologists do when trying to predict future behavior. Failing to discern that distinction is why their track record ranges from not bad (2024) to abysmal (2016, 2020).

Endless things can happen between when a pollster gets an answer and when it’s time to vote. A respondent to a poll who says a) I like this candidate; b) I am a registered voter; and c) I intend to vote for that person — does not guarantee a ballot will be cast as stated, if at all.

We know that the earlier the polls are, the worse their predictive values are; we also know that all models are wrong, but some are useful. Models also presume the future looks like the past, and during periods of disruption, that assumption tends to be problematic.

In a nation where partisan lines are fairly well drawn, the middle 5-10% will decide many elections. Those independent undecideds are often equal to the margin of error.3  No wonder polling has had such a mixed track record.

Get out and vote!

See also:
The Undecided Voters Are Not Who You Think They Are
For most, the big decision is about whether to vote at all.
By Ronald Brownstein
The Atlantic, September 26, 2024

The Polling Imperilment
Presidential polls are no more reliable than they were a century ago. So why do they consume our political lives?
by Rick Perlstein
American Prospect, September 25, 2024

Previously:
Bad Polling is a Behavioral Problem (October 6, 2024)

Finding Investing Lessons in 2020 Election Fails (December 10, 2020)

Polls Were Wrong (again). They Often Are (November 4, 2020)

What Models Don’t Know (May 6, 2020)

10 Points About Forecasting and Why We Suck at It (November 1, 2017)

Forecasting & Prediction archive

Prediction Markets archive

__________

1. Things Are Rarely Black and White: Admitting Gray Into the Converse Model of Attitude Stability
Charles J. Brody
American Journal of Sociology, Volume 92, Number 3 1 November 1986
DOI: https://doi.org/10.1086/228545

2. Issue polling typically generates far worse results than election polling…

3. Who casts their vote? A third to half of eligible voters don’t. This is also why a 2-3% margin of error is laughably wrong—it’s much closer to a 6-8% margin of error.

DISCLOSURE: I wrote this myself; I did research with Claude and Google. The graphic at the top was created by ChatGPT

The post Why Polls Fail: Attitude, Intention, Behavior appeared first on The Big Picture.