Thinking About Likely Voters...

Dear Readers,

The October 2024 Franklin & Marshall College Poll, conducted October 9 – 20 and released on October 24^th, was the first time this election cycle that the poll included data about “likely voters.” That decision invited many comments and questions. In today's newsletter I address two of the most common questions we received.

Thank you for reading,

Berwood Yost

How do the October likely voters compare to likely voters in earlier surveys?

The likely voter model shows something different for each race. The likely voter results were much more favorable for Senator Casey in August and September than in October—he had a sizable lead in both prior surveys that has largely evaporated (Table 1). In the senate race, the likely voter (LV) estimates were only two-points different from the registered voter (RV) estimates in August (+11 RV and +9 LV for Casey) and September (+8 RV and +10 LV for Casey), but there was a six-point difference in October (+7 RV and +1 LV for Casey).

The likely voter model in the presidential race suggested a tie in August, but the September likely voter model data gave Vice President Harris a solid advantage that was even better than the advantage we saw for her among all registered voters (+3 RV and +7 LV for Harris). That four-point swing in September toward Harris was large and it is mirrored by the five-point swing among LVs toward Trump in October (+4 RV and -1 LV for Harris).

Table 1. F&M Poll Likely Voter Estimates, Pennsylvania 2024

Some of the September poll’s data was gathered after the presidential debate and it seems that Harris benefitted from her performance, at least for a short time. Do the October numbers represent some kind of Trump advantage due to the generally positive news his campaign received while the poll was in the field? We don’t know. However, it does suggest that voters are clearly not penalizing former President Trump for skipping another debate, inexplicably, and these results validate his decision not to re-engage with his opponent on the debate stage.

I think your likely voters screen stinks, why do you use it?

The other predominant question was about how we identified likely voters. The question below captures concerns about our approach from someone who closely follows and scrutinizes polling data:

Your recent poll of Pennsylvania, showing Harris up 4 among registered voters and Trump up 1 among likely voters, is an absolute disgrace. The likely voter filter you applied (where registered voters were counted as likely only if they said they were "certain" to vote AND claimed to be "very interested" in the election) is hilariously flawed. Merely asking voters if they're certain or probable to vote is good enough (and you can also ask about voting history to make it a bit more robust). You do not need to further filter based on a purely arbitrary notion like level of interest.

It's highly unusual for a poll to produce a 5-point swing from the registered to the likely voter pool, and almost always when it happens, it's because the pollster applied a flawed likely voter screen (the recent TIPP poll of Pennsylvania is also a great example, as they squashed almost the entire Philadelphia sample for no good reason whatsoever, and therefore showed Trump up 1 among likely voters after having him down 4 among registered voters). This poll is not based on sound methodology. It's based on a pre-existing desire to show a tight horse race, probably because you're afraid of getting it wrong. Shame on you.

The F&M Poll has used the same likely voter screen for more than two decades. It has provided estimates of voter turnout comparable to the share of registered voters who actually voted, which is why we continue to use it (Table 2). These turnout estimates have improved since the F&M Poll started sampling from lists of registered voters in 2012. The survey literature is clear that asking only about intention to vote significantly overestimates the number of likely voters, which is obvious from the share of “certain” voters in every poll.

Table 2. Estimates of Registered Voter Turnout, F&M Polls 2004 - 2024 

Note: 2004 - 2008 used RDD telephone samples; 2010 used RDD plus cell samples; 2012 - present used registered voter list samples

Still, there are many ways to create a likely voter screen and there is no generally agreed upon best practice among pollsters. Instead of allowing people to self-identify their intentions, another strategy is to use a screen based on past voting. For example, if we identified likely voters as those who participated in three or more of the past five general elections or those who've newly registered in the past year, the results for a likely voter model based on past voting behaviors would be:

Harris 49% - Trump 49%

Casey 50% - McCormick 46%

The likely voter model based on voting history is not too different from the self-reported model the F&M Poll uses, except that it produces a turnout estimate of 68%, which is probably a bit low.

Why try to describe likely voters at all? Analyses of pre-election polling find that identifying which registered voters will actually vote is important for improving poll accuracy, particularly in races where the attitudes of voters and non-voters are significantly different. Normally, there are only small differences between the preferences of likely and registered voters in our samples. In all the top-tier races for president, governor, and senate conducted since 2012, the estimated gap between the candidates among the likely voter subset was on average one point narrower than the estimates for all registered voters, which is a minor improvement in accuracy.

Undoubtedly, the large swing between the registered voters and likely voters in the October F&M Poll was unusual, as our reader noted.

Final thoughts

The purpose of reporting on likely voters is to help readers understand how the range of outcomes in this election might change depending on the defining characteristics for the final pool of voters, not to make the race look closer than it is. Most of the time there is little difference between the preferences of likely and registered voters, but this time there was. Perhaps that had to do with the timing of our poll, but it was clear that when the October survey was conducted those who seemed more motivated to vote, and those with a more consistent history of voting, produced an electorate with different characteristics and preferences than what we would see if everyone voted. That could and probably will change in the next week.

We appreciate the questions we receive and encourage you to continue asking them. Your questions encourage us to think critically about our work, our methods, and the ways we communicate about our data. Consumers should expect those conducting polls to respond to questions about methodology that aren’t clear, because methodological transparency is essential for understanding the results of every poll.