L ike most political junkies during the summer of 1992, I closely followed every news report and analysis to see who was winning the presidential race. Yet political science colleagues told me it was unnecessary to pay attention to the twists and turns of the campaign since we already knew who was going to win. A number of social scientists had developed models that could predict the outcome of presidential elections months before anybody had cast a ballot.
Eager to benefit from the advances in my discipline, I looked up the models. But different models predicted different outcomes. I then tried to figure out which were the better models, so I could know which predictions to believe. My close investigation of the history and statistical techniques of presidential election forecasting models led me to conclude that none of the existing models deserves our faith.
My colleagues in political science are not the only ones who have been drawn to the predictive power of presidential forecasting models. The Washington Post, the New York Times, the Wall Street Journal, and NBC News all featured the models' predictions. One Bush insider even said the campaign suffered from overconfidence and delayed aggressive tactics because they put credence in a forecasting model that predicted easy victory.
The problem, however, is not that the Bush campaign picked the wrong model to believe; the problem is that predicting the outcome of presidential elections accurately and reliably is simply not within the current grasp of social science. To make meaningful predictions, the statistical techniques used by the modelers require far more information about past presidential elections than exists. Even if we were to have enough information, voter behavior in presidential elections might just be too random to predict accurately.
Despite the deficiencies of these models, their reputation for success continues to grow. Why have these models received this reputation, and why is it mistaken?
HOW THE MODELS WORK
All presidential forecasting models use statistical methods common in the social sciences. The technique, known by the daunting name "multi-variate ordinary least squares regression," estimates the independent effect of a number of factors on the phenomenon being studied. Using this method we could, for example, estimate the relative influence of presidential popularity, economic growth, and incumbency on presidential election outcomes. Most social scientists use this technique only to examine historical relationships.
To generate a prediction from this sort of model, the presidential election modelers compare calculations from previous elections of such independent variables as presidential popularity and economic growth with their current values (low popularity, moderate growth) to estimate the result in a future election. Essentially, the forecasters measure the historical relationship between various factors and election outcomes and then predict the outcome of a current election based on the present status of those factors.
The models differ in the factors they include to predict the election as well as the way in which they measure the election outcome. Of those models that forecast the result in terms of the two-party popular vote percentage, Emory University professor Alan Abramowitz's model was closest to the mark in 1992. Using economic growth, presidential popularity, and length of time in office as factors, he predicted that Bill Clinton would win with 53.7 percent of the two-party popular vote, while the actual figure was 53.4 percent. On the other end of the spectrum was Ray Fair of Yale University, who had the ear of the Bush campaign. His indicators included economic growth, inflation, and incumbency. He predicted a landslide with George Bush winning 55.7 percent of the two-party popular vote.
Since the electoral college vote actually determines the winner, some models predict the percentage of the electors rather than the popular vote. James Campbell of the National Science Foundation has a model that does this by forecasting the outcome in each state. The factors he uses include state partisanship, home state advantage, and economic growth in each state as well as the nation. He predicted that Clinton would win 61.7 percent of the electoral college vote. Clinton actually received 68.8 percent. Michael Lewis-Beck of the University of Iowa and Tom Rice of the University of Vermont have a model that forecasts the national electoral college outcome without including state-level information. They factor in national economic growth, presidential approval ratings, mid-term House elections, and party unity. The pair, generally viewed by political scientists as the deans of presidential election forecasting, predicted a Bush victory with 57.5 percent of the electoral college vote.
ARE CAMPAIGNS IRRELEVANT?
These models share not only a methodology but also a political assumption: campaigns do not significantly affect election outcomes. Campaign strategies, media spin, and debate performances are not incorporated. Instead, the models rely on "fundamentals," such as the rate of economic growth and the power of incumbency, to generate their predictions. In addition, the modelers make their predictions by Labor Day, so they forecast without knowledge of the fall events.
The premise that campaigns are essentially irrelevant appeals to many political scientists who have long suspected as much. There are two arguments for such skepticism. First, perhaps both parties are equally competent at running presidential campaigns. Every thrust by one side is more or less parried by the other. Like players in any competitive market, political parties must sell their products as best they can. Because the presidential campaigns of both parties have frequent access to the media, vast financial resources, and dedicated staff, the outcome may be more related to the conditions of the market than to the marketing. The belief that Republicans have a monopoly on competent campaign personnel or that Democrats receive excessive press support may be nothing more than rationalizations of defeats determined by economic factors.
Second, voters may discount campaign promises and positions and rely more on the fundamentals like the state of the economy to make their decisions. Voters know that campaign promises are often not kept. Moreover, the average citizen may be in no position to evaluate the proposed policies. Experts can hardly agree on the likely consequences of proposals, so how can individual voters be expected to sort through the clutter?
Instead, many political scientists believe that voters judge candidates and parties retrospectively. That is, they evaluate the incumbent or the incumbent's party based on their personal and financial well-being. If they are "better off than they were four years ago," voters will be inclined to reelect the incumbent party. If they are worse off, voters will "throw the bum out." According to the retrospective voting school, the incumbent's past performance on easily measured criteria should be more decisive in elections than unpredictable campaign maneuvers.
S ome readers may deplore the economic determinism in this entire approach. After all, to accept the idea that voters base decisions primarily on variables other than candidates, issues, or campaigns is to believe that politics as ordinarily understood doesn't matter much. Is it really the case that all the sound and fury of modern campaigns simply cancels itself out, and that voters in the end largely ignore candidates and messages? If so, a great deal of time and money are being wasted by smart people.
Yet as politics has been reduced to the manipulation of symbols, and as sound bites have driven out informed public discussion of issues, perhaps people's own conditions really are the more solid basis of their voting behavior--assuming those conditions can be accurately measured. In parliamentary systems, where the personality of the prime minister is structurally less of a factor, the electoral swing in favor of or against the government is often fairly uniform and closely correlated with perceived economic well-being. The same is true of the vote for state legislatures in the United States.
Despite the structure of our system, with its greater emphasis on the chief executive, perhaps we overstate personality. Voters' perceptions of leaders--as measured in the apparent ups and downs of approval ratings and the see-sawing of leads during campaigns--may be shallowly held, ephemeral, and far less reliable than people's subjective feelings about their own well-being. Thus this brand of political science is not as ipso facto foolish or deterministic as it may seem. Indeed, to suggest that voters are swayed by campaign slogans and sound bites seems the more cynical perspective. Perhaps voters are not the sheep they are imagined to be and cannot be manipulated so easily by the campaign strategists lauded so frequently in the press as geniuses. We should take heart that voters cannot be sold a candidate as if he were a box of soap.
If it is true that presidential campaigns make little difference and if voters' choices are largely a function of known economic and political fundamentals, then predicting election outcomes should be possible. One would only have to determine the historical relationship between these fundamentals and past election results to determine when people are likely to think that they are better off and when they are likely to throw the bum out.
Retrospective voting theory is supported by a variety of evidence other than presidential election forecasting models. But the fact that presidential election prediction further buttresses a theory that many political scientists are already inclined to believe has led to little critical examination of the models. Even if retrospective voting theory were valid, forecasting models lack enough information, a glaring defect that should have been detected much earlier. The neglect of critical examination, assisted by misleading statistics from the modelers, has imbued presidential election forecasting with too much credibility. The shocking thing about this brand of forecasting is not that social scientists have tried to do it but that so many in academia, the press, and campaign staffs have been fooled into believing that we have succeeded.
MISSING THE MARGINS OF ERROR
For one thing, forecasting models are limited by our lack of historical information on the relationship between political and economic fundamentals and elections. While there have been more than 50 presidential elections, reliable information about the economy is available only for the past 20 elections, and polling data exists only for the last 12. With as few as 12 observations upon which to make their predictions, the models gain little leverage from statistical techniques. Normally statistics condense many pieces of information into a few, but in this case we only have a few at the start.
Because our estimate of the historical relationship between the fundamentals and election results cannot be very strong with so few observations, we would expect a great amount of uncertainty in our prediction. This uncertainty can be expressed in something called the margin of error. Virtually everyone who has seen an opinion poll is familiar with the idea of a margin of error. A poll may say that a result has a margin of error of plus or minus 3 percent. Generally, this means that if we were to repeat the survey 100 times under the same conditions, we would expect 95 trials to have results within three points of those reported.
The forecasting modelers, however, do not report their margins of error. They only tell us their best estimates of the outcome without indicating how confident we can be in their predictions. Looking only at this best estimate, one might think that Alan Abramowitz's model was highly accurate since it generated a prediction within three-tenths of a percentage point of the actual result. Yet a model's prediction consists of not only the best estimate but also the range of uncertainty around that estimate. According to my calculations, the margin of error for Abramowitz's model is plus or minus 5 percent. The model, then, really says that we can be highly confident that Clinton would get somewhere between 48.7 percent and 58.7 percent of the two-party popular vote. Bush could win by a narrow margin or lose in a landslide. With such a wide range of possible outcomes, his prediction could end up being accurate by chance alone.
The margins of error for the other models are even larger. Ray Fair's has a margin of error of plus or minus 8 percent around his prediction of Bush winning with 55.7 percent of the two-party popular vote. So according to his model, Clinton could have won comfortably or Bush could have won in an unprecedented landslide. Once we consider margins of error, we can no longer say that Fair's model was wrong. Almost any outcome was possible according to his model. The fact that his prediction was off by more than Abramowitz's may simply be Fair's bad luck.
The percentage of the electoral college vote is harder to predict because all of a state's electoral votes go to the winner even if the popular vote is close. The margin of error for the model advanced by Michael Lewis-Beck and Tom Rice, therefore, is much larger: plus or minus 29 percent. James Campbell's model is a little better, with a margin of error of plus or minus 18 percent. The margins of error for these models are so wide, encompassing victory for either candidate, that they are useless for prediction.
PROBLEMS WITH MAKING THE MODEL
Margins of error capture the level of uncertainty in a prediction--assuming we have the right model. The choice of factors to include in the model adds to the uncertainty. The decision to include one set of variables, such as presidential popularity and growth in GNP, rather than another, such as the rate of inflation and unemployment, changes our prediction. Because we may be unsure of which model to choose, the predictions of each model have greater uncertainty than is conveyed in their margins of error.
Presidential elections might be explained by a long list of factors. Whether we choose to include one set of variables or another in a model makes an enormous difference in the prediction. The models advanced by Alan Abramowitz and by Lewis-Beck and Rice, for example, are theoretically and methodologically quite similar. Yet Abramowitz's best estimate was right on target in 1992 while Lewis-Beck and Rice were off by a long shot. How could we have known, before the election, whether to believe the prediction of one model or the other? Both models were equally good at explaining past election outcomes and neither was more theoretically compelling. Very little distinguishes the models, but they offered conflicting forecasts.
Because we have only a dozen or so presidential elections to help us sort through all of the possible factors to include in a model, we cannot be sure how best to compose a model. Slight changes that have no empirical or theoretical significance can greatly alter the prediction. Since we do not have enough information to know before the fact what to include in the models, and since including different variables produces different predictions, we are in no position to predict the outcome with any serious level of certainty. We would have to predict the right model before we could predict the right outcome.
DRAWING THE TARGET AFTER FIRING THE ARROWS
If picking the right presidential election forecasting model is currently indeterminate, and if the margins of error are large even if we were to know the right model, one would expect that the models would be sullied by a record of poor prediction. But forecasting models maintain a reputation for success. In part, this reputation can be explained by the lack of critical examination by political scientists inclined to believe the theory behind forecasting. But more important, the modelers themselves have sometimes reported statistics that exaggerate a model's performance.
The two most prominent presidential election forecasters, Lewis-Beck and Rice, tout their model's excellent record. They report that it successfully "predicted" 10 of the last 11 elections with an average absolute error of 5.6 percent of the electoral college vote. That would be an impressive record--if they were making predictions. Instead, they are reporting how well their model fits certain events after those events have taken place. With fewer than a dozen elections, it is no surprise that they could find some combination of variables that could account for past outcomes.
Other political scientists have offered humorous models that successfully "predict" past presidential elections. One joke model predicts elections based on the quality of the Beaujolais harvest. Another is based on the baseball league that wins the all-star game, and still others are based on candidates' height, attractiveness, and left-handedness. All of these joke models have a record of successful "prediction," but that is because they were made after the fact. If you already know where the arrow lands, it isn't hard to draw the target around it.
Even worse, some of the modelers have changed their models after an election to maintain its successful record of prediction. Ray Fair, for example, changed whether Gerald Ford should be counted as an incumbent in his model after 1976 and altered one other variable to improve the model's track record. Rather than altering their model, Lewis-Beck and Rice have simply proposed three different models in three elections to ensure their history of success. Perhaps the only predictions skeptical readers should believe are those made before an election occurs.
Unfortunately, even successful predictions made before an election do not necessarily validate a model. According to one estimate there were more than 20 presidential election forecasting models making predictions in 1992. Since there are only two realistically possible outcomes and relatively little variation in the popular vote percentages, it is possible that some of those models will pick the right winner and even come close to the right margin of victory by chance alone. Only if we could observe the same model over several future elections could we really develop confidence in its predictive power.
While retrospective voting theory offers intriguing hypotheses about the irrelevance of campaigns, presidential election forecasting models are simply incapable of providing a meaningful test of the theory. Presidential election forecasting models, like the Beaujolais or baseball models, should remain part of "recreational political science." They cannot tell us who will win the presidency, nor can they test any theories, but the models can provide an entertaining distraction from the pundits and polls during a campaign.
