Viewers of the presidential and vice-presidential debates on CNN this year have a novel feature on their screen -- a box at the bottom showing the real-time reactions of a focus group of undecided voters. The results are undeniably bewitching, even for those who don't believe them. "I knew it was completely unreliable and irrelevant," wrote screenwriter Nora Ephron at Huffington Post, "and yet my heart sank and rose according to it."
The visual is the product of a focus-group technique known as dial-testing. Dial-testing relies on hand-held dials that can be turned to register positive and negative reactions in real time. Participants in the focus group -- 30 is a typical size -- sit together and are instructed to continually adjust the dial to reflect how they react to a word, phrase, or sentence.
CNN has seized on the visual power of dial-testing data -- the positions of each dial are aggregated and the resulting numbers are plotted as a line on a graph with the vertical axis representing how positive people feel and the horizontal line representing the time -- as a way to modernize the look of its debate coverage. "If you look at all my colleagues' coverage of the debate, it looks like it could have been done 25 years ago," explains CNN election-coverage producer David Bohrman, who made the decision to broadcast dial-testing results during the debate.
But, besides goosing CNN's ratings, what exactly is the point of broadcasting dial-testing results during a debate? (Both Fox and MSNBC use dial-testing groups for post-debate analysis but do not put the results on screen during the debate.) Skeptics of the network's use of dial-testing point to uncertainty over the meaning of the results, problems with the composition of focus groups, and the distorting effect of watching the debate with a dial in hand. CNN counters that displaying the results keeps viewers engaged and gives them a baseline against which to measure their reactions.
Dial-testing is about 25 years old -- it is the outgrowth of earlier focus-grouping techniques where a moderator would show a brief clip and ask participants to rate a particular phrase on a numeric scale. CNN's dial device is called a perception analyzer and was invented in 1984 by Columbia Information Systems, an opinion-research firm. Perception analyzers and similar devices have been used to evaluate, improve, and fine-tune everything from lawyers' arguments to commercials for at least two decades. In politics, dial-testing has been used at least since the early 1990s to find words and phrases that resonate with voters.
Dial-testing is generally used for much shorter events and is specifically aimed at developing commercial or political messages. Moreover, according to Steve Hinkson, who ran dial-testing groups for two years in his former job as political director of the leading messaging firm Luntz, Maslansky Strategic Research (which currently runs a dial-testing group for Fox), collecting the dial data is typically not the most important part of the process. The moderator uses dial data to confront the group with facts about its own reactions, removing some, but not all, of the subjectivity of traditional focus groups, which can be swayed by the biases of the moderator or the opinions of the most outspoken members. CNN's Soledad O'Brien does ask the dial-testing group questions after the debate, but the audience consumes the raw dial data during the broadcast without the benefit of her questions, which, in a normal dial-testing group, would be the entire point.
So is the raw data meaningful? CNN's focus group is administered by Rita Kirk and Dan Schill, professors of corporate communications and public affairs at Southern Methodist University. Kirk instructs participants to adjust their dials based on whether they have "a favorable or unfavorable response" at any given moment, but leaves determining what that means to the individual.
Some response are fairly consistent. "Audiences don't like it when candidates fight with each other," Kirk says. And, in a bit of news surely heartening to Jim Lehrer, "audiences want people to follow the rules, so any time you see rule-breaking they dial it down." Facts are preferred to rhetoric, and while audiences don't typically like personal anecdotes, says Kirk, if "they believe it's a natural expression, they react very favorably." Joe Biden's misty-eyed moment at the end of the vice-presidential debate was a big hit.
These are heartening findings, but there's no way to know if they represent participants' true reactions or what they think they should feel -- as Kirk acknowledges, a significant body of research shows negative ads work even though voters, independents especially, claim to hate them.
What's more, the process of dialing itself changes how the participants experience the debate. "[Participants] say they pay more attention, because they are focusing on the words," says Kirk.
Cliff Zukin, director of the public-policy program at Rutgers University and former head of the American Association for Public Opinion Research, argues that dial-testing is unhelpful and misleading. He points to the fact that the sample of voters is far smaller than even the tiniest poll.
"It has no scientific validity -- it's not a sample of anything that has generalized validity," he says. What's more, he argues, it introduces inaccurate numbers that assume a power of their own. "The problem with bad numbers is that people tend to believe their eyes."
Simon Jackman, a professor of political science at Stanford who has used an online version of dial-testing to examine voters' reactions to political ads, argues that, though no one knows for sure how accurate dial-testing is, it can sometimes be helpful. "You're also able to pick up reactions to things that if you read the transcript would not be apparent."
It's also unclear whether CNN's dial-testing actually represents the reactions of undecided voters. Kirk says her team takes all the steps it can to ensure that the undecided voters in the focus group are actually undecided -- from examining their Facebook pages to concealing that the pre-interview is for CNN -- but she concedes that people who appear on CNN aren't representative of the average voter. "People who agree to come in to dial-test a debate are already people who are committed to the political processes, who are paying attention."
Undecided voters are, on the other hand, on average less informed about politics than voters who have a clear idea of who they're voting for. And even traditional polls have trouble determining who is really undecided, according to Jackman. "We think a lot of our undecideds are actually masquerading -- there's this great tradition of not wanting to sound like your mind's made up," he says.
Kirk and Bohrman insist that they don't intend for the numbers to be taken as direct measures of independent voters' sentiments, but rather as something viewers can use to compare their own perception of the debate with that of others. "I don't want to treat this like a sporting event, but it's not unlike hearing the crowd cheering or booing -- people are social creatures," Bohrman says. That, for Zukin, is the problem. "It's the blurring of information and entertainment," he says.
Kirk argues that in the long run dial-testing could improve our political debates. "When we started this, our pitch to CNN was that 'talking debate analysis is pretty boring, it doesn't help the body politic.'" The dial-testing has revealed, she says, that viewers are interested in substance.
CNN thinks dial-testing has boosted its debate ratings, helping it reach 9.2 million viewers, more than any other cable news channel, and giving it the highest percentage of 25- to 54-year-old viewers (the age group advertisers are most interested in). That suggests that dial-testing is likely to play an increased role in the coverage of future debates and, it's not much of a leap to imagine, even other political events like State of the Union address or press conferences (Bohrman declined via e-mail to comment on that possibility).
Future dial-testing may become much more accurate by including more people. "There are companies that are scrambling to put [dials] in people's living rooms; four years from now there could be thousands of people doing this," says Kirk.
Distributing the dial-testing across thousands of viewers at home could, Jackman says, solve the problem of sample size -- which could in turn be made representative and statistically significant through careful sifting of the data.
What's more, having viewers dial-test from home would remove some of the other distortions inherent in a television focus group.
"One thing that I like about doing [dial-testing research] over the Internet is that people are doing it at their house with all their disagreements," says Jackman. "People are watching it as they watch anything else -- at their couch."
