Jonah Lehrer, in a post about human reasoning, mentions a famous study in which a pair of scholars tried to analyze the phenomenon of the "hot hand" in basketball, and concluded that it didn't exist:
After analyzing all the shots of the 76ers, the psychologists discovered that there was absolutely no evidence of "the hot hand." A player's chance of making a shot was not affected by whether or not their previous shots had gone in; each field goal attempt was its own independent event. The short runs experienced by the 76ers were no different than the short runs that naturally emerge from any random process. Taking a jumper was like flipping a coin. The streaks were a figment of our imagination.
The 76ers were shocked by the evidence. Andrew Toney, the shooting guard, was particularly hard to convince: he was sure that he was a streaky shooter, and went through distinct "hot" and "cold" periods.
This study has always bugged me, because I think it misunderstands the nature of hot and cold streaks. I doubt too many basketball players (Toney may be an exception) would claim that their hands are always either hot or cold, and that sinking a shot increases the chances that the next one will go in, while missing a shot decreases the chances that the next one will go in. Instead, hot and cold hands happen only sometimes. After spending a few hundred hours in pick-up games, I'd say that real hot and cold streaks happened around one out of every eight or 10 games I played. Some games were better and some worse, but every once in a while, I'd have a game where I just couldn't find the basket, and every once in a while, I'd have a game when I couldn't miss.
Ball players know that feeling -- the days when every time you go up for a shot, even before it leaves your hand you just know it's in the bucket. It's almost a spiritual experience, as though you've clicked into the rhythm of the universe's physical laws and are moving in smooth sync instead of bumbling your way through reality. Bill Bradley famously described being "in the zone" as time seeming to slow down, and the basket looked to him like it was 5 feet across. This isn't some kind of post-hoc narrative one imposes after realizing one hit lots of shots; it's a feeling you are conscious of while it's occurring.
But if those good and bad days happen infrequently enough, from a statistical point of view, they look exactly like random noise. If you flip a coin a thousand times there will be runs where you'll get 10 tails in a row, and if you play a hundred games there will be some where you'll hit 10 shots in a row. They may look the same statistically, but that's only because the magical games are infrequent. But that doesn't mean that the player isn't playing differently during that game, in ways that are so subtle they're probably impossible to detect. At the same time, that doesn't mean we may not be overestimating the degree to which someone is a "streaky shooter." It may well be that that phenomenon is less significant than it appears, and we just notice good and bad streaks more if they happened in important games.
All of this is to say that I'm all for scholars systematically examining folk theories, whether about politics, basketball, or anything else -- that's one of the primary ways human knowledge advances, and we stop doing things like using leeches to treat the flu. But it's important to understand exactly what the folk theory says and make sure that when you operationalize it into measurable variables, you haven't departed from its original meaning. And -- to bring this back to politics! -- this is a way political scientists could improve what they do. During the early stages of research, when they're operationalizing variables (which means taking broad ideas and turning them into specific things you'll be measuring), that's the time when they should be talking to journalists and political professionals, to make sure they understand as much as possible about what's happening on the ground. Because if your variables move too far from reality (as they often do), then you've lost the ability to accurately judge the phenomenon you're investigating.
