Today's Availability Cascade

All of the experts thought that Barack Obama would win the New Hampshire primary. Perhaps more significantly, the prices at Intrade, the political prediction market, suggested that Obama was overwhelmingly likely to win. Many of us have been quite excited about the potential of prediction markets, in which people "bet" on political (and many other) outcomes. Such markets have a terrific track record in politics and elsewhere. And yet Clinton was a huge underdog in New Hampshire. Should we conclude that the prediction markets are unreliable after all? That nobody knows anything? That polls themselves mean nothing?

The answers are no, no, and no. The problem is that as in so many domains, a single, unlikely event (here, the stunning surprise in New Hampshire) is leading to a false generalization about the world.

More particularly: Intrade had Clinton at about 8 percent likely to win, and 8 percent of the time, 8 percent chances should come through. (As in horse races, so on Intrade.) It would be a great surprise if 8 percent chances come though 0 percent or even 2 percent of the time. But on prediction markets, prices have generally been close to probabilities. Events that are priced to be 90 percent likely to happen occur about 90 percent of the time, events that are priced to be 70 percent likely to happen occur about 70 percent of the time, and so on.

This week's skepticism about prediction markets is being driven by the well-known availability heuristic, in accordance with which people assess probability by asking whether events come easily to mind, or are cognitively "available." The success of the long-shot in New Hampshire is now very available indeed. In fact we are witnessing an availability cascade, in which the single incident spreads rapidly from one person to another, leading to the (false) belief that the prediction markets are quite unreliable. This kind of availability cascade is similar to what is observed in the domains of risk and fear, in which a particular event becomes highly salient and is taken, wrongly, to suggest something general about social hazards.

There is a broader point. Some people are now doubting not only the prediction markets but also the polls, saying that no one knows anything, and that anything is as likely as anything else. Don't believe it. To be sure, we are continuing to obtain information about how prediction markets perform and when they do well and poorly. Perhaps they will turn out to be less reliable than they seem -- and in all likelihood, we will obtain a better understanding of when they work. And of course no one has a crystal ball. But the polls are generally pretty good -- and if you want to have a sense of the probabilities, you'd probably do best to consult Intrade.

Comments

[bcowan]

I agree that the occurrence of statistically improbable events should not be taken as discrediting prediction markets.

Their predictive power arises from concentrating enormous numbers of inductive judgments. The resulting probabilities (on which the "prices" are based) are statistical in character, because they express what most judging subjects take to be salient factors for the particular case in the general outcome-producing operations of the contemporary world.

It does not seem wise to ignore this information, since it takes account of what is, to the ordinary observer, otherwise unattainable depth of resources and breadth of perspectives.

But the occasional failure to judge rightly from all these concentrated perspectives raises the question of how it is that they go wrong when they do. What is the information about the contemporary world and its operations that is sometimes hidden from these clouds of witness? What is it that persuades many to vote contrary to the signals emanating from the same broad environment that prompt the statistically predominant judgment?

This is a different question from whether an unusual failure of accurate prediction casts doubt on the efficacy of the practice.

I think there is a kind of probability judgment that is not essentially statistical but is rather based on the intrinsic appeal of a new potential predicate to the judging subjects. Something happens that reveals a new factor beyond the usual or recent patterns of potential value to be realized. Often it is something vivid that changes the picture in a way that obscures, perhaps temporarily, the massive regularities that have pointed in the usual directions.

The market may simply not have been able to process the meaning of some genuinely new or anomalous pattern. In this case it might have been the wet-eyed Hillary episode or, as one professional pollster pointed out in an op-ed in the NYTimes this morning, a resurgence of race-based aversiveness to Obama once he was leading.

What this shows about prediction markets is not that nobody knows anything, nor that the regularities of patterned processes on which they depend are not really there somehow.

But it does show that markets emphasize data that are effectively in the past and depend for their predictive accuracy on some realizations of patterned values in objective events recurring in roughly the same way as they have played out before.

To me this is a more satisfying account than that "8% events occur 8% of the time," when there is no way whatsoever to show that what actually occurred was actually in that 8%. What occurred actually occurred and so there can be no statistical question relevant to it at all.

[minderbender]

bcowan expresses an idea I see a lot: "Their predictive power arises from concentrating enormous numbers of inductive judgments." This may be true, but it seems much likelier to be false.

Literally, the price is set by two people, a buyer and a seller - that's not an enormous number of people. More importantly, the price is not set by a process of supply and demand for that particular contract. If I decide to sell Microsoft stock, and nothing else changes about the world (expected profits at Microsoft, interest rates, etc.), then the price won't change even though the supply of Microsoft stock has increased (and this is true even if I sell massive amounts of Microsoft stock - temporarily the prices may fluctuate, but unless my sale of stock transmits new information about the fundamental value of Microsoft stock, it won't affect the prices for long). The price is actually set by the discounted expected returns on the asset in question.

So in all likelihood these markets don't aggregate information from random people. Rather, they reward people who bring information to the market. These are the people who move the prices and whose judgment is supposed to be so good.

At least, this is what I've been taught. The prediction markets might still be pretty good, but let's not pretend that they do something they don't.

[Nigel Eccles]

I think this ruckus over prediction markets getting it wrong is hilarious. Have these commentators never been to a horse race? I could see them all standing around muttering that the market got it wrong every time a 25/1 shot romps home!

This episode reminds us that most people find it really hard to think using probabilities. My concern is that the US's restriction on online gambling further dulls people's understanding. Gambling is the best teacher you can have in the concepts of probabilities, risk and return.

[Joey Fishkin]

Sure, the New Hampshire data point is no reason to throw out polls or prediction markets. That would be ridiculous.

Nonetheless, this incident does illustrate important limits of both.

Polls report sampling error, but not other sources of error (such as non-response bias and problems with the "likely voter" screen). When many polls say the same thing, as they did in New Hampshire, the aggregate sampling error is very small. That is to say, if sampling error were the only source of error, then 10-12 polls (which predicted a significant Obama victory) would be correct nearly all the time. As far as polls are concerned, this incident is illustrative because it is almost certainly _not_ a case of statistical variation, a 1%-likelihood-happens-1%-of-the-time sort of phenomenon. Some more systematic bias, or late change, generated error other than statistical error here (my bet would be something in the "likely voter" models, which are full of hand-waving voodoo). We pretend that polls are as accurate as the sampling errors indicate at our peril.

As for prediction markets, this incident illustrates once more that they are only as good as the data available to the market participants. If you ask a crowd to estimate the number of jelly beans in a jar, you can expect a very accurate average prediction precisely because each market partipant makes her own independent observations. If you ask a crowd to predict the outcome of an election going on in a state where very few of them live, they can make very few independent observations. Instead of being independent observers, the market participants are, by and large, independent interpreters of a common core of data consisting of the observations and opinions of other people (pollsters, journalists) that are posted on the internet. And this core may not be enriched by the large number of journalists and pollsters scrutinizing New Hampshire: the journalists talk to one another, reinforcing storylines that are always partial at best.

In other words, election prediction markets are fine if you want a quick numerical distillation of the conventional wisdom currently available on the Internet. Maybe the prediction markets even do a slightly better job than a single expert would do with the same data -- although I would bet that as you shrink the pile of data, this ought to become less and less true. In any event, when all are basically working with a common pile of observations -- observations that are limited and subject to known and unknown biases -- it really is just false to imagine that the magic of the prediction market trumps the flaws in the data, especially polls, that we're all working with.

[Kimball Corson]

Scuttlebutt is the poll results were in before Hillary did her tear up scene which brought out a larger than expected women vote in New Hampshire. Obama largely got what the polls predicted because he did not have a last minute tearful break down. Statistically, no one got to factor in the tears.

[Sean O'Hara]

I don't think the market was as wrong as people are saying. Yes, Clinton did win the popular vote, but it's the delegates that really matter, and in that she only tied with Obama.

[JohnLopresti]

Statistics are a viable tool. Consider the intolerable 1% doctrine Cheney has espoused and about which the author has written. But there are political reasons for Cheney's opting for the percent doctrine. In the NH matter, my personal guidance supports what one commenter said about the gender factor adding to HC's margin but insufficiently to accout for the shift. The horse race paradigm while delightful nevertheless ignores the cerebral considerations voters incorporate which are beyond the ken of mere 2-year-old equestrians chomping at the bit to excel in competitive exercise; so, there should be a dissociation in our willingness to accept this as a completely homologous fit to political races. Speaking of which latter, regrettably, having lived some time during my young years in New England or thereabouts, there is an ethnic bias well known to be carefully hidden in poll responders' answers, resulting in another minor shift imbalancing the prognisticators' adumbrations preelection in this instance. There is another factor, ongoing; NH evidently has a new statute allowing it to sequester and not recount ballots first rejected as uncountable; plus, the most prominent tally electronic apparatus in use in NH is the same brand and production model known for erratic results in several tests. Here I will ascribe a countershift toward Obama; in fact, in one partly recounted urban area the recount showed a 10% shift away from the first announced result for Clinton.

In sum, politics at its most intense in the granite state; a drama ongoing.

There are other dynamics between the leading candidates probably best left for a less probability and statistics oriented thread. But I think we have yet to hear the entire story from the NH primary, and that the present moment is an attuned time during which vote strategists are seeking public fora to characterize NH's results as configurations quite disparate from what actually occurred. I will omit mention of which horse I am rooting for in that race.

[Joan A. Conway]

It appears to be a race between a thorough bred by the name of Mitt Romney, and a disused pommel horse by the name of Hillary Clinton, as far as individual political parties are concerned.

However, the contest is really about the political parties between the Republican Camp and the Democratic Camp, both experiencing faction strife, like the private war between Alexander Hamilton and John Adams, and Aaron Burr and Thomas Jefferson in the election cycle of 1800. My bet is still on the Republican Party to garner the electorial vote to win the Presidency. The Democratic Party's strong political machine has an extension problem in mid-america.