Prediction Markets & the 2006 Midterm Elections - Predicting the Donald Rumsfeld Resignation More Reliable

I had been planning on talking about Prediction Markets and their power to transform decision making as evidenced by their application to the midterm elections yesterday, plus, one of my favorite sites for commentary on highbrow issues of all sorts - Slate.com - was publishing a special feature on the subject of futures exchanges.

But then, just a few minutes ago, my brother Derek let me know Donald Rumsfeld stepped down as Secretary of Defense (depending on your political stance, either a few years or a few hours too late to help Republicans) in an entirely predictable move following the crushing defeat handed to the GOP in yesterday's midterm elections.

How could anyone reliably predict such an outcome you may ask? Why, prediction markets, my dear reader - that's how.

In fact, the leading PM site on such matters predicted this very thing - the Tradesports contract expired at 100.0 just a scant few seconds later. But first, the announcement on Rumsfeld:

Defense Secretary Donald Rumsfeld, architect of an unpopular war in Iraq, intends to resign after six stormy years at the Pentagon, Republican officials said Wednesday.

 

Officials said Robert Gates, former head of the CIA, would replace Rumsfeld.

 

The development occurred one day after midterm elections that cost Republicans control of the House, and possibly the Senate, as well. Surveys of voters at polling places said opposition to the war was a significant contributor to the Democratic victory.

 

President Bush was expected to announce Rumsfeld's departure and Gates' nomination at an afternoon news conference. Administration officials notified congressional officials in advance.

 

Last week, as he campaigned to save the Republican majority, Bush declared that Rumsfeld would remain at the Pentagon through the end of his term.

 

Rumsfeld, 74, was in his second tour of duty as defense chief. He first held the job a generation ago, when he was appointed by President Ford.

 

Gates is the president of Texas A&M University and a close friend of the Bush family. He served as CIA director for Bush's father from 1991 until 1993.

Gates first joined the CIA in 1966 and served in the intelligence community for more than a quarter century, under six presidents.

 

His nomination must be confirmed by the Senate.

 

Whatever confidence Bush retained in Rumsfeld, the Cabinet officer's support in Congress had eroded significantly. Rep. Nancy Pelosi (news, bio, voting record), D-Calif., the House speaker-in-waiting, said at her first post-election news conference that Bush should replace the top civilian leadership at the Pentagon.

 

And Senate Majority Leader Bill Frist, who had intervened in the past to shore up Rumsfeld, issued a statement saying, "Washington must now work together in a bipartisan way Republicans and Democrats to outline the path to success in Iraq."

 

Here's the Tradesports contract page for the Rumsfeld resignation to give you an idea of what the volume and range looked like this year:

Now, in an era where betting on elections is considered ethically fraught with the potential for serious mischief-making, the Iowa Electronic Markets have happily made book on all manner of outcomes political for almost 20 years now. But that pales in comparison in terms of accuracy and the potential for trouble using the current spectrum of voting equipment seen below (courtesy the Washington Post):

Indeed, the widespread problems with electronic voting that had been seemingly unavoidable didn't really materialize, according to this excerpt from CIO Insight:

With roughly 33 percent of all voters using new electronic equipment to cast their votes in yesterday's mid-term election, there was little surprise when precincts began reporting problems with the machines. In fact, counties in at least seven statesGeorgia, Illinois, Indiana, Ohio, Pennsylvania, North Carolina and South Carolinaextended polling hours as a result of system glitches.

 

At the moment, however, the reported errors have been attributed to poll workers who were improperly trained on how to set up and operate the machines, not to malicious tampering or hacking of systems, as many e-voting experts had fearedexcept in one case where a disgruntled voter smashed an electronic voting machine in Allentown, Pa.

I spotted this article on Newsweek about PMs and Politics Monday, when the head of one of the software vendors who provides PM apps - David Perry of ConcensusPoint - popped up having started a new beta site called Washington Stock Exchange:

What could Tom Patrick possibly have to do with the future of politics? He is 49 years old. He once won a national bridge tournament. He lives in Yekaterinburg, the fifth-largest city in Russia. To his west are the Ural Mountains; to his east is Siberia. His business card says commodities trader, but right now he is between jobs. Which means that the first thing he does each morning is fire up his 128K-modem and steer one of his five computers (three desktops, one laptop, one HP iPaq handheld) to the just-launched Washington Stock Exchange.

 

For much of the rest of the day, Patrick, an avowed political junkie with undergraduate degrees in psychology and math and an MBA from the University of Chicago, will trade on the WSX (thewsx.com). He will buy shares of a stock called George Allen (R) to Win VA Senate Seat. He will sell Barack Obama (D) Announces '08 Campaign. And when all is said and doneat 2 a.m. in Yekaterinburg, when the U.S. business day is endinghe will have pocketed a substantial sum of money. (Internet gambling is illegal on U.S.-based sites, so the exchange deals in play money, which it calls Washington dollars.) Patrickor 20010101, his nom d'écrancurrently tops the WSX's list of most profitable players. His take: W$45 million. His closest rival trails by W$31 million. I'm doing this every waking hour, he says. Much to my wife's chagrin.

 

On the eve of this year's crucial and contentious midterm elections, prediction markets like the WSX are packed with Tom Patricks. For such traders, if not their spouses, the WSX is merely a game. But a growing chorus of experts believe that the site and others like it have the potential to revolutionize how we take America's political pulsea revolution that could, in turn, affect how politicians and policymakers make decisions. Prediction markets are becoming a standard part of the political discourse, says Justin Wolfers, a markets specialist at the University of Pennsylvania's Wharton School. They're tremendously exciting.

 

From Patrick's perspective, such markets are simple enough. He logs on. He checks his net worth. He buys and sells shares in future political events, using information and intuition to predict whether those events will come to pass. But it's only when thousands of traders like Patrick start placing bets that things get interesting. While each individual may not know much, collectively they know a lot. The market aggregates this knowledge, then reflects it, in real time, in the form of prices. Right now, George Allen (R) to Win VA Senate Seat is trading at W$40.13 on the WSX, giving Allen a 40.13 percent chance of winning on Nov 7.

 

Chances are, those odds are on target: prediction markets have proven to be surprisingly accurate. Much more so, in fact, than polls. The oldest political prediction market has forecasted, on average, presidential vote shares within 1.5 percentage points of the actual outcome, besting traditional surveys by a full point. Overall, markets have outperformed polls in recent elections three-quarters of the time. So even though the major national surveys show Allen and opponent Jim Webb stuck in a statistical dead heat, the smart money is on Webband the WSX.

 

The driving force behind the WSXbetting on election outcomesis nothing new. Even though the first organized markets didn't materialize until after Abraham Lincoln took office, Americans have wagered on politics since Washington's day. By the early 20th century, political betting at New York's Curb Exchange, complete with frenzied crowds and brokers barking out bid-and-ask odds, often exceeded trading in stocks and bonds, with traders investing a record $165 million in real dollars during the 1916 presidential contest. The old axiom in the financial district, wrote The New York Times in 1924, [is] that Wall Street betting odds are 'never wrong'. In 1936, however, a new-fangled scientific poll (designed by some guy named George Gallup) correctly predicted the outcome of the presidential election. The public was smitten.

 

To this day, our obsession with polls is undiminished--but prediction markets have clawed their way back. First came the small-sums Iowa Electronic Market (IEM), founded in 1988 and run by the College of Business at the University of Iowa. Sites like InTrade, NewsFutures and the Hollywood Stock Exchange soon followed. Now prediction market success stories pop up in the papers every election season, and poll-tracking sites like RealClearPolitics post up-to-the-minute market forecasts.

 

The WSX arrives, however, at a turning point in the history of the medium. Over the past few years, smart folks have started to recognize that the wisdom of crowds reflected in the market can actually help them make wiser decisions. Until recently, these sites served as hubs for research (IEM) and entertainment (the U.K.-based InTrade, where you can bet real money). But they've performed so well at prediction that companies like Google, Microsoft, Yahoo and General Electric have launched in-house versions designed to let employees bet on key questions, like future sales or potential product launch dates. All told, at least two-dozen companies, according to a recent story in BusinessWeek, have shifted some of the burden of prediction from high-priced expert executives to the guys on the shop floor. Nowadays, real firms in the real world are looking to markets to help them make tough calls, says Wolfers, who advises NewsFutures and the WSX. Markets work.

 

The WSX is simply the first prediction-market company to realize that politics is nothing but another industryand that Democrats and Republicans are clients in waiting. That's really why we built the exchange, says founder David Perry, who's also the president of Consensus Point, Inc., a Nashville-based software and consulting company that has sold its prediction-market engine to GE, Best Buy and Samsung. We wanted to create a database of intelligence about the future of politics so that we can help political parties and individual candidates make better policy decisions.

 

Perry claims that he's currently in touch with Congressional staffers and senior members of opinion research organizations who are frustrated with the inefficiency and inaccuracy of modern polling. (He declined to give names.) He says they've expressed interest in paying to pose particular questionsWill Missourians vote for a ban on embryonic stem-cell research?to the WSX's most accurate traders. (Caller ID and cell phones have made it increasingly difficult for pollsters to round up large, representative samples of the electorate; the WSX only needs 100 Tom Patricks to work.) Right now, there are limited tools for predicting legislative or regulatory outcomes other than gut instinct, says Arik Ben-Zvi, senior vice president of the Glover Park Group, a powerful D.C. consulting firm. Say two major companies are merging. We'd want to ask the WSX if the Justice Department will approve the merger. That predictive power is very importantto companies, to investors and to policy folks. And it could be applied to almost anything.

 

Still, Perry's exchange is young, and potential pitfalls await. Without monetary incentives, prices can fluctuate wildly (though they remain just as accurate, on average, as in a real-money market). Attempts to manipulate stock pricesunsuccessful so farwill intensify if Washington gets involved and raises the site's profile. And wagering on the likelihood of, say, a terrorist attack is enough to make most politicians squirm. (Just ask DARPA, the U.S. Defense Department agency which in 2003 developed a program to project geopolitical risks that was quickly labeled a terrorism futures market and shuttered amid bipartisan furor.)

 

But Perry is optimistic. You've got a lot of political junkies out there who live on this site because they want to demonstrate their political prowess, he says. But at the same time, they're creating accurate intelligence about future events. This is the next generation of politics. Ladies and gentlemen, place your bets.

Unquestionably, PMs predicted election outcomes more accurately than the pollsters did. Their application in the world of business to decision making and a collectivizing of cognitive intelligence across an enterprise is what I'm really interested in and we're at the tip of the iceberg of where this science will take us going forward... and that's maybe one reason why SCIP has Jim Surowiecki keynoting next year's annual conference.