July 30, 2003

More on DARPA's Policy Analysis Market

Alex Tabarrok from George Mason gives some details on the DARPA Policy Analysis Market proposal:

Moral hazard in these information markets is a non-issue. First, these markets were intended for information gathering and aggregation and not for insurance and thus positions were to be limited to relatively small amounts. Second, if terrorists want to make money from terrorism there are far better ways than investing in the DARPA market - for example, by shorting airline stocks, taking various positions in oil, shorting nuclear energy companies, agricultural companies (e.g. shorting beef and then threatening to introduce mad cow disease) etc. These markets are more anonymous than the DARPA market and you can take much larger positions without calling attention to one's self.

The bubble issue is real but 1) the issue is comparative - how do information markets compare with say "experts" in forecasting? Evidence from the stock markets and betting markets indicate that markets perform better than the experts. HP used a market to predict future printer sales and found that it outperformed traditional methods (see Charlie Plott's Presidential address to the SEA.) Note that the experts are also subject to herd behaviour, irrationality etc. 2) There are things one can do in the design of information markets and in how one extracts the information in these markets to reduce bubbles.

Finally, one should note that these markets were primarily about pricing securites that were conditional on various policies - e.g. if the US pulls troop out of Saudia Arabia does the probability of terrorism/uprisings etc. go up or down? The conditionality of the bets precludes the reverse causality critique.

Alex Tabarrok
Department of Economics
George Mason University

I sense a little tension here. If the stakes are too small to create moral hazard, why should experts spend their time and money in the market? Markets are good information-aggregators because they offer those with good information the chance to make a healthy profit.

I will be interested to see how well it does--when somebody with a thicker skin than DARPA funds this, that is. I think it will create useful knowledge about information aggregation mechanism.

Posted by DeLong at July 30, 2003 10:57 AM | TrackBack

Comments

"The insensitivity of the idea boggles the mind."

NYTimes
7/29/03

Them alley cats is nutty and mean as the day is long. They play video games to stimulate fear, while American soldiers risk their lives each hour in Iraq. Just how many lives will these nuts save? Doh.

Posted by: lise on July 30, 2003 11:24 AM

____

The point has been made by others, but is worth making again. Coups are not the same things as soybean harvests. The determinants of outcomes (supply and demand determinants) in many markets are mostly public knowledge. Coup planners, on the other hand, generally try to keep plans under wraps. Arguing that the market for HP printers is like the market for, say, whether Turkey will allow US troops to invade Iraq from Turkish soil, requires a real stretch. It might someday come to light that markets provide better forecasts of narrowly determined political events (assassinations and coups, not elections) than do analysts. DARPS's project might have provided evidence. For now, citing the stock market to clame that markets "perform better than experts" ignores efforts to make the performance of publicly owned firms, and of the economy, transparent.

Posted by: K Harris on July 30, 2003 11:34 AM

____

KHarris ... Don't sports teams try to keep there plans "under wraps" too. But somehow the betting markets still work for sports. In what sense are Hamas' plans any more secret than the 49ers' super bowl "plans"?

And on the other hand I'm not so sure the supply and demand info on soybeans is as transparent as you imagine. What do you think the supply curve for soybeans looks like? Isn't decentralized information a problem even in commodity markets? And don't markets help address that problem?

Posted by: boban on July 30, 2003 11:46 AM

____

Lise and K Harris

Right. The lunacy of such an options "game" should be immediately evident to anyone. How have these "thinkers" become so detached from reality, how can they be so lacking in moral compass?

Posted by: arthur on July 30, 2003 11:53 AM

____

arthur - since the idea was to save lives, it would seem that the "thinkers" have their moral compass pointing north.

Posted by: Andrew Boucher on July 30, 2003 11:57 AM

____

The Iowa market does better than polls even though investments are limited to $500. The Hollywood Stock Exchange, has similarly proved to be an good predictor of outcomes based on movies and other entertainment items even though the shares are purely virtual (people play for fame, fun and occasional prizes.) see Science 291: 987-988, February 9 2001 (Letters). HSX apparently sells information derived from these markets to the industry suggesting that the idea passes a market test.

Also note that the idea is not really or not just to get "experts" involved it is also to remove political influences on intelligence gathering and analysis and to quantify and make public a consensus viewpoint. For example, in the case of HP the market may have performed better because using the traditional methods no one on the sales force had an incentive to tell their superiors that sales were going to be low. In the case of intelligence gathering the analysts may not want to contradict what higher-ups think - the yes man problem. Using the market CIA analysts and others could put their money where their mouth *isn't* and this (in the aggregate) would be revealed in a public price that would be difficult to ignore.


Alex Tabarrok
Department of Economics
George Mason University

Posted by: Alex Tabarrok on July 30, 2003 12:00 PM

____

It's funny. I immediately thought it was an excellent idea. I wonder if Bush would have wagered 25 million on finding nukes in Iraq? I'm guessing not.

www.tradesports.com has historical charts showing, for example, the odds of finding WMD in Iraq by September 30. They go from a high of 90 in April to 25 today. If nothing else, it shows that people believed that there were weapons there (and that they no longer do).

Of course, no weapons have yet been found - so the 90 asking price of FIND WEAPONS thus far has been a poor predictor of reality.

Posted by: Saam Barrager on July 30, 2003 12:04 PM

____

The main problems I see are the following

1. The market data would be a reliable indicator of US intelligence and defense agency activity, but an unreliable indicator of terrorist activity.

ie, the market would become a good source of intelligence for terrorist organizations.

2. The market concept relies on the terrorists buying high - ie, launching attacks which correspond to high-priced contracts.

Because high price would correspond to high security agency interest, this would be unwise. So terrorists are likely to prefer targets corresponding to low-priced contracts, or to no contract at all.

ie, they'll "buy" low, good investment behavior. Note that I don't mean they'll buy contracts. That's unlikely and immaterial. I'm using the phrases "buy low" and "buy high" figuratively; instead of 'buying' a piece of paper, they're 'buying' a real-world terrorist attack.

3. The traders on the market will probably be acting primarily on the basis of White House announcements. Regular news will be a factor, but White House warnings of possible impending attack will cause the big moves. If the market were running today, there'd be a spike on contracts involving hijackings.

As a result, the market will be easily manipulated by the administration. If they want to justify an attack on Syria, they can start talking up a Syrian threat; Syria-related contracts will rise; the administration can then use the market as another data point to justify an attack, possibly as a counterweight to persnickety CIA analysts discounting the threat.

Posted by: Jon H on July 30, 2003 12:10 PM

____

Since the idea was dropped at once when it became public, I suppose the American public has a hell of a lot better moreal compass than the lunatics who came up with such a scheme. Wonder if the President would care to be associated with such a "death game." Not.

Posted by: arthur on July 30, 2003 12:24 PM

____

Alex Tabarrok writes: " HP used a market to predict future printer sales and found that it outperformed traditional methods"

So you're saying that the market wasn't good enough, so they relied on a market?

Posted by: Jon H on July 30, 2003 12:25 PM

____

The "traditional methods" Alex was referring to were not market methods, but typical internal corporate information-gathering methods: asking for forecasts, having division heads set targets, etc. The internal market outperformed those methods, presumably in part because it eliminated bureaucratic hurdles to information aggregation (as well as all the other reasons markets work well).

Re: Brad's point about markets needing major incentives to work well, that may be true. But the empirical evidence from decision markets suggests that it isn't necessary. IEM's wagers are very small, HSX is played with play money, and few people are getting rich at TradeSports. More substantively, the vast majority of bettors on, say, the NFL are not putting down huge sums of money, yet the NFL point spread is an exceedingly good predictor of the outcome of football games.

Posted by: James Surowiecki on July 30, 2003 12:39 PM

____

Amazing how morally obtuse these DARPAs are. Too much masking tape on the wondows, I suppose.

Posted by: moen on July 30, 2003 12:40 PM

____

James,

Money does appear to make some difference in a market's accuracy, however. Tom Gruca has run small-money box office markets and found that, even though they had far fewer participants they were as accurate as HSX. Theoretically, if it were opened up to more traders (only academics have been able to play), the Iowa market would be more accurate than HSX's.

While we're at it, we should point out that there's been some research that suggests the OJ futures markets are better at forecasting frost than the national weather service. If only someone had told the Dukes...

Posted by: Justin Lahart on July 30, 2003 01:02 PM

____

James Surowiecki writes: "The "traditional methods" Alex was referring to were not market methods, but typical internal corporate information-gathering methods: asking for forecasts, having division heads set targets, etc."

Well, yes, but the item in question, printers, is an item that is *currently* subject to trade in a market.

The real-world market in which they compete was somehow not good enough; they had to come up with yet another, non-real-world market in order to get their predictions.

Mostly I'm just being snarky. I'm sure there's a lag in getting the numbers from the real-world, slowing down decision times and muddying things.

But it does seem as though such an approach could risk an internal market forcasting sunshine forever, if everyone's been drinking the kool-aid.

Such a system at Enron might not have had a good result. Perhaps HP employees are sufficiently well-grounded for this to work. Or maybe it was designed well to prevent this.

Posted by: Jon H on July 30, 2003 01:05 PM

____

Jon --

What HP was trying to do was anticipate what the market was going to do in the future, so they would know how many printers they should plan to make. (I think actually it wasn't just printers, but printer-supply stuff, too.) Today's "real-world market" for printers can't tell you what the market for printers will be six months from now. But the internal market -- in which the commodity traded was forecasts of future sales -- did a pretty good job of predicting what the real-world market would be.

Posted by: James Surowiecki on July 30, 2003 01:12 PM

____

More substantively, the vast majority of bettors on, say, the NFL are not putting down huge sums of money, yet the NFL point spread is an exceedingly good predictor of the outcome of football games.

Doesn't the same point about transparency apply, though? The bettors on the NFL have huge quantities of precise information about both teams. There's also minimal temporal uncertainty (the schedule's known in advance) which means that current information can be extrapolated with some degree of reliability. Finally, crucial information about the NFL doesn't fall under the rubric of "national security"; while football playbooks are secret, of course, enough plays are run in the open -- transparency, again -- that people can have a pretty good idea about how a given team's offense and defense stack up.

I can't really see how this can be compared to a struggle against terrorists, where opacity (utter opacity for the terrorists, moderate opacity for the government), temporal uncertainty and a broken feedback loop (can anyone know the full scope of governmental successes and failures without high-level clearance?) are the norm.

[All that is leaving aside the moral dimension, btw, which I too decry.]

Posted by: Anarch on July 30, 2003 01:15 PM

____

Jon --

What HP was trying to do was anticipate what the market was going to do in the future, so they would know how many printers they should plan to make. (I think actually it wasn't just printers, but printer-supply stuff, too.) Today's "real-world market" for printers can't tell you what the market for printers will be six months from now. But the internal market -- in which the commodity traded was forecasts of future sales -- did a pretty good job of predicting what the real-world market would be.

Posted by: James Surowiecki on July 30, 2003 01:17 PM

____

Sorry, IE yakked while posting. That first paragraph should be italicized (James Surowiecki at 12:39 PM).

Posted by: Anarch on July 30, 2003 01:22 PM

____

Sorry, IE yakked while posting. That first paragraph should be italicized (James Surowiecki at 12:39 PM).

Posted by: Anarch on July 30, 2003 01:27 PM

____

Sorry, IE yakked while posting. That first paragraph should be italicized (James Surowiecki at 12:39 PM).

Posted by: Anarch on July 30, 2003 01:27 PM

____

It's only a matter of time before someone starts "Fantasy Terrorist Groups", and begins collecting statistics - Hamas' runs, hits, errors, etc.

Posted by: Jon H on July 30, 2003 01:31 PM

____

Sorry, IE yakked while posting. That first paragraph should be italicized (James Surowiecki at 12:39 PM).

Posted by: Anarch on July 30, 2003 01:32 PM

____

boban,

Others have done a pretty good job pitching in for me, but I will speak up for myself, nonetheless.

The audience for a major football team, TV and all, numbers in the millions. They all can have access to more information on the teams headed for the Superbowl than the best analyst of Iran's domestic political situation can hope to have. There are stats on all the players' performances for as long as they have been playing. There are even computer programs readily available to those who want to run simulations, based on those stats. Plans for the superbowl are kept secret, yes, but one thing is certain. Those plans will make optimum use of all knowledge about the strengths and weaknesses of both teams. When Hamas makes plans, analysts do not know who the players are, much less have stats on them. The goal in the Superbowl is clear (sign the fattest contract you can), end the game with at least as many points as the other team, maybe more. It is not even possible to know what short term goal will motivate Hamas. The notion that Superbowl plans and Hamas plans can be equally well priced by outside observers is just nuts. The question, however, is whether much, if any, new information can be had by running a market on Hamas efforts. Superbowl analogies don't help answer that question.

Your suggestion the soybean market is less transparent than I think presumes that you know how transparent I think that market to be. You know exactly one thing about my view - that the soybean market is more transparent than plans for assasination, coup, military adventure and the like. And, yes, I do think that a market in which supply has a good deal to do with acres planted, the timing and amount of rainfall and sunshine, in which demand has a good deal to do with the size of live cattle, pig and chicken inventories (all published), is easier to figure out than whether Saddam's kids will meet their end in any given month.

Want a really compelling reason that financial and commodity futures markets are differnt from markets for political events? I know what soybeans traded at yesterday, and the day before, in a cash market. If there is a daily cash trade in political events, then it is always at the same price (the army didn't go on strike again today) until one day it isn't. All that random walk stuff? It may not apply perfectly to soybeans, but it doesn't apply at all to most of the underlying events that political futures markets try to describe.

Posted by: K Harris on July 30, 2003 01:41 PM

____

It seems to me that the real, i.e. political, point of DARPA's market "idea" was to develop "information" in a way that didn't involve any intelligence experts or agencies, because the administration doesn't want to hear from anyone who really knows anything.

I'm not sure the discussions of market knowledge here get to this. The DARPA idea seems to assume that the good information would come *because* nobody in this market would know anything, except maybe the odd terrorist who wanted to cash in. It would be a faith-based market, not knowledge-based.

As far as the HP experience goes: if I understand this situation correctly, what it really shows is that people in a corporation have a lot of knowledge that they don't dare put their names to. The internal market looks like a way of ferreting out this kind of information anonymously.

Corporate America needs to look to its governance if people have to jeopardize their companies' futures just to avoid retribution from a boss.

Posted by: Altoid on July 30, 2003 02:18 PM

____

NFL point spreads are not good predictors of the outcomes of football games. For an academic analysis, see http://www.pitt.edu/~dejong/Chartist.pdf.

Point spreads are designed to predict outcomes. They are meant to get half the bettors on each side of the bet, so the bookie can collect from losers, pay winners, and keep the vigorish. This might actually predict outcomes, if bettors relied on information and never bet team loyalty. But any bookie knows that the spread on the Arkansas-Texas game will vary widely between Little Rock and Dallas. My old man used to make a lot of calls to Texas in the Seventies, trying to even out his book.

Posted by: R Stanton Scott on July 30, 2003 02:32 PM

____

Actually, point spreads are designed to predict the exact outcome of the game. They're designed to ensure that there's no advantage to betting on a particular side. If a point spread is accurate -- in other words, if the market is working as it should -- the favorite should cover about half the time and the underdog should beat the spread about half the time. And all the studies I've seen show that that's what happens. (In a typical year, the breakdown will be something like 50.4-49.6).

Posted by: James Surowiecki on July 30, 2003 02:44 PM

____

K --

I don't understand the soybean analogy. If PAM had gone into effect, there would have been a daily price on, say, the likelihood that there would be a coup in Jordan. As conditions in Jordan changed, on a day-to-day basis, so too would the price. (Today, a general said some unfavorable things, so the price of the "coup by August 2004" contract rises. Tomorrow, he issues a statement of confidence in the king, and it falls.) Coups don't just happen out of the blue. There are all sorts of signals that suggest that they are or are not going to happen. The idea behind PAM was that some of those signals would emerge if a market that made it profitable to disclose them was made available.

In any case, I don't understand why, for PAM to have been a good idea, markets would have to be equally good at pricing political events as they are at pricing Super Bowl outcomes. The relevant question isn't whether DARPA's market would be as accurate as traditional gambling markets. It's whether it would add to our intelligence-gathering capabilities. Would it give us access to information that we would otherwise not have, and would it do a relatively (that is, relative to other forms of intelligence) good job of forecasting events in the Middle East? I'm fairly certain that it would. But, in any case, the point of PAM was to find out. It was a six-quarter trial run. If it didn't work, we would have known. And if it did work, U.S. intelligence would be stronger. Now all we can do is debate the question on Brad's website.

Posted by: James Surowiecki on July 30, 2003 02:54 PM

____

I'd like to hear of a single example of a market based predictor that works, and also exhibits the same kinds of characteristics as the model suggested for terrorism.

That is, the things being predicted must, at minimum, be based on the quite deliberately concealed decisions of others and designed to be a surprise.

If you can't come up with such a precedent for success, why in God's name believe that the terrorism predictor would succeed? And if you have no good reason to believe it would succeed, why introduce something into the world that on its face is so very grotesque?

Posted by: frankly0 on July 30, 2003 03:32 PM

____

I think the discussion of PAM has been really distorted by the "terrorism" question. The vast majority of the contracts that PAM was planning to offer had nothing to do with specific events. They had to do with macro questions about things like the stability of the Assad regime or future economic growth in Palestine. These are questions that it's plausible that markets would be good at aggregating information on.

The "specific-event" contracts would have been generated by movements in the prices of these macro contracts. In other words, let's say that there was a sharp decline in the price of "Jordan's civil stability, August 2004" contracts. The idea then would have been to ask "What might provoke such a decline?" and to offer a few speculative "specific-event" contracts -- fall of Mahmoud Abbas, increased activity by Hamas, resurgence of pro-Hussein sentiment in Iraq -- that might refine the signal that the price decline was sending.

The point is that PAM would not be trying to read the minds of terrorists, but would rather be reacting to information that was already, in a sense, in the water. In any case, it's clearly wrong to say that there are never any interpretable signals of impending political events.

As for the grotesqueness of the plan, the CIA right now surely has analysts at work calculating the odds on whether Mahmoud Abbas will still be alive by August 2004. Why is it okay for the CIA to do those calculations but grotesque for a market to do it?

Posted by: James Surowiecki on July 30, 2003 03:55 PM

____

In response to R Stanton Scott and James Surowiecki I just want to add this...It doesn't matter what the point spread is designed to do for the casino/house. It matters whether it predicts the final score/spread. If it does, then it may serve as an argument in this case (not that it would settle the issue, but atleast you can point to how it works)
If it doesn't predict the final spread it demonstrates that these markets don't always work.
However, what is the criteria for demonstrating it works or not. Since it is not going to predict accurately all the time, we need to have a null model to test against. I am having a difficult time coming up with a null model to test against (should it perhaps be based on
weighted averages of last years scores and this years scores for each team). I don't see any such tests to show that the market doesn't work.
In fact, I don't think the paper that Stanton Scott refers to sets out to make the case that these markets don't work. It makes the point that we could do better than this market. But it also makes the point that the spreads on the last day are better predictors than spreads on the first day (or it seems to...I am a little dense when reading papers written by economists...and the missing charts/tables don't help). This would be evidence that the market works, somewhat.
Now given time, the market will incorporate this insight and become more efficient...so we need to see if the model did better or worse in subsequent years.
So like any good economist (not that I am either...good or economist), I am going to say on the one hand, these models may work....and on the other, they may not!

Posted by: Sam Jackson on July 30, 2003 04:14 PM

____

Whether the favorite covers the point spread about 50% of the time isn't a good measure of whether the point spread is good at predicting outcomes. You'd want to measure the standard deviation of the difference between the spread and the actual outcome.

Posted by: J. Michael Neal on July 30, 2003 04:35 PM

____

James,

Agreed that markets don't have to be equally good at pricing market and political events in order for a market in political events to provide information. My answer to boban had to do with the analogies he offered. As I wrote in my answer to him - "The question, however, is whether much, if any, new information can be had by running a market on Hamas efforts. Superbowl analogies don’t help answer that question."

The soybean analogy has to do with the underlying cash market. A soybean futures contract has a strong connection to cash trade in beans. Your notion of events which help predict a coup as analoguos to a cash market in soybeans is way wide of the mark. What you are talking about is more like watching the weather to determine soybean futures prices, rather than watching cash soybean prices - weather helps determine bean futures prices, but weather ain't beans. Statements by a general may give an indication of the odds of a coup, but such statements do not constitute a cash market in coups. (There are weather contracts now traded which serve as hedging instruments for those who deal in commodity futures, by the way.)

As with the Superbowl issue, this does not directly answer whether a market in political events could provide information not otherwise available. Rather, I was intending to point out differences that make some analogies hollow.

The central question is whether there is information not readily available to intelligence agencies, or not readily digested by them to come up with accurate predictions, that could be better handled in a market. The experiment may be worth running, but making heroic statements about the value of such a market before hand is silly. Thinking about why such markets might have value leads me to suspect it is because various collecters and digesters of information are have personal and institutional agendas that are getting in the way of doing a good job. It seems highly unlikely that enough people to make an efficient market have sufficient information and experience to out-guess our intelligence agencies, if those agencies were doing the best job possible, but I do not know that for a fact.

By the way, the "Poindexter" market at Tradesports was giving Poindexter 70% odds of being in his Pentagon job at the end of August. TV news reports say he is "expected" to resign. This is a decision made by a few individuals based on assessments of risks and benefits which are not known to those trading the market. That is very different from the way prices are made in most markets. You know, that "eveybody is a price taker" stuff that Cargill takes as gospel.

Posted by: K Harris on July 30, 2003 04:36 PM

____

I think the "personal and institutional agenda" problem is a real one, and is one of the best reasons to think that something like a Middle East prediction market might add value. One of the great virtues of well-functioning markets is that they incorporate diverse points of view, since you can make money buying or selling. (That's why limits on short-selling in, say, the stock market are a bad idea.) I actually think it's more than plausible that there are lots of people with small bits of information that could be germane to U.S. interests in the Middle East who are, for a variety of reasons, never in contact with intelligence agencies. Certainly there are lots of people in the Middle East who would never talk to the CIA or the DIA out of fear, patriotism, or whatever. If even a small percentage of them participated in PAM, it would seem to be an advance over what we have now.

I don't know whether we've been making "heroic statements." We know markets work well at predicting outcomes in a wide variety of environments. It seems plausible that they might work well in this situation, too.

Posted by: James Surowiecki on July 30, 2003 05:07 PM

____

A historical note: limits on short selling come out of the same kinds of ideas (creating an interest group wishing for disaster, et cetera) that produce opposition to the policy analysis market...

Posted by: Brad DeLong on July 30, 2003 05:46 PM

____

I realize this parrot is long since deceased, but regardless:

I spent some time today talking to the CEO of Net Exchange, the company that had set up and was going to be running PAM. As far as I can tell,
PAM was going to be a market in which there were essentially two types of futures contracts offered. The first would be contracts relating to three categories: economic health, civil stability, and military preparedness. (Contracts would be available in these categories for eight countries.) These contracts would be pegged to indices compiled by the Economist Intelligence Unit. (There are obviously problems with how accurate any such index could be, but the point is that it would serve as an independent arbiter that could measure improvements or declines.) In other words, if the value of the index on the day trading began was assumed to be 100, a contract might be something like: "the civil stability index in Jordan will be 86 in August 2004." It's more complicated than this, since combinatorial contracts would be available, but in essence these contracts would attempt to forecast predict, in broad strokes, the region's economic, political, and military future.

The second type of futures contract would be event-specific. These contracts would not attempt to predict things like when Hamas' next bombing in Israel would occur. Instead, they would attempt to forecast major events, events which arguably have some component of non-randomness built into them. These contracts might ask questions like: "Will Mahmoud Abbas still be in power by the end of next year?" or "Will the U.S. still be taking daily casualties in Iraq six months from now?" or even "Will Hamas join a coalition PA government?"

Now, the idea that this market might offer good forecasts seems not just reasonable, but likely. It's not asking anyone to penetrate an opaque terrorist cell. (In any case, the idea that Hamas does not have long-term patterns of behavior which can be interpreted seems to me wrong.)
To me, both of these contracts seem like matters about which lots of people -- Israeli businessmen, Iraqi academics, Jordanian civil servants, American policy analysts -- probably know something relevant. Most of them probably don't know a lot, but together I suspect they know quite a bit. The point of the market would be simply to aggregate this information, which is not being aggregated anywhere else. The principle here has nothing to do with markets per se, though markets present unique advantages and unique deficiencies. It has to do with collective intelligence. Call it PAM or call it Google. In both cases, the idea is the same.

Posted by: James Surowiecki on July 30, 2003 07:40 PM

____

"Canceling the Pentagon's futures market is cowardly and dumb."
http://slate.msn.com/id/2086427/

"...for all the hype about 'terrorism futures,' the vast majority of the 'wagers' that PAM traders would have been making would have been on more mundane questions, like the future economic growth of Jordan or how strong Syria's military was. At its core, PAM was not meant to tell us what Hamas was going to do next week. It was meant to give us a better sense of the economic health, civil stability, and military readiness of Middle Eastern nations, with an eye on what that might mean for U.S. interests in the region. That seems like something about which the aggregated judgment of policy analysts and would-be Middle Eastern experts (the kind of people who would likely have been trading on PAM) would have had something valuable to say."
~~~~

As to my Home Town Paper...

"The insensitivity of the idea boggles the mind."

Insensitivity! The mortal sin to the editors. This is the lead objection they pick. ;-)

"Markets do not always operate perfectly in the larger world of stocks and bonds."

Though markets don't have to work perfectly to make valuable contributions. Don't they have an economist on board who could tell them that?

"The idea that they can reliably forecast the behavior of isolated terrorists is ridiculous."

As if that was the real purpose.

I'd like to see a futures market prediction of when my HTP's editors, when given the chance to quickly wax righteous and call those with whom they disagree ninnies or worse, will instead bother to take the time to learn both sides of an issue and acknowledge that opinions other than their own may be reasonable.

Posted by: Jim Glass on July 30, 2003 08:56 PM

____

"I sense a little tension here. If the stakes are too small to create moral hazard, why should experts spend their time and money in the market?"

Why do we blog?

Well, props to James S (and others), for attempting to explain what this program was, rather than recycling the hype offered by the critics.

A DARPA doc describing the program is here (p B-8 of Appendix, or p. 68-9 of the .pdf file)

http://www.darpa.mil/body/tia/TIA%20DI.pdf

So, sort of related - why the fierce opposition to what is arguably a useful idea? Was it a hit on Poindexter?

And why did the plan proponents get so stupid? Letting the critics make headlines with these examples of "assassination contracts" was incompetent. Why did they not have their ideas thought through, with clear, sensible example that they were ready to defend? Why are we dragging the details out of tired old James, when they should have had a press kit available for that hearing?

Or was it a mousetrap, where the subject was sprung on an unsuspecting Wolfowitz?

Just wondering. Maybe they were trading the Poindexter contract on tradesports.com.

Posted by: Tom Maguire on July 30, 2003 09:19 PM

____

>>If a point spread is accurate -- in other words, if the market is working as it should -- the favorite should cover about half the time and the underdog should beat the spread about half the time

This is not true. Note that you could use similar analysis to claim that the options market was only efficient if 50% of options expired unexercised. Mr Stanton is correct; bookies, like market makers, set the spread to even up the money and collect the (risk free) vigorish, not based on probabilities. This fact about bookmakers is actually used in options textbooks to introduce the concept of "risk-free probability measures".

Posted by: dsquared on July 30, 2003 11:02 PM

____

Just to spell out some of this model’s absurd problems:

One is that a futures market is subject to distortions due to moral hazard, herd behaviour etc, and is thus a lousy basis for prediction.
Leading on from that, this particular market would be especially subject to moral hazard (all those spooks and oil companies enabled to quite legally buy the opinion of the US govt), and even more so to asymmetric information (since good intelligence is by its nature kept secret).

But I think the most fundamental error of all is that a futures market works by rewarding investors who make the correct guesses about future events. If the US Government does use this market as a source of warnings about the future, it will do precisely the opposite.

Very simply: in a normal futures market I put money down to say that the price of copper per tonner will be x in August 2003, and if it does turn out to be x I make money, whereas if it is >x or <x then I lose money.
Whereas here, if I and most other market players guess that ‘Saudi terrorists will attack New York in August 2003’, then if the US Govt takes this market seriously it will take security measures to foil the attack. So, if I did correctly guess the terrorists’ intention, the US Govt’s action will mean that I lose money by being right. On the other hand, I would have kept my money if I guessed incorrectly. ‘Perverse incentives’, anyone?

Posted by: Dan Hardie on July 31, 2003 03:50 AM

____

Just to spell out some of this model’s absurd problems:

One is that a futures market is subject to distortions due to moral hazard, herd behaviour etc, and is thus a lousy basis for prediction.
Leading on from that, this particular market would be especially subject to moral hazard (all those spooks and oil companies enabled to quite legally buy the opinion of the US govt), and even more so to asymmetric information (since good intelligence is by its nature kept secret).

But I think the most fundamental error of all is that a futures market works by rewarding investors who make the correct guesses about future events. If the US Government does use this market as a source of warnings about the future, it will do precisely the opposite.

Very simply: in a normal futures market I put money down to say that the price of copper per tonner will be x in August 2003, and if it does turn out to be x I make money, whereas if it is >x or <x then I lose money.
Whereas here, if I and most other market players guess that ‘Saudi terrorists will attack New York in August 2003’, then if the US Govt takes this market seriously it will take security measures to foil the attack. So, if I did correctly guess the terrorists’ intention, the US Govt’s action will mean that I lose money by being right. On the other hand, I would have kept my money if I guessed incorrectly. ‘Perverse incentives’, anyone?

Posted by: Dan Hardie on July 31, 2003 03:53 AM

____

Just to spell out some of this model’s absurd problems:

One is that a futures market is subject to distortions due to moral hazard, herd behaviour etc, and is thus a lousy basis for prediction.
Leading on from that, this particular market would be especially subject to moral hazard (all those spooks and oil companies enabled to quite legally buy the opinion of the US govt), and even more so to asymmetric information (since good intelligence is by its nature kept secret).

But I think the most fundamental error of all is that a futures market works by rewarding investors who make the correct guesses about future events. If the US Government does use this market as a source of warnings about the future, it will do precisely the opposite.

Very simply: in a normal futures market I put money down to say that the price of copper per tonner will be x in August 2003, and if it does turn out to be x I make money, whereas if it is >x or <x then I lose money.
Whereas here, if I and most other market players guess that ‘Saudi terrorists will attack New York in August 2003’, then if the US Govt takes this market seriously it will take security measures to foil the attack. So, if I did correctly guess the terrorists’ intention, the US Govt’s action will mean that I lose money by being right. On the other hand, I would have kept my money if I guessed incorrectly. ‘Perverse incentives’, anyone?

Posted by: Dan Hardie on July 31, 2003 04:01 AM

____

Just to spell out some of this model’s absurd problems:

One is that a futures market is subject to distortions due to moral hazard, herd behaviour etc, and is thus a lousy basis for prediction.
Leading on from that, this particular market would be especially subject to moral hazard (all those spooks and oil companies enabled to quite legally buy the opinion of the US govt), and even more so to asymmetric information (since good intelligence is by its nature kept secret).

But I think the most fundamental error of all is that a futures market works by rewarding investors who make the correct guesses about future events. If the US Government does use this market as a source of warnings about the future, it will do precisely the opposite.

Very simply: in a normal futures market I put money down to say that the price of copper per tonner will be x in August 2003, and if it does turn out to be x I make money, whereas if it is >x or <x then I lose money.
Whereas here, if I and most other market players guess that ‘Saudi terrorists will attack New York in August 2003’, then if the US Govt takes this market seriously it will take security measures to foil the attack. So, if I did correctly guess the terrorists’ intention, the US Govt’s action will mean that I lose money by being right. On the other hand, I would have kept my money if I guessed incorrectly. ‘Perverse incentives’, anyone?

Posted by: Dan Hardie on July 31, 2003 04:11 AM

____

Would the PAM allow terrorists to practise risk management, to reduce the volatility of their business? e.g. Al Qaeda betting that their training camps won't be pummelled by air strikes in the next ten years--if they aren't they get to keep their bases, if they are, they're able to rebuild.

Posted by: Michael S. on July 31, 2003 06:23 AM

____

Dan says:
"Whereas here, if I and most other market players guess that ‘Saudi terrorists will attack New York in August 2003’, then if the US Govt takes this market seriously it will take security measures to foil the attack. So, if I did correctly guess the terrorists’ intention, the US Govt’s action will mean that I lose money by being right. On the other hand, I would have kept my money if I guessed incorrectly. ‘Perverse incentives’, anyone?"

Same thing happens in the copper market. Say you, I, and everyone else expect very high demand for copper in, say, August 2003. We purchase copper futures in response to that. Copper suppliers will see this spike, and in response, increase their output. So if I did predict the demanders intention, I will lose money by being right. I would have won if I predicted incorrectly. (Or just break even either way, if price reverts to my purchase price.) 'Perverse incentives' anyone?

Seriously, any bidder is going to take into account government action. For example, the spread between the "attempt" and "succeed" future for the same act may be a measure of preparedness. I think you just reinvented the "efficient markets hypothesis", with government as an efficient player in the marketplace.

Posted by: rvman on July 31, 2003 08:29 AM

____

>Same thing happens in the copper market. Say you, I, and everyone else expect very high demand for copper in, say, August 2003. We purchase copper futures in response to that. Copper suppliers will see this spike, and in response, increase their output. So if I did predict the demanders intention, I will lose money by being right. I would have won if I predicted incorrectly. (Or just break even either way, if price reverts to my purchase price.) 'Perverse incentives' anyone?

No- the copper futures market is NOT there to force the government to take action. The Terrorist Futures market can have NO OTHER PURPOSE but to provide the US Government with advance warning of future terrorist attacks, and force it to act against them. Thus the effect that Government will have on market outcomes and on information is incomparably greater. And if the government bases potentially lethal actions on spikes in a market, not knowing if these are NASDAQ-style bubbles or the result of market-rigging, or whatever- that is very questionable in moral terms.

Case a) above posits a spike in 'Terrorist attack on US' futures, with the market then realising that it was wrong when no such attack occurs. But if the US takes the movements in this market seriously, it will be bound to act to prevent 'predicted' attacks- so if the market does falsely 'predict' a terrorist attack, the US might well feel that it has to attack the putative source of the terrorist attack. In a strictly economic sense, this again increases the amount of noise in the model (the market falsely predicts a terror attack; the President therefore orders the USAF to carry out pre-emptive strikes; the terror attack does not occur and the market does not actually know whether this was because no terror attack was planned or because the preemptive strike solved the problem. Do you stick with traders who predicted an attack or not?)

In a slightly wider moral, strategic and political context, I have to say that I'm not too keen on the idea of airstrikes or similar being oredered because a bunch of futures traders have all followed the same instinct and bought 'Algerians hit San Diego' shares.

This is all on a public Terrorism Futures market; a market for intelligence officers would be different, but I don't think it addresses the problems that lay behind the failure before September 11th.

Posted by: Dan Hardie on July 31, 2003 10:03 AM

____

Dan --

It's simply incorrect to say that the "only purpose" of PAM was to provide the US government with advance warning of future terrorist attacks. As lots of people have now pointed out, the vast majority of the contracts on PAM would not have been about specific events, but about broader matters that would not easily yield themselves to US action. If the market predicts a significant increase in instability in Saudi Arabia over the next year, that's very important for the US to know, and would undoubtedly have consequences for US policy. But it's not as if the US can singlehandedly (or manyhandedly) keep Saudi Arabia stable. The same, for that matter, goes for many specific events. Again, if the question is: "Will Mahmoud Abbas still be head of the PA by August 2004?", that's an outcome the U.S. could potentially influence, but not one it can guarantee. So I think the analogy to the copper market is relevant. You're suggesting that the relationship between the contracts and U.S. action is a tight and direct one: this will happen, so the U.S. will act to stop it. That might be true in a few -- very few -- cases, but in most the relationship will be much looser.

Your reference to the air strikes also implies that PAM would have been the only source of intelligence that the U.S. would have relied on to decide what to do. There's just no evidence that this was ever planned. You're worried about a specter that does not exist.

Posted by: James Surowiecki on July 31, 2003 10:50 AM

____

Sorry, just realised after I posted how fundamentally wrong Rvman was.

Rvman says: >Same thing happens in the copper market. Say you, I, and everyone else expect very high demand for copper in, say, August 2003. We purchase copper futures in response to that. Copper suppliers will see this spike, and in response, increase their output. So if I did predict the demanders intention, I will lose money by being right. I would have won if I predicted incorrectly. (Or just break even either way, if price reverts to my purchase price.) 'Perverse incentives' anyone?expect very high demand for copper in, say, August 2003purchase copper futures in response, increase their output<.
Yup.
And you know why that is? Because in free markets
if one actor or set of actors acts in a profit-maximising way, thus sending out a certain set of signals, other actors can base their profit-maximising behaviour on responding to their signals. Markets= not zero-sum.

Contrast markets with war. Bit of a zero-sum game, war. I kill you or you kill me. Can't actually get more zero-sum than that.
So if one set of actors sets out one set of signals, the opposing set of actors (the enemy, in plain english) will *not seek to achieve a Nash equilibrium* (or any other mutual profit-maximising state).
No- get this- they will seek to reduce the profits of ('damage', 'attack', 'kill') the enemy.

So if the market signals that non-terrorists expect terrorists to, for example, strike in Saudi Arabia in October 2003- they've got an incentive not to do it.

Copper suppliers *can* co-operate to maximise profits with copper future traders. And both copper suppliers and copper futures traders have a strong vested interest in each others' continued existence. Unlike- say- Al Qaeda and the US military.

Can Islamicist terrorists co-operate with the US Government to maximise their mutual gain?

Well, Rvman thinks so. Maybe someone should tell him markets don't equal war.

Posted by: Dan Hardie on July 31, 2003 12:28 PM

____

So the implication is that the terrorists had spent all this time planning a strike in August 2003, but had to cancel their plans because the market anticipated them? Is that really a bad thing? (Even accepting the premise, which is dubious, that PAM was going to offer contracts in specific terrorist events.)

In any case, I don't understand the nonzero-sum stuff. Sure, markets are often not nonzero-sum. But in Rvman's example, the people who bought copper futures low anticipating a shortfall of copper in August are sabotaged by the copper suppliers' increasing supply. In other words, their profits are reduced by the copper suppliers' actions. How is this an example of the suppliers cooperating with futures traders to maximize profits?

Posted by: James Surowiecki on July 31, 2003 01:38 PM

____

Betting on soybeans is very much the same as betting on the weather. The cash price you observe is the financial barometer. Betting on whether or not their will be a widespread drought is not so different from, say, whether Arafat will or won't be assasinated. With DARPA we would have had a similar measure for terrorism.

Posted by: boban on July 31, 2003 03:09 PM

____

>So the implication is that the terrorists had spent all this time planning a strike in August 2003, but had to cancel their plans because the market anticipated them?<
If this happens, the market penalises those who correctly called the terrorists' intention, and rewards those who incorrectly called the terrorists' intention. It doesn't function as a predictor of events, and cannot be relied on as a source of intelligence.All the vaunted stuff about the market 'learning', providing better intel than bureaucracies, etc, goes out of the window.
Meanwhile the terrorists keep an eye on the market and may well decide to strike when the market says they won't be striking- that's if anyone at the Pentagon is fool enough to take it seriously. But all this was explained in the original post, and someone wasn't reading.

War is always zero-sum. Markets are always potentially non-zero sum, if possible profits from cooperation exceed those of obliterating the competition.

Posted by: Dan Hardie on July 31, 2003 03:11 PM

____

... and this transparency meme a bit silly. Markets aggregate decentralized information. To argue that markets should only be allowed when all the relevant information is widely known is something new to me.

Posted by: boban on July 31, 2003 03:16 PM

____

>Betting on whether or not their (sic) will be a widespread drought is not so different from, say, whether Arafat will or won't be assasinated.<

It is very different. Because the weather isn't a sentient being who can decide to do the opposite of what the market predicts, but a terrorist is. The crops can't decide to drink less, but Arafat can change his behaviour to avoid assassins. And because the point of issuing futures for a range of crop prices (high if there was a drought to low for a bumper harvest) is to insure against possible loss; whereas the point of the PAM market is to give the government intelligence so it can act to remove threats.
This completely changes the way the market processes information, and also raises the happy prospect of a trading bubble being taken as one of the pieces of information that lead to another war.

Posted by: Dan Hardie on July 31, 2003 03:21 PM

____

Eye wheel admit that soybeans have never bean known to change there behaviour. But farmers can and do respond to market signals. And since soybeans don't regularly plant themselve, the problems you anticipate with a market for terrorism futures should be manifested in the market for soybeans.

And yet the commodity markets seem not to have bean abandoned. Why? Because farmers don't exist to upset the commodities market. And terrorists aren't likely to modify their behaviour based on a little known, widely criticized experiment that depends upon assumptions wildly inconsistent with their own world viewz.

Posted by: boban on July 31, 2003 03:34 PM

____

Has nobody read the DARPA paper ? This was going to be an expert market only:
[Quote follows]

"The application of FutureMAP within TIA will answer questions such as "Will terrorists attack Israel with bioweapons within the next year ?" To answer this question, FutureMAP would aggregate information from a variety of experts, e.g. analysts for Israel and the Middle East and specialists in Bioweapons and other technical areas. The technology question is how to combine this disparate information.... FutureMAP's innovation is to use markets to replace today's approach of discussion and consensus amongst experts. "

So most of the moral hazard concerns go out the window.

Posted by: Naunihal on July 31, 2003 03:55 PM

____

I think things are different for expert-only markets. Such a market might be good in that it enabled the majority of the middle-ranking officers to voice opinions unpopular with their bosses.
It doesn't solve what one might call teh 'Colleen Rowley problem': ie one or two officers making a reasoned case that something is very badly wrong, not being able to share this info with most of their colleagues for security reasons, and being ignored by their bosses. An experts-only market might just end up enforcing the tyranny of the majority.

Boban, you're getting feebler by the post. Soybean harvests are discrete events with predictable start and finish dates, and the statistics on them are entirely trustworthy. Hence the soybean futures market can reliably punish bad predictions and reward good ones, and players in that market can learn and improve. Which part of those sentences applies to, say, coups or revolutions, let alone such a nebulous concept as 'general instability'?

Posted by: Dan Hardie on August 1, 2003 03:54 AM

____

Dan -- Your ad hominem responses grow stronger as your arugements unravel. I'm not quite sure which "statistics" on soybeans you refer to, but if they were "entirely trustworthy" no futures market would be required. Of course uncertainty about future soybean prices -- ultimately arising from uncertainty about global and local weather patterns in a global market -- does exist.

The relevant statistic -- the commodities price --will work for terrorism just as it works for soybeans. Unless, that is, you believe people can forecast, say, next years weather in Argentina better than next years suicide bombings in Israel.

Posted by: boban on August 1, 2003 12:56 PM

____


'The relevant statistic -- the commodities price -- will work for terrorism just as it works for soybeans.'Nuts.

To repeat: the only conceivable purpose of the DARPA futures market is to provide the US Government with advance warning of terrorism, coups against allies, etc, so that action may be taken to forestall such events. If the US Government does forestall such events, that falsifies the predictions of those who correctly foresaw the events, rewards those who didn't foresee the events and means that the market has an inbuilt anti-learning mechanism.

And if traders in soybean futures display herding behaviour, and end up making false forecasts and losing money, so what? Capitalism means that you risk your money on the market, and you know you have a good chance of losing it. If DARPA market traders all herd into buying 'Syria to attack US Forces' futures, and the US takes this seriously enough to thump the Syrians, that is a big deal.

The underlying rationale of a futures market is to insure against the losses that can be made by uncertainty about future prices. These markets do sometimes act as predictors of future prices, but it is not possible to look at a futures market now and state with anything like certainty that its prediction of prices in 6 months will be accurate. Boban believes the opposite. Maybe this is why he is so upset with my 'arugements'.

As for the DARPA people, they are trying to create a futures market in which prediction is the purpose, rather than an unreliable by-product.
Lotsa luck.

Posted by: Dan Hardie on August 4, 2003 07:00 AM

____

It does not matter how slowly you go, so long as you do not stop.

Posted by: Marcus David on December 10, 2003 10:20 PM

____

There is a new archive on PAM, at http://hanson.gmu.edu/policyanalysismarket.html

Posted by: Robin Hanson on January 9, 2004 07:33 AM

____

There is no benefit in the gifts of a bad man.

Posted by: Perlov George on January 10, 2004 03:01 AM

____

Post a comment
















__