Harvesting the wisdom of the crowd

The “superforecasters” who show remarkable proficiency in answering big geopolitical questions - despite their lack of knowledge.

By James O'Malley

Can the wisdom of crowds make more accurate predictions than top intelligence experts? Philip E Tetlock, a University of Pennsylvania political scientist, set up the Good Judgment Project in 2011 in order to find out. Funded by the US government, the programme identified a group of “superforecasters” who, despite possessing no particular expertise nor access to any secret information, have shown remarkable proficiency in answering big geopolitical questions – such as whether Serbia would be granted European Union candidacy by the end of 2011, or whether Bashar al-Assad’s regime in Syria would survive in 2012.

Supers, as they are known, are those in the top 2 per cent of participants. Tetlock’s thesis is that by aggregating the guesses of supers, he can derive a probability that comes close to the truth. Although no one can know everything about a given topic, everyone knows a little about something. Tetlock argues that aggregation amplifies what people get right and cancels out what they get wrong.

It sounds too good to be true: a relatively simple approach that overturns decades of received wisdom, a little like the case studies in Freakonomics or Moneyball, but specific to intelligence. What is astonishing is that the approach has been shown to work, with participants achieving accuracy scores well above random chance and supposedly even beating CIA analysts (though the agency apparently refuses to confirm this).

In October, the supers held a public conference at the London School of Economics. Before it opened, I met Andreas, an Italian super who lives in London and works in finance. When I asked him which forecast he was most proud of, he told me that he had closely predicted the number of banks that would fail the EU “stress test” in 2014. But was this forecast his “best”? Rather than giving a quick answer, Andreas instinctively began to deconstruct my rephrased question, asking rhetorically what might be meant by “best”. According to Tetlock, the ability to break down questions is a key skill in becoming a super.

The methods used to collate judgements have varied since 2011 but one of the most successful has been to use a market, or “forecasting tournament”, which works much like a real stock market. Each super is given 50,000 units of a fictional currency known as “Inklings” to invest in various outcomes. If, say, you had bet on Assad surviving a year ago and the odds were then at 10 per cent, you would have won a big return.

What was clear as I sat in the conference room – listening to the likes of the former Ministry of Defence analyst Nick Hare speaking about forecasting defence issues, or the conference organiser Michael Story bemoaning the predictions of media pundits – was that I was surrounded by very smart people. Pablo, the super next to me, was the dean of MBA students at a major business school. One questioner identified himself as working for the Bank of England. Contrary to the promotional pitch of Tetlock’s recent book, Superforecasting: the Art and Science of Prediction, these are not, for the most part, ordinary people, but highly qualified, well-connected and well-read individuals.

One participant suggested that the questions the supers performed worst on were those in their own professional field. When it came to breaking down a question and engaging with it objectively, he claimed, he was unable to see the wood for the trees. Chris, a strategy management consultant who joined the programme in its second year, described this quality positively as “conscious incompetence” – in essence, having an awareness of what you do not know and being open to having your forecasts challenged by others so that together you can arrive at better judgements.

Wanda, a Spanish-speaking guest from Texas (something of a rarity at the conference: the demography of supers seems to be overwhelmingly white and male), told me that she had performed best on questions relating to Iran. To do so, however, she had to be dispassionate, something she found difficult because she has strong opinions about Barack Obama’s recent nuclear deal with Iran. Ironically, for a programme that focuses on contentious political issues, the one taboo among supers seems to be personal politics. When I asked another participant if he would discuss his voting patterns with his super colleagues, he said that to do so would hinder co-operation.

So, what are the chances that this method of intelligence gathering will be taken seriously beyond this select group? I’m not so sure, but I know the right people to ask.