Good Judgment Superforecasters Use Evidence-Based Collaboration to Help Organizations Quantify Risks and Predict Outcomes

In a Nutshell: In 2011, the Intelligence Advanced Research Projects Activity (IARPA) held a four-year research competition looking for ways to improve its forecasting of geopolitical events in the wake of 9/11 and the Iraq War. Emerging victorious was the Good Judgment Project — a team-based approach developed by researchers at the University of Pennsylvania — which outmatched forecasts from intelligence analysts with classified data by 30%. Good Judgment now puts its winning methodology to work on behalf of financial services firms and other public and private organizations, with elite teams of “Superforecasters” collaborating to formulate actionable predictions with unparalleled accuracy.

Common sense says that when we put our heads together, there’s almost no limit to what we can accomplish. That basic idea is what makes teams of Good Judgment Superforecasters so good at seeing what lies ahead.

Good Judgment helps decision-makers improve the precision and timeliness of their forecasts. And when leaders have confidence in the information they’re working with, they make better decisions on behalf of their organizations.

The story of Good Judgment begins in the U.S. intelligence community. After the shock of 9/11 and the revelation that Saddam Hussein was actually not holding weapons of mass destruction in Iraq, the government began searching for ways to improve the accuracy of its assessments.

In 2011, teams of academic researchers were invited to compete in what was essentially a series of forecasting tournaments. The Good Judgment Project, led by Phillip E. Tetlock and Barbara Mellers of the University of Pennsylvania, emerged with a superior approach.

Tetlock later outlined reasons for the project’s success in a New York Times bestseller entitled Superforecasting: The Art and Science of Prediction, co-authored by Dan Gardner.

What the competition revealed first was that some Good Judgment forecasters consistently performed better than all the others. And when those top performers assembled to tackle problems systematically, performance rose further still.

An idea for a company began to take shape. Good Judgment incorporated in 2015, with Warren Hatch — originally a Superforecaster himself — now serving as CEO.

“We found out later that Good Judgment Superforecasters outcompeted intelligence analysts on the same questions by 30%, which is pretty substantial — even with the analysts having access to privileged information,” Hatch told us. “What we’ve been doing these past few years is building a business where Superforecasters lend those sorts of talents and capabilities to organizations everywhere.”

Superforecasters serve corporate, nonprofit, and public-sector clients. In the energy space, they help providers make better-informed pricing and investment decisions. In financial services, they provide context for forecasting consumer trends and weighing risk and regulatory variables.

“We also provide training to organizations so they can identify and cultivate their own forecasting talent and put more tools in their analysts’ toolboxes,” Hatch said. “And, in some cases, we help organizations build these processes into their own organizations and form their own Superforecasting teams.”

The key to the project’s original success, Hatch said, was pragmatism. “The approach was ‘let’s try everything,’ all kinds of different combinations that make small improvements that add up to be really dramatic,” he said. “They didn’t have one big idea they were pushing, and that was a big part of the secret.”

For example, the team did not anticipate a hierarchical performance distribution emerging among project forecasters as it did. And the Superforecaster idea emerged from a willingness to test whether taking a team approach would cause elite-level performance to rise or fall.

“Thousands participated in the project as test subjects, and only the top 2% were identified and put on these elite teams,” Hatch said.

Today, Good Judgment recruits Superforecasters in much the same way, selecting the best of the best from top performers on its public site Good Judgment Open, where anyone can hone their forecasting skills, access training resources, and engage with other forecasters.

Questions in a wide range of subject areas pose competitive challenges for participants, with new Superforecasters emerging from these rigors every year to take their place among the professionals.

“It’s the training room for future Superforecasters,” Hatch said.

What separates elite-level forecasters from their peers — and enables Superforecasting teams to outperform individuals — is the ability to follow the evidence even when your gut feeling tells you otherwise.

One of these fundamentals is the idea of a base rate — what Nobel laureate Daniel Kahneman has characterized as having an outside versus an inside view.

“Let’s take a question like who’s going to win the next U.S. presidential election,” Hatch said. “Most of the time, people answer that based on their personal views of the candidates and issues.

“And that is a very inside view in Kahneman’s terms,” Hatch said. “A better place to start is by taking a few steps back and finding comparison classes. What is a base rate? How often have things of this sort occurred in the past?”

That will lead to looking at the frequency of incumbents winning reelection to the presidency, and then even looking at rates for incumbent victories in elections outside the U.S. “That’s where I’m going to start, and only then will I start factoring in the particulars of the upcoming election,” Hatch said.

Another fundamental is the willingness to change one’s view. “In the case of the presidential election, for example, the COVID crisis has introduced a lot of new information that’s probably affecting the probabilities — you want to factor that in,” Hatch said.

What then turns those individual virtues into Superforecasting is collaboration. A group with cognitive diversity — different points of view — will amass a broader perspective and accelerate its mutual learning process.

“That’s what the Superforecasters do on all the questions they forecast — they work as teams, pooling the limited information they have to end up with a more accurate picture,” Hatch said. “It’s a great way to rapidly improve the signal and dampen down the noise.”

Regarding the COVID crisis, Good Judgment recently introduced a public COVID Recovery Dashboard that presents Superforecaster analysis pertaining to the most relevant questions raised by the pandemic.

Concrete analysis on vaccine availability, economic recovery, employment rates, stock and commodity prices, and even the return of professional sports changes dynamically in response to new inputs.

In financial services specifically, productive focus areas for Good Judgment center around specialist topics like mergers and acquisitions and credit default swaps.

“For an M&A deal, there might be a few factors you really want to zero in on, and then put together as a forecast — will this deal go through? You can then compare your view to the view of the market, and if it’s materially different, you can invest in it,” Hatch said.

In another example, German pharmaceutical Bayer AG recently assumed a significant legal liability through a corporate acquisition.

“So we posed the question — will they settle in Q1, Q2, Q3, or Q4 — and we have probability bins where we distribute the probabilities that a settlement will be made within those time frames,” Hatch said. “Armed with that information, you can improve your investment decisions.”

Good Judgment also plans to extend the COVID dashboard idea to corporate verticals, with financial services first on the list. The upcoming Market Movers dashboard will serve as a one-stop-shop for Superforecasting events driving movement in the markets.

“What we want to do in key strategic sectors is identify a number of questions that matter broadly to that industry or subindustry,” Hatch said. “Subscribers can then get access to that dashboard and add their own custom questions on top of it.”

To that end, Good Judgment is always open to receiving proposals for new topics and questions — in financial services and elsewhere.

“If you’d like to pose a topic, send me an email at info@goodjudgment.com,” Hatch said. “I’d love to hear it.”