How to become a superforecaster

Prediction: Arts and Science

I can't even describe how much I would like to take a look at the crystal ball right now to see how the next three, five, ten years will develop. Who wouldn't want that? The world around us has changed so much that our old knowledge, insights and information no longer seem applicable when it comes to the further development of our companies. When will normal operations be possible again? Will customer needs still be the same? Who are the future customers actually? And do we have to be prepared for the fact that short-term failures will be part of everyday life in our society?

Finding the answer to the emotional, long-term effects of the pandemic on people is the biggest challenge we are currently facing. The foundations of Maslow's Pyramid were deeply shaken and many of us simply fell down a few steps. Our values are in the process of reforming, and we need to review the long-term effects and (fundamentally) revise our business models. Just think how strangely face mask wearers felt just a few weeks ago. Soon, those who don't wear them could be the exotics: “It will be like smoking in a restaurant: It will quickly turn from an indignation when people want to stop it, to a sudden indignation when someone does it.” Keith Chen, professor of economics, UCLA.

You would love to be one of those superforecasters who find answers to all these questions. It seems too difficult to learn the art of predicting well. But predicting the probability of certain events isn't magic. You can train yourself this skill and even if you are a non-expert, you can take on the great analysts in this profession.

Philip E. Tetlock and Dan Gardner have dedicated a very exciting book to this topic: “Superforcasting,” a masterpiece of prediction based on years of research and a competition in which hundreds of ordinary people pitted against the elite of analysts: The “Good Judgement Project.” They have analysed what constitutes success and failure and supplemented them with exciting interviews with decision makers.

They prove that foresight is not a gift, but a quality that you can train. I recommend reading this book to anyone who wants to predict how the world will change or who wants to understand the probability of certain events. For those who want to get a sneak peek at the content, here are my top tips for becoming a future superforecaster. But don't forget: In the end, training counts the most.

Welcome to forensic economics and quantifying uncertainty

Archaeologists find artifacts from the past and try to connect the connections, and superforecasters do the same. They are simply looking to the future. They use artifacts from the present and the past (news, statistics, studies,...) and combine them to calculate the probability of certain events in the future. Of course, there is a methodology and some basic rules that you have to follow. But that's about it. The most important quality that you have to have is the willingness to accept that everything is an eternal beta version and that we must therefore constantly update and optimize. This ability beats in-depth expertise by far.

Modeling a superforecaster

You should make sure that you either have the following qualities or that you have partners who can authentically represent or complement them:

Philosophical outlook:

· Careful: Accept and respect that nothing is certain

· Humble: Understand the complexity of reality/your task

· Indefinite: What happens shouldn't happen and doesn't have to happen

Capabilities:

· Active open-mindedness: Convictions are hypotheses to be tested, not something you need to protect

· Intelligent and eager to learn, with a “need for knowledge”: intellectual curiosity, joy in puzzles and mental challenges

· Thoughtful: introspective and self-critical

· Number loving: number jugglers

Attitude to work:

· Pragmatic: Not tied to an idea or agenda

· Analytical: Able to leave your own perspective (helicopter perspective) and take other points of view into account

· With the eyes of a dragonfly: Evaluate different perspectives and combine them into one

· Probability-oriented: Evaluate based on many probability levels by asking why several times

· Attentive observers: When facts change, change your mind

· A good intuitive psychologist: Be aware of how important it is to check thinking for cognitive and emotional biases

Attitude:

· A growth-oriented mindset: Believe that it is possible to get better

· Drive: Determination to get the job done — no matter how long it takes 

If you can check off the above points, either yourself or with the help of your team, you've created the perfect basis for the work of a superforecaster.

Set a common language for superforecasting 

Regardless of whether they are superforecasters or not, the first contact with forecasts is their assessment. An unclear choice of words opens the door for misunderstandings. Or it provides a protective cover for dirty work. Let's take Steve Ballmer's oft-quoted statement from 2007 as an example, in which he did not predict any significant market share for the iPhone. He is often ridiculed for this today:

“There is no chance that the iPhone will achieve significant market share. Not a chance.” 

This has often been considered one of the 10 biggest misconceptions about technology. But let's pause for a moment and think about his words. The key message here is “significant market share.” What does “significant” mean? In the USA, worldwide and in which market? Smartphones or cell phones in general? And until when? This is the biggest problem with Superforcasting. The inaccurate use of language.

When you look at the context of this statement, it relates to the global mobile phone market. It would therefore be wrong to see this statement only in connection with the smartphone market, which was only just emerging at that time.

The next question would be “until when.” In short: If you look at 2013, when Ballmer was criticized for making this statement in connection with his resignation from Microsoft, the mobile market share of these devices was around 6%, according to Gartner. Well, judge for yourself whether that is significant or not.

We learn how important it is to understand the full context of predictions, how omitted information can mislead us, and how important it is to use clear words. Tetlock even goes a step further and suggests that we combine percentage probabilities with an exclusive choice of words:

certitude The general realm of what is possible

100% safe

93% (plus/minus about 6%) Almost certain

75% (plus/minus approx. 12%) Probably

50% (plus/minus approx. 10%) Approximately equal chances

30% (plus/minus approx. 10%) Probably not

7% (plus/minus 5%) Almost certainly not

0% Impossible

It would be one of the first things you should learn. The use of clear language.

Superforecasting success lies in asking questions intelligently and in a structured way

If you want to become a successful superforecaster, you should always focus on the really tough questions and not spend time on the ones that can be answered with simple rules of thumb. It's best to focus on questions that don't feel too distant, but that have enough variables for predictive analysis. For example, answering the question of who will win the US elections in 2020 seems difficult but feasible. On the other hand, it is almost impossible to answer the question of who will win the elections in 2024 or even 2028.

Once you've found your question, you need to make sure it's clear and concise.

Schlecht: “When will we have more than 50% electric vehicles?” - That is far from accurate enough.

Better: “Will we be until 2030 more than 50% electric vehicles in private property on the roads Europe have?” - Clarity about the type of electric vehicle, location and time.

Are the questions too broad? Break down seemingly unsolvable problems into manageable sub-problems

Next, divide the problem into its knowing and ignorant parts. Of course, there will be a lot of questions that you won't be able to answer because there are no studies or information. But don't worry. You will be fascinated how often rough estimates lead to a relatively valid result (see also Enrico Fermi and his Fermi questions).

The discipline consists of finding enough small questions that enable a trend-setting “best estimate.” This method is known as digression.

Excursion:

In his book, for example, Tetlock describes a calculation that is based almost exclusively on guesswork (Fermi questions):

Question: “How many piano tuners are there in Chicago?”

He starts by breaking down the question into sub-questions, as follows:

How many pianos are there in Chicago?

· How many people are there in Chicago? About 2.5 million (best estimate, as LA has around 4 million inhabitants)

· What percentage of people own a piano? It's pretty expensive — I'm guessing it's 1%.

· How many institutions, schools, concert halls, bars,... own a piano? I estimate that many of them own one, which doubles the 1% to 2%.

Best estimate: There are 50,000 pianos in Chicago

How often are pianos tuned per year?

Another black box -Maybe once a year.

How long does it take to tune a piano?

Another black box consideration- 2 hours

How many hours does the average piano tuner work per year?

· The average American working week is 40 hours times 50 weeks, That's 2,000 hours a year.

· But piano tuners also need some time on voyages spend - so Draw 20% (at best) their working time from = 1,600 hours 

The rest is math: 50,000 pianos x 2 hours of voices = 100,000 hours of voices divided by 1,600 hours per vote = 62.5 piano tuners in Chicago. 

And the reality? 83. But hey, for the fact that we have no idea, that's incredibly close. And imagine if you had done some research. So be bold and add guesses as to where you have empty spaces.

Ask questions from all possible angles

Superforecasters often approach questions in a similar way. They start with the external perspective (draft basis, ask good questions and break them down into knowable and unknowable parts) and add the internal perspective (hypothesis and events that must happen).

Let us return to our “better” question from earlier and apply an external and internal perspective:

“Will we have more than 50% privately owned electric vehicles on Europe's roads by 2030?”

Basis (external view)

· How many cars are there in general and what is their life cycle?

· What is the image of driving or owning a car?

· How many people have a driver's license?

· Who buys, when and why do people buy vehicles?

· What is the population in Europe and what is the age structure?

· How many private cars are there in Europe? Where do the owners (city/country) live?

· What type of drive system do they use?

· Are there legal requirements that change this?

· How many private drivers and owners are there?

· What are the barriers to buying a car?

· What is the average annual growth rate (CAGR)?

Be creative in your research and try to make a comparison with previous events whenever possible. Nothing is really new. How often do the events you want to investigate take place this way? And how?

Hypothesis (interior view)

This is where you start turning your fundamentals into hypotheses for when the event will happen.

For example, based on your foundations, you expect that 50% of private electric vehicles will be on Europe's roads by 2030:

Transforming questions into quantified forecasts  

Finally, you need to make an assessment of how important each point is for the main event to happen. This is your best estimate based on the overall basis.

probability

Next, break down the individual hypothesis into a series of testable short-term indicators. Be sure to cover a combination of logical and psychological events. E.G.

Objectives set by authorities (Here you would include announcements such as those from California, which wants to be carbon-neutral by 2045, Europe by 2050,...)

General image of cars/drivers (Greta effect, proportion of young adults with driving licences,...)

Based on the analysis of these short-term events, you then determine the probability of each hypothesis:

Translating vague phrases into figures may feel unusual at first and requires a lot of patience and practical practice. But it is feasible and THE superforecaster tool. Think more carefully about uncertainty and reduce complex guesswork into scalable probabilities, as we've done before. The more shades of probabilities you can develop for yourself, the better. A “maybe” isn't really differentiated enough. Anyone who can describe the opportunity as 65:35 instead of 60:40, has a clear advantage. But you must find the right balance between too much and too little self-confidence, between caution and determination. Be aware of the risks of making a final verdict too quickly or spending too much time with the “maybe.”

The rest is a simple rule of three:

Hypotheses 1 share x probability + hypotheses 2 share x probability + hypotheses 3 share x probability + hypotheses 4 share x probability + hypotheses 5 share x probability = 74%. 

prediction

It is probablythat more than 50% of private electric vehicles will be on European roads in 2030. But it is not as good as certain, nor are the opportunities nearly equal.

Superforecasting experts ask others to plug holes

The final prediction requires both regular adjustment and weighting of individual events, which can lead to a revision of the hypothesis. So don't be afraid to prove you were wrong from the start. It's better to discover mistakes quickly than hide behind lots of words and self-fulfilling research.

Collaborate

Regularly checking your own hypotheses is as important as flossing between the teeth every day. This can be boring and sometimes annoying, but it pays off in the long run. It's about finding the right balance between overreacting and underreacting. Filtering out the important information from the jungle of news and not being deceived by wishful ideas is hard work.

This is the moment when interruptions and different perspectives make a lot of sense (if you didn't have the opportunity to consider them from the start). Three or four perspectives for the merger are an important factor for forecast accuracy. So try to collaborate as you adjust.

1. Review your superforecaster modelling from the start and fill in the empty spaces.

2. Review your list of events and consider who could help you gain a new perspective.

For example:

General image of cars/car drivers (Greta effect, proportion of young adults with driving licences,...)

Here, it would be useful to involve someone who has a good insight into the soul of young adults. For example, how would a product designer for smartphones see the development of the values, motivations and needs of this target group?

Superforecasting works best in teams. A group of experts in an open, collaborative, and inquisitive environment brings out the best in each individual and in the entire team. Different knowledge and questions as well as constructive discussions help to create clarity. I can confirm that this has been one of the core elements of... and dos Santos' success in recent years — combining expertise across a wide range of areas.

Another reason for cooperation is the identification of sporadically occurring opposing forces. 

No discussion without a thoughtful counterargument: The thesis meets the antithesis and forms the synthesis. But the challenge for a superforecaster is much more complex, because you have to deal with not just one, but a variety of possible syntheses. Synthesis here is an art that requires an incessant reconciliation of subjective assessments. The danger lies in blowing the horn too early due to the media and current phenomena or ignoring a brewing storm. If you have the opportunity to play this game with experts from different fields, it makes your work a lot easier.

A few final words 

That's it. In the end, it's about trying again and again. Theory alone isn't enough to make progress — it takes practical experience and good feedback loops to improve your analytical and assessment skills. Superforecasting is the result of intensive practice. Look for the mistakes behind your mistakes. Don't try to justify or excuse failure. Face them. Perform sober autopsies. Learn and find the errors in your basic assumptions. It is important that you take a sober look at not only failures but also your successes.

I hope I've been able to give you some ideas for your forecasts — maybe I've sparked your interest and you'd like to take a closer look at this fascinating topic.

If you would like to discuss one or the other point in detail, have questions about our experience with the topic or are looking for assistance in using superforecasting for you, we at... and dos Santos will be happy to help you. With the help of our opinion leaders from the fields of business, science, technology and art, we can help you put together a super forecasting team tailored for you. Please feel free to contact me to discuss the details. Just send me a personal message.

‍