The result of combining a prediction community such as Metaculus and a language model like ChatGPT 4 It’s the future. And it sees the future. A new artificial intelligence (AI) system, developed by researchers at the University of California, Berkeley, is able to predict future events more accurately than humans. To do this, the AI searches the Internet for information, selects the news articles it considers relevant, summarizes them, and feeds them into a reasoning system. “Forecasting requires a human to sit down and gather a bunch of sources, figuring out which ones to trust and how to weigh them. A linguistic model can do this very quickly,” explains Danny Halawi, one of the authors of the study. studypublished in the magazine New Scientist.
“Will a nuclear weapon be detonated in 2023?” The system said no. To be able to answer questions like this, Halawi and his team have trained ChatGPT-4 through a process called “fine-tuning”: this method uses thousands of forecasts from prediction markets, such as Metaculus or Polymarket, and together with news retrieval, it obtains better reasoning. “To make it good at forecasting, we train the model with good examples,” Halawi adds. For each prediction, the system asks Google up to six times — as if it were going to six different people — and consults up to 20 articles. That’s six questions and no more, because it costs a lot of money even though the ideal would be to do it “100 times,” Halawi confesses. Asking an internal question to the system costs $1 (0.93 euros), while for OpenAI, for example, a question to ChatGPT costs “less than $0.01 (0.0093 euros),” he adds.
The forecaster works like someone searching for something on the internet: the more specific the question, the better. But is it reliable? Halawi is confident in the system: “If you want to predict accurately, the best strategy is to be unbiased, try to get the most accurate sources of information and make a reasonable and sensible prediction based purely on evidence,” he argues. The system scores 0.179 compared to the crowd’s 0.149 on the Brier scale, which measures the chances of being right on a scale of zero to one. The difference is not huge, but neither was it when the Deep Blue computer beat world chess champion Garry Kasparov in 2017. 1997, winning the match in six games by 3½-2½. And yet, this milestone has gone down in history.
For Ángel Delgado, an Artificial Intelligence engineer at Paradigma Digital, the key to the predictive model is that by combining “the models of forecasting “Traditional language models” allow the system to be able to “process numerical and textual information.” In this way, “by taking into account more information, they can become more precise than humans,” he explains.
The wisdom of the crowd
Researchers have borrowed the idea of “wisdom of the crowd” for their development, i.e. the effect that a set of solutions proposed by a group works better than individual solutions. EL PAÍS journalist Kiko Llaneras corroborates this effect in his list of predictions for 2023: by combining all the predictions made by readers and averaging them, the accuracy rate rose. Similarly, the system compiles all the information and arrives at an average of the answers. “Individual people have limited access to information and have their own biases. If we take different predictions and then average them, more information is indirectly gathered,” explains Halawi.
But what practical effects can this application have? We would hardly ask our computer about the possibility of finding love or a new job. The answers that this technological crystal ball can give us are much more practical. Richard Saldanhaa professor of machine learning at Queen Mary University of London, prediction can be useful for economic and political analysts’ decision-making, although the financial market “is not necessarily based” on the wisdom of the crowd: “The weight of buying and selling securities usually dictates the negotiated prices. There is often a relationship between the wisdom of the crowd and the actual price activity of the relevant securities, but these relationships should not be naively trusted,” he explains. Delgado agrees that it is good for estimating the impact of political and social factors on the economy and predicts that, in the long term, the system will change how markets are understood. “One of the factors that most affects the market is uncertainty and as we get used to this type of tool, it will be reduced,” he explains.
More probable than uncertain
One peculiarity of the development is that it works best with “uncertain” questions, meaning those with a higher chance of being answered. This is because the system uses ChatGPT-4 as one of its sources and it prefers to protect its answers for security reasons: “ChatGPT is trained not to give people advice that could harm or affect their lives. If it tells you there is a 0% chance of something happening and then it happens, it will cause you harm,” explains Halawi. The probability of the world ending tomorrow, close to 0%, is an example of this.
Uncertainty also encourages the minimization of biases, according to Luis Herrera, solutions architect at Databricks Spain: “Although they are not completely free of biases, AI models can be designed to minimize common biases, such as overconfidence or confirmation. AI models tend to be inaccurate due to their preference for conservative responses.” He also agrees that the model outperforms humans due to its modus operandi“They can integrate new information and update their predictions in real time, essential for rapidly changing situations.”
The type of forecast in the study is called “judgmental forecasting,” and it involves predicting something with very little data, such as the detonation of a nuclear weapon. “Because it happens once in history, there’s very little to rely on. You have to be much more creative in asking the question and coming up with a probability,” Halawi says. The system is different from other prediction models like GraphCast, the artificial intelligence from the University of California, Berkeley, and the University of California, Los Angeles. Google DeepMind which predicts the weather, which does have a lot of data to obtain the forecasts: “The forecast they use is time series modelling. This is usually done when there is a lot of data, for example, when you are predicting the weather and you have the meteorological history of the last month or data on humidity in the air,” says Halawi.
Halawi and his colleagues’ system based on collective predictions has proven to be better than humans. However, Herrera raises the possibility of testing it on superforecasters, those people capable of predicting future events with a level of accuracy that surpasses the general public and experts. They also use analytical and statistical methodologies to eliminate biases and constantly adjust their predictions based on new information. Will the new system also outperform them?
You can follow THE COUNTRY Technology in Facebook and X or sign up here to receive our weekly newsletter.
Subscribe to continue reading
Read without limits
_
