ChatGPT and other language models have become an increasingly common resource in many jobs. However, they have a fundamental problem that tends to get worse: these systems often give incorrect answers and the trend is not positive. “New systems improve their results on difficult tasks, but not on easy ones, so these models become less reliable,” summarizes Lexin Zhou, co-author of an article which is published this Wednesday by the scientific journal Nature, which he wrote together with four Spaniards and a Belgian from the VRAIN Institute (Valencian University Institute for Research in Artificial Intelligence) of the Universitat Politècnica de València and the University of Cambridge. In 2022, several of the authors were part of a larger group hired by OpenAI to test what would become ChatGPT-4.
The article was reviewed for a year before being published, a common period for this type of scientific work; but outside of the study, the researchers have also tested whether the new ChatGPT or Claude models solve these problems and have found that they do not: “We found the same thing,” Zhou concludes. “There is something even worse. ChatGPT-o1 (OpenAI’s most recent program) does not avoid tasks and if you give it a handful of very difficult questions it does not say that it does not know, but rather it spends 100 or 200 seconds thinking about the solution, which is very expensive in terms of computation and time for the user,” he adds.
It is not easy for a human to detect when one of these models may be making a mistake: “The models can solve complex tasks, but at the same time they fail in simple tasks,” says José Hernández-Orallo, a researcher at the UPV and another of the authors. “For example, they can solve several PhD-level mathematical problems, but they can get a simple sum wrong,” he adds.
This problem will not be easy to solve because the difficulty of the challenges that humans set for these machines will become increasingly difficult: “This discrepancy between human expectations of difficulty and the errors in the systems will only get worse. People will increasingly set more difficult goals for these models and pay less attention to the simpler tasks. This will continue if the systems are not designed differently,” says Zhou.
These programs increasingly avoid apologizing for not knowing something. This unrealistic confidence makes humans more disappointed when the answer turns out to be wrong. The paper proves that humans often believe that incorrect results on difficult tasks are correct. This apparent blind confidence, coupled with the fact that the new models tend to always respond, does not offer much hope for the future, according to the authors.
“Specialized language models in sensitive areas such as medicine could be designed with opt-out options,” the paper says, or collaborate with human supervisors to better understand when to refrain from responding. “Until this is achieved, and given the high use of these models in the general population, we call for awareness of the risk of relying on human oversight for these systems, especially in areas where truth is critical,” they write in the paper.
According to Pablo Haya, a researcher at the Computational Linguistics Laboratory at the Autonomous University of Madrid, speaking to SMC Spain, the work helps to better understand the scope of these models: “It challenges the assumption that scaling and adjusting these models always improves their accuracy and alignment.” He adds: “On the one hand, they observe that, although larger and more adjusted models tend to be more stable and provide more correct answers, they are also more prone to making serious errors that go unnoticed, as they avoid not responding. On the other hand, they identify a phenomenon they call ‘difficulty discordance’ and which reveals that, even in the most advanced models, errors can appear in any type of task, regardless of its difficulty.”
A home remedy
One home remedy for these errors, according to the article, is to adapt the text of the request: “If you ask it several times, it will improve,” says Zhou. This method involves putting the onus on the user to get the question right or guess whether the answer is correct. Subtle changes to the text prompt (request) such as “could you please respond?” rather than “please respond to the following” will yield different levels of accuracy. But the same type of question may work for difficult tasks and poorly for easy ones. It’s a game of trial and error.
The big problem with these models is that their presumed goal is to achieve a superintelligence capable of solving problems that humans are unable to handle due to their lack of capacity. But, according to the authors of this article, this path has no way out: “The current model will not lead us to a super-powerful AI that can solve most tasks in a reliable way,” says Zhou.
Ilya Sutskever, co-founder of OpenAI and one of the most influential scientists in the sector, has just founded a new company. Speaking to Reuters, he admitted something similar: this path is exhausted. “We have identified a mountain that is a bit different from what I was working on, once you climb to the top of it, the model will change and everything we know about AI will change once again,” he said. Zhou agrees: “In a way, he supports our arguments. Sutskever sees that the current model is not enough and is looking for new solutions.”
These problems do not mean that these models are useless. Texts or ideas that they propose without any fundamental truth behind them are still valid. Although each user must assume their own risk: “I would not trust, for example, the summary of a 300-page book,” explains Zhou. “There is certainly a lot of useful information, but I would not trust it 100%. These systems are not deterministic, but random. In this randomness, they can include some content that deviates from the original. This is worrying,” he adds.
This summer he became famous among the scientific community the case of a Spanish researcher who sent the European Data Protection Committee a page of references full of hallucinated names and links in a document specifically about auditing AI. There have been other cases in court and of course a multitude that have gone undetected.
