Artificial intelligence models answer nearly 90% of climate-related questions correctly but readily abandon established scientific facts when users challenge them, according to a new report by Unesco and University College London (UCL) released at COP31 in Antalya.

The study, titled "Can We Trust AI on Climate? Why it seeks to please the user, and fails on complex reasoning and multilingualism", was launched on the sidelines of COP31 on Monday. Researchers found that in eight out of 10 cases, models pivoted at the first sign of mild disagreement from users, at times confirming errors such as "CO2 is not the main driver of climate change". The models were also only 75% accurate when asked whether new evidence supports or contradicts established science, and scored 33.8% on assessing how well policies work together, barely above the 33.3% expected from random guessing.

The report said accuracy across 13 languages varied widely, averaging 88.6% in English and performing comparably in Spanish, French and Portuguese, but dropping to 83.6% in Hindi and Bengali, 81.4% in Urdu and 78.3% in Swahili. UCL researchers tested 15 large language models, while 33 different models were tested across different experiments.

Leona Verdadero, Unesco's lead author of the report, said it is concerning that models have a design flaw whereby they are incentivised to agree with their user and deny facts so readily. Dr Maria Perez-Ortiz of UCL Computer Science, the report's senior author, said AI systems are remarkably good at explaining scientific knowledge, but this can give a false sense of their capabilities. The report warns policymakers and researchers who use AI models to develop climate policies to reconsider their reliance on them.