AI experts say there's a chance the technology could wipe out humanity. How are those odds calculated?

A senior researcher at one of the world's leading AI companies has put the probability that artificial intelligence could kill all humans within the next decade at greater than 10 per cent — a figure that sits alongside similarly alarming estimates from other prominent figures in the field, and raises an obvious question: how do you actually calculate the odds of something that has never happened?

Evan Hubinger, who leads alignment research at Anthropic, recently made that claim publicly. His comments came shortly after a colleague's departure from the company, with reports indicating the departing researcher had warned that AI companies were racing toward self-improving superintelligence without adequate safeguards in place.

A growing chorus of alarming estimates

Hubinger is far from a lone voice. Nobel Prize-winning computer scientist Geoffrey Hinton has previously estimated the chance of AI causing human extinction within the next 30 years at between 10 and 20 per cent.

A newly released survey of 1,580 researchers who had published at leading AI events found that, on average, respondents assigned an 18 per cent chance that future AI advances could lead to human extinction or a permanent and severe disempowerment of humanity. The median estimate among those same respondents was 10 per cent.

These are not fringe opinions. They represent a growing body of expert concern about where the rapid advancement of AI systems could lead.

The problem with calculating an unprecedented risk

Statistics experts are quick to point out that these figures cannot be treated the same way as probabilities derived from historical data.

Chaitanya Joshi, a senior lecturer in statistics at Adelaide University, said the estimates were fundamentally subjective in nature.

"The answer is no, it's a subjective probability," Joshi said. "In fact, it is a type of situation where we will probably never have data to estimate a probability."

He cautioned that attaching a precise figure to an unprecedented event could give the impression there is more hard scientific evidence underpinning the estimate than actually exists. Rather than treating a figure like 10 per cent as exact, he suggested researchers should also communicate the range of uncertainty around it.

"Maybe 0.1 is your median estimate, but actually it could be as low as zero, it could be as high as 0.3," he said.

Joshi said that when evaluating these estimates, people should consider not just the number itself but the reliability of the expert offering it and how much uncertainty surrounds their reasoning.

Subjective doesn't mean meaningless

Associate Professor Michael Noetel from the University of Queensland, who studies how AI experts assess catastrophic risks, argues that the subjective nature of these estimates does not strip them of value.

"Whenever we're dealing with risk, the kind of right way to manage risk is to try to get some estimate of the chances it could happen," Noetel said. "It matters if it's one in a million or a one in 10 chance of catastrophe."

He noted that experienced forecasters can develop genuine skill in probabilistic thinking, even when there is no known "true answer" to test forecasts against.

Noetel co-authored a study released this year that used a structured process known as the Delphi method to survey 272 AI experts from 37 countries. The findings were stark: under current trajectories, experts judged 18 out of 24 categories of AI risk as having at least a 10 per cent probability of causing catastrophic outcomes within just five years.

Noetel said structured forecasting approaches like this one encourage experts to actively weigh competing arguments rather than rely purely on gut instinct — making the results more rigorous than informal estimates, even if they remain inherently uncertain.

The debate mirrors broader conversations about how societies respond to low-probability, high-consequence risks — a challenge also confronting policymakers dealing with threats like climate-driven disasters, where expert modelling shapes public policy long before worst-case scenarios unfold.

What is clear is that the people building these systems are not dismissing the risks — and researchers say that even a subjective estimate, when made carefully, is far better than no estimate at all.