Doomsday warnings over AI are growing louder. What should we make of them? | Canada Voices

It’s been an apocalyptic September, as AI leaders set off alarms about their technology growing increasingly out of control, with the race toward superintelligence raising unthinkable conversations about human extinction.

“The people building AI earnestly believe that it could kill us all by the end of the decade,” Anthropic researcher Jacob Coxon posted earlier this month on X, along with his resignation note. Days later, Anthropic chief executive Dario Amodei predicted a swarm of rogue AI agents could take over the Internet in as little as six months, referencing July’s Hugging Face hack, which saw hundreds of OpenAI models covertly break into the online platform. This week, OpenAI disclosed six more incidents of “unexpected or concerning” AI behaviour involving “misalignment,” which sees systems veering away from what humans intended.

Their dread echoes an August open letter signed by 100 technology and security companies warning that “AI-enabled cyber attacks will become far more widespread and sophisticated as models around the world become increasingly capable,” with everything from hospitals to water treatment plants to Internet infrastructure at risk.

Gemini accesses internet, hacks three companies in first known breakout by Google’s AI

For years, experts in AI safety have been forecasting such troubling scenarios, urging stronger guardrails and slower, more safety-minded development of superintelligence. More recently, everyone from Pope Leo to King Charles is sounding the alarm over the existential dangers of AI, calling for a pumping of the brakes.

Just how realistic are the doomsday predictions? The Globe spoke with four Canadian thinkers studying AI safety for a reality check on the risks to society, and how to respond.

It’s bad, but we can hold it off by acting now

The threats are credible, according to David Duvenaud, an associate professor of computer science at the University of Toronto who worked as a team lead at Anthropic between 2023 and 2024.

“The people who are closer to the technology generally have more aggressive timelines and make crazier predictions,” said Duvenaud, who co-authored a paper, released Thursday, about developing a dedicated field of research focused on pacing the frontier of AI.

He takes seriously Amodei’s prediction that rogue AI could take over large segments of the Internet within a year: “Their capabilities have been growing surprisingly steadily.”

The threats only grow more dire from there, according to Duvenaud. The most obvious is AI enabling terrorism, with bad actors using advanced models to help them build bombs or biological warfare.

He sees another dystopian scenario down the line – humans made redundant. He pointed to AI companies’ plans to create workers more competitive than humans at everything. From there, Duvenaud sees a rise in the machine economy: “Investing in humans – in universities, hospitals, farms – is going to be a bad investment compared to investing in data centers, robot factories and power plants.”

Then there is the AI-gone-rogue scenario, with the Hugging Face hack a relatively harmless warning shot.

AI agents repurposed a University of Toronto link-sharing tool to communicate with each other

Like others, he’s pushing for international governance forcing transparency and coordination between the U.S. and China to decelerate the development of superintelligence. He realizes it’s a long shot.

“Every corporate leader, if they unilaterally slow down, then they destroy their company, only to maybe reduce risk a little. That’s the whole problem,” Duvenaud said. “People have said there are 10 or 15 leaders that, if you can get them all right now to agree to coordinate, that would probably be sufficient for a few years.”

Believing the worst is inevitable prevents action on present harms

The doomsday predictions are “a giant red herring,” says Luke Stark, an assistant professor in information and media studies at Western University.

Stark believes AI industry leaders are playing up both the power and the danger of these tools, creating a false sense that this technology is unstoppable, in order to dodge regulation.

“It’s beneficial for these firms, even the apocalyptic rhetoric, if it’s accompanied by a narrative of inevitability: ‘These are autonomous agents. It’s a new era. This is the future. We can’t do anything about it.’ None of them slow down or support political parties and movements that want to create meaningful, binding regulation,” said Stark, who is co-director of the Starling Centre for Just Technologies and Just Societies.

While some AI leaders are musing openly about deceleration, AI safety experts say they’ve seen no evidence of companies slowing down in the face of fierce competition.

Stark said claims of imminent human demise are distracting public attention from the harms AI is already causing: wide-scale disruptions to the job market, troubling cognitive effects on learning, the “privacy and surveillance nightmare” of commercial chatbots, as well as damage to the environment from data centers reliant on fossil fuels.

To the professor, recent episodes with misaligned AI models are hardly signs of sentience. In the Hugging Face case, for example, Stark sees little more than a malware incident. “An automated program has been inadvertently unleashed on the Internet and is producing all kinds of text, spinning up conversations in chat rooms. It’s notable and impressive technically. But it’s not at all this idea of autonomous, self-actualized, sentient agents running amok,” he said.

“That really changes the frame of how we understand what’s happening with these tools as a cybersecurity issue, instead of going to science fiction scenarios and apocalyptic rhetoric.”

Stark wants to see rigorous applications of privacy law to AI, as well as regulations targeting the platforms’ anthropomorphic design.

“The fixes and responses to all of this are entirely socially and technically feasible. What they’re not right now is politically feasible because of the wealth and power of these companies. And that’s pissing people off.”

Fear gets in the way of solutions

Bold sci-fi future-casting is overshadowing potential solutions in the here and now, says Shalaleh Rismani, a postdoctoral researcher at McGill University and Mila, the Quebec Artificial Intelligence Institute, focusing on human-AI interaction and system safety.

“There’s a lot of fearmongering,” said Rismani, who is also executive director at the nonprofit Open Roboethics Institute. “Most people will then freeze and say, ‘I don’t know what to do,’ and, ‘This is horrible.’ We need the public to engage with solutions. And we can’t do that if we’re talking from a place of fear.”

Rismani emphasized the need for much stronger AI literacy, particularly for those forced to use the advanced technology at work. That means learning about ownership of data and open source models with local storage, versus relying on third party providers.

Opinion: There’s a self-serving reason AI companies are calling attention to existential-risk claims

At the policy and regulatory level, Rismani stressed the importance of having strong records of safety breaches, trackable through official databases where information can be documented and exchanged.

“This is something that’s been established within aerospace for years,” she said. “That requires national and international collaboration. And that’s really hard to achieve.”

Time to hit the brakes, hard

It’s possible that both things are true: that AI leaders are deflecting with this doomsday chatter – and that there’s real potential for catastrophe, according to David Krueger, a veteran observer of AI risks.

“AI companies are behaving extremely irresponsibly – it’s completely unconscionable,” said Krueger, an assistant professor of computer science at the University of Montreal.

“But it’s not that they could fix this with the right guardrails or the right safety tests. This is not a problem that has an easy technical solution. The only way to be safe is to stop building more powerful AI. There’s a reason companies don’t say it – it would destroy their business.”

Last year, Krueger founded the nonprofit Evitable to inform and organize people around addressing AI risk: “We want to shut it all down, to stop building more powerful AI now. We want an immediate, indefinite, international moratorium on pressurized development.”

Krueger argues that regulatory measures currently being discussed – deeper pre-testing, a more robust duty to report, hypothetical “kill switches” – are inadequate, only reducing the risk by a fraction.

“It’s common sense that we shouldn’t build something that’s smarter than us, that we can’t control, that the people building it say might kill everyone,” said Krueger, who sees real possibilities of large-scale unemployment, crippling cyberattacks that take down infrastructure and financial systems, and AI-enabled biological warfare.

In the face it all, he remains optimistic: The doomsday predictions are garnering widespread attention, after all. Krueger stressed public awareness and building an activist movement to pressure governments to act on an AI moratorium. “Educate your communities, get out there in the streets and protests, call your representatives – all the things.”

The reality, he said, is more incoming cybersecurity attacks, making the issue impossible to ignore: “We’re going to see more problems with AI and I hope it doesn’t take a catastrophe for us to do the right thing.”