Anthropic Seeks Global AI Safety Pause as OpenAI Calls for Government-Led Rules

Anthropic wants the major players in artificial intelligence (AI) to agree on a shared system that could slow or stop work on the most powerful AI tools if the dangers become too great, saying the pace of progress could make human control harder to maintain.

The maker of the Claude chatbot wrote in a Thursday blog post that, as frontier AI becomes much quicker at completing tasks, "it would be good for the world to have the option to slow or temporarily pause" further development.

Anthropic said its in-house research institute intends to study the matter with outside partners and "take actions" towards creating practical arrangements for a believable slowdown or halt, though it did not spell out what those steps would be.

OpenAI, one of Anthropic's main competitors, set out a different position in a report released Wednesday. It said "democratic governments, not private companies acting alone, must ultimately determine the rules, safeguards, and accountability mechanisms".

"Our view is that decisions about the pace of AI innovation should not be left to any one lab, company, or special interest group," OpenAI said.

In its post, Anthropic said AI systems are improving rapidly, including in their ability to complete software work such as coding without direct human handling. If present trends continue and enough computing resources are available, it said, an AI model could one day create and build its own replacement, a process known as "recursive self-improvement".

Anthropic said AI that can help construct its next generation would mark a major step forward and could support breakthroughs in science, medicine and other fields. However, the company warned that it "also might increase the risks of humans losing control over AI systems."

That concern has been raised for years by some figures in the technology sector. Anthropic's appeal follows another warning this week from University of Toronto researchers, who demonstrated how AI tools could be used to make a new type of AI "worm" that changes its hacking method while moving across devices and seizing a large computer network.

"I think it's really important that people understand that it's not just the biggest, most powerful language models that pose the security concerns," lead researcher Nicolas Papernot said in an interview.

The Anthropic blog post was written by company co-founder Jack Clark and Marina Favaro, who leads its research institute. They said any pause would give "societal structures and alignment research" more time to catch up with AI development. In the industry, alignment refers to efforts to ensure AI systems act in keeping with human values and aims.

Under the kind of arrangement Anthropic described, leading AI laboratories would be able to check whether international competitors had genuinely slowed or stopped their projects, "and that a bad actor could not use the auspices of a coordinated slowdown to jump ahead in secret."

Anthropic argued that a worldwide coordination system is necessary because, without one, any company that eases off could allow the "least cautious" operators to gain ground, increasing the strain on firms and governments facing difficult AI safety decisions.

The post comes as Anthropic and ChatGPT developer OpenAI are racing to sell shares on the stock market, with an IPO that could place Anthropic's value near US$1 trillion.

Before publishing his report, Papernot alerted Canadian cybersecurity officials. The report explains that the research team built the worm in a laboratory using an "open-source" AI tool that software developers can access and alter at low cost.

"In the past, cyber-attackers would focus on targets that are very high value," he said. "Banking systems, hospitals, electricity grids, water treatment systems, schools."

Papernot said he agreed that companies, government bodies and university researchers should work more closely on defensive measures as AI-assisted hacking tools make it far cheaper and faster to find weaknesses in computer systems.

"That old laptop you have in your basement that you don't check on regularly doesn't seem like a very high-value target, but it can be used as a launch pad to attack these higher-value targets," he said. "Anything connected to the Internet is now at risk because of how low the cost has become to mount these cyberattacks."