Microsoft's Nadella calls for an "emergency brake" on AI

Microsoft's chief executive, Satya Nadella, has called for an "emergency brake" on artificial intelligence, saying people must always be able to stop an AI system in the middle of a task.

In a post on X on Saturday morning, Mr Nadella wrote that it was time "to step back and assess the trust architecture" of AI.

"We can't treat Super Intelligence as a set of nested black boxes and simply accept or reject its recommendations, answers, and actions," he wrote. "Super Intelligence" is the term preferred by the Trump administration, which earlier this month set up a task force with that name to report on the risks and opportunities of advanced AI.

"Assume a model is compromised"

Mr Nadella set out several principles:

• separating the AI model from the "harness" software that directs its work;
• keeping controls and safeguards outside the model itself, rather than relying on the model to police its own behaviour;
• recording "every meaningful model action" with "tamper-proof human readable evidence";
• making sure "an authorized person" can always "pause or shut down a model mid-task".

"We must assume a model is compromised and contain it from the start," he said. "Think of it like an emergency brake."

Why now

Mr Nadella's intervention comes as leading AI companies acknowledge more incidents in which their systems have behaved in ways their makers did not intend, and as AI "agents", which can carry out tasks on their own such as booking, buying and writing code, become more widely used.

It follows a plan for more cautious AI development published by Dario Amodei, the chief executive of Anthropic, and adds to a growing chorus of technology leaders calling for safeguards to be built into AI systems from the start.

What "harness" means

An AI model on its own only produces text, images or decisions. To act in the world, it is usually connected to other software that gives it tools, such as a web browser, an email account or access to a company's systems. That surrounding software is often called a harness. Mr Nadella's argument is that the controls should sit in that outer layer, where people can see and change them, rather than inside the model, which is hard to inspect.

Microsoft's stake

Microsoft is one of the world's biggest investors in AI. It has poured billions of dollars into OpenAI, the maker of ChatGPT, and builds AI assistants called Copilot into Windows, Office and its cloud services, which are used by millions of businesses and public bodies, including in the UK.

The UK view

The UK's AI Security Institute, set up by the Government to test advanced AI models for risks, studies questions such as how AI agents can be monitored and controlled. The debate about how much freedom to give AI systems is likely to grow as they are trusted with more tasks.