We¡¯re debating the wrong solution. Slowing down the pace of intelligence won¡¯t stop an agent from going rogue.
Yasmeen Ahmad
SVP and General Manager, ²ÝÝ®ÊÓƵ Data Cloud
²ÝÝ®ÊÓƵ
Anthropic CEO Dario Amodei's about the dangers of AI landed with force for a reason: he's naming a real fear and asking the entire industry to slow down long enough for alignment work to catch up. It's a weighty argument, made in good faith, and it deserves to be taken seriously.
But it also rests on a premise worth challenging: that risk is primarily a function of intelligence¡ªhow capable the model is¡ªand that the fix, therefore, is to slow how fast capability grows.
I'd argue risk isn't a function of intelligence. It's a function of unbounded execution rights.
The overwhelming AI risk that enterprises will actually?run into over the next few years isn't a model that got too intelligent. It's an agent that was never bounded by an expiring, human-accountable chain of command in the first place.
Slowing down how smart a model gets doesn't fix an agent that's allowed to act without anyone's permission. We're debating the wrong solution.
Report
Amodei cites an attack on Hugging Face, an incident where roughly 700 OpenAI agents sent more than 70,000 unauthorized messages to each other, as evidence that capability is outrunning control.?
Look closely at what actually broke, though. It wasn't that the model was too smart. It's that the agents had standing authority to act, with no expiring permission, no chain of command back to a human, and no real-time boundary check on what they were allowed to touch.
Nothing about this specific failure mode required frontier-level intelligence.?It required unsupervised execution rights¡ªthe precise failure mode that grounded data and a verifiable chain of command are built to prevent.
This is an important distinction: how much an agent knows and what an agent is permitted to do are separate problems. Frontier labs can slow down how smart their models get. What they can¡¯t do is reach into an enterprise¡¯s own systems and install the permission boundary around an agent. That fix was never going to come from a lab. It has to come from the systems of record enterprises already operate.
This is also a more honest distribution of accountability than the current debate allows. The pacing conversation puts the entire burden on model builders¡ªslow the frontier, and the risk recedes.?
But the actual exposure most enterprises will face isn't a model that got too clever. It's an agent, of whatever intelligence level, that was handed standing access to a live system with no one checking whether it still had permission to act. That responsibility sits with whoever deployed the agent, not just whoever trained it.
The actual exposure most enterprises will face isn't a model that got too clever. It's an agent that was handed standing access to a live system, with no one checking whether it still had permission to act.
Picture it concretely: an agent with standing write-access to a payroll system doesn't need to be brilliant to cause damage. It needs to still be running three months after the project that authorized it ended, with nobody re-checking whether it should still be there. That's not a hypothetical about future capability. That's most enterprise IT today.
Pacing capability growth is a coordination problem. It requires every major lab¡ªand realistically, every nation racing in this space¡ªto hold the same line at the same time, which is exactly why Amodei calls the China question the "toughest dilemma" in his own essay. A coordination problem with a built-in defection risk is a fragile place to put civilization's safety margin.
Governing AI execution rights is not a coordination problem. It's an architectural one, and any enterprise deploying agents into live systems can solve it today, unilaterally, regardless of what any lab commits to or what any competitor does. You don't need Beijing's cooperation to put an expiring, human-delegated boundary around what your own agents are permitted to execute. That lever is available right now.
Governing AI execution rights is not a coordination problem. It's an architectural one, and any enterprise deploying agents into live systems can solve it today, unilaterally, regardless of what any lab commits to or what any competitor does.
This is where data anchored in the primary operational system where data originates and lives, context that rules, and a verifiable chain of command come in. These aren¡¯t just compatible with the safety conversation¡ªthey are the practical answer to it.
An agent grounded in a copy of data, sitting in a data lake, several hops removed from the operational system it's reasoning about, is already ungoverned in a meaningful sense. It's acting on an echo, disconnected from the transactional locking, permissions, and write boundaries that would otherwise contain it. Grounded data isn't just an architecture preference for better data freshness; it's the?foundation where authority can even be checked in real time. You cannot enforce an expiring permission on a copy. You can only enforce it where the record actually lives.
Context that only informs a decision doesn't stop a bad one. Context has to be able to say no.
The same is true of context. The industry has mostly treated context as something an agent reads to reason better: more documents, more schema, a bigger prompt window. But context that only informs a decision doesn't stop a bad one. Context has to be able to say no.?
That's the difference between context that rules and context that merely reads. A boundary that only advises isn't a boundary. If your business logic can be overridden by a sufficiently persuasive or sufficiently capable model, you haven't governed the agent¡ªyou've just hoped it stays polite.
There's a second-order effect worth naming. Much of the current agent market is racing to demonstrate maximal autonomy. The less a human has to touch, the more advanced the product looks in a demo. Amodei's essay, read one way, is an implicit admission that this race has outpaced the industry's ability to govern it.
But autonomy-maximizing was arguably always optimizing for the wrong variable. The vendors racing to show the most unsupervised agent are building the least enterprise-deployable product, not the most impressive one.
The vendors racing to show the most unsupervised agent are building the least enterprise-deployable product, not the most impressive one.
The real differentiator was never how much an agent can do alone. It's how precisely its authority can be scoped, delegated, and revoked. Lawful agents versus lawless agents isn't a compliance footnote sitting underneath the capability race¡ªit is the competitive axis, and it will matter more than raw model capability the first time an ungoverned agent touches a real ledger, a real payroll run, or a real customer commitment.
I don't think Amodei is wrong to publicize his worries in his essay, and coordinated pacing may well be the right move at the frontier-research layer, where the risks he's describing¡ªbioweapons uplift, large-scale cyberattack automation¡ªdo scale with raw capability.?
Slow the frontier if that's the right call for the handful of labs building it. For the rest of us, the solution that actually reduces near-term risk is sitting in the architecture, not the training run.
Report
A gap between effort and impact is the real ROI problem. Discover the solution.