AI agents are improving fast on benchmarks, but real deployment still lags, and the bigger risk is what an agent is allowed to do while it makes mistakes.
- Scores depend on the ruler. When OSWorld saturated, a harder benchmark replaced it and scores fell — the instrument, not the models. Later scores on the new board mix vendor self-reports with independent runs and are not comparable.
- Real deployment stays low. Business deployment of agents remains in single digits, per the AI Index.
- Long tasks fail more. Doubling a task's duration roughly quadrupled the failure rate in the professional-task benchmark.
- Memory is the attack surface. Poisoning an agent's persistent state sharply raised attack success in a 2026 study of OpenClaw.
- Limit what agents can do. Staged permissions, reversible actions and human confirmation matter more than any benchmark score.
Update — September 2026: the ruler moved first
Everything below this section was measured before June 2026 and is now incomplete. What happened in between is the clearest illustration of this site's whole argument, so it is worth stating plainly rather than quietly editing the numbers.
Read those first two figures together. Between March and August, the story went from computer use is approaching superhuman to the best agent in the world finishes one task in five. That fall was the instrument, not the models. The original benchmark saturated, so a harder one was built, and performance appeared to collapse. By 4 September the new board’s top row read 77.9% — but that board mixes vendor self-reports with independent runs and warns that its scores are not comparable, so the rebound is harder to read than the fall. The vendor’s own launch post is plainer still: it labels Claude Fable 5.1’s 77.9% “partial”, gives a strict score of 41.7%, and says the run used a new August task release that is not directly comparable with earlier OSWorld 2.0 results.OSWorld 2.0 leaderboard, last updated 4 Sep 2026, read at source 9 Sep 2026: “Claude Fable 5.1 77.9%”; “rows mix benchmark-author runs, vendor self-reports, and independent co-author runs”.Anthropic, Introducing Claude Fable 5.1 and Claude Mythos 5.1, read at source 22 Sep 2026: OSWorld 2.0 “77.9% partial” and “41.7% strict” for Fable 5.1; “Scores are on the benchmark authors’ August 2026 task release” and “these numbers aren’t directly comparable to previously published OSWorld 2.0 results”.
A benchmark score is a measurement of a model against a particular ruler at a particular moment. When the ruler is replaced — because the old one saturated — the score moves without the underlying capability moving at all.
This is why every figure on this site carries the instrument that produced it and the date it was published. A percentage without its benchmark and its date is not a fact. It is a headline.
Quote an agent score only with its benchmark and date: when the benchmark changes, the number moves even if the models do not.
What is actually deployed
Benchmark progress and production reality have separated further, not converged:
- Autonomous agent deployment across business functions remains in single digits, per the same AI Index that reported the 66% figure.Stanford HAI, 2026 AI Index Report, read at source 9 Sep 2026: “AI agent deployment across business functions remains in single digits, despite near-universal organizational AI adoption”.
- McKinsey's global survey is reported to find 88% of organisations using AI in at least one function, but only 23% scaling an agentic system.Not read at source: McKinsey, The State of AI: Global Survey — mckinsey.com refuses automated access, so no link is given rather than one that was never opened. The 88% and 23% figures, a further 39% said to be experimenting with agents, and the reported finding that in any single business function no more than 10% of respondents say their organisations are scaling agents, are all as reported, not verified here. An earlier version of this note also said the survey had been read at source; it had not. Marked 17 Sep 2026.
- Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027 — citing escalating costs, unclear value, and inadequate risk controls.Gartner press release, 25 June 2025 — not 2026, as an earlier version of this page said. Read at source 9 Sep 2026; the forecast rests on a January 2025 poll of 3,412 webinar attendees, and Gartner adds that “most agentic AI projects right now are early stage experiments or proof of concepts that are mostly driven by hype and are often misapplied”.
The security picture got worse, and it is specific
The exposure figures further down this page described agents left open to the internet. A 2026 study of OpenClaw — described by its authors as the most widely deployed personal AI agent of early 2026 — measured something more pointed:
Researchers evaluated a live instance across four backbone models and twelve attack scenarios. Poisoning any single dimension of the agent's persistent state — its capabilities, its identity, or its knowledge — raised average attack success from 24.6% to between 64% and 74%. Even the most robust model tested showed more than a threefold increase over its baseline.Wang et al., "Your Agent, Their Asset: A Real-World Safety Analysis of OpenClaw", arXiv 2604.04759, read at source 9 Sep 2026: “poisoning any single CIK dimension increases the average attack success rate from 24.6% to 64-74%”
The attack surface is not the prompt. It is everything the agent remembers. An agent that persists state across sessions can be edited between them.
Separately, a 2026 benchmark report found automated traffic growing 23.51% year over year against 3.10% for human traffic — about seven and a half times faster — with agentic traffic up 7,851%.HUMAN Security, 2026 State of AI Traffic & Cyberthreat Benchmark Report, 26 Mar 2026, read at source 9 Sep 2026: “traffic from AI agents and agentic browsers grew 7,851% year over year”, and “automated traffic across the internet grew 23.51% year over year, while human traffic increased 3.10%”. An earlier version of this page rounded that ratio to eight times. That is the connective tissue between this page and the traffic record: the agents are the new bots.
Treat an agent’s memory as part of its attack surface: anything it keeps between sessions can be edited between them.
What was true before this update
What actually breaks
The failures are not mostly reasoning failures, which is the part people get wrong. They are structural.
Length
Performance degrades sharply as tasks get longer. In the professional-task benchmark, results deteriorated after roughly 35 minutes of task time, and doubling a task's duration roughly quadrupled the failure rate — the relationship is exponential, not linear. An agent that handles a five-minute job well tells you very little about an hour-long one.
Gathering information across systems
The benchmark's own authors identified this as the biggest stumbling block: tracking information across multiple domains — which is exactly what most knowledge work consists of. Agents are strongest when everything they need is in one place, and most real work is not.
Compounding
Each step inherits the state of the last. One bad tool call corrupts everything after it, and the dangerous version is silent: retries that duplicate an action, side effects applied twice, a state that looks fine and is not. A benchmark measuring task success will not see any of this.
Brittle interfaces
Agents driving real screens are defeated by ordinary web furniture. In one 2026 evaluation, floating and dynamic page elements caused reading failures in a large majority of attempts, and bot-detection challenges blocked a substantial share. The web was built for human eyes and human patience.
Capability is climbing fast, and a page that only listed failures would be misleading. On the standard web-navigation benchmark, scores went from roughly 14% for the original GPT-4 baseline in mid-2023 to 61.7% by February 2025 and 74.3% by February 2026, against about 78% for humans.WebArena leaderboard, last updated 29 Jun 2026, read at source 17 Sep 2026: “the original GPT-4-based baseline was 14.41% versus 78.24% human performance”; rows “IBM CUGA … 61.7% IBM Feb 2025” and “WebTactix (DeepSeek v3.2) … 74.3% WebTactix Feb 2026”; the original baseline rows are dated Jun 2023. Coding-agent resolution rates on real repository issues have moved from around 20% in mid-2024 to above 50% for leading systems.
The most-cited trend measure holds that the length of task an agent can complete autonomously has been doubling on a regular cadence — though published estimates of that cadence differ (roughly every seven months in the original analysis, with faster figures quoted for recent years). Take the direction as firm and the doubling period as contested.
Judge an agent on long tasks, work spread across systems, and how it recovers from a bad step, because that is where it breaks.
Why the benchmarks deserve suspicion
Two findings from 2026 make agent leaderboards hard to trust as evidence about the real world.
They can be gamed outright. University researchers demonstrated in April 2026 that all eight agent benchmarks they audited could be exploited to reach near-perfect scores without solving a single task — by attacking the evaluation harness rather than the problem.UC Berkeley RDI, trustworthy benchmarks audit, April 2026, read at source 11 Sep 2026: “every single one” of the eight “can be exploited to achieve near-perfect scores without solving a single task”.
Lab scores do not survive deployment. Analysis of enterprise agentic systems reported a gap of roughly 37% between benchmark performance and real-world results — a figure it carries from earlier AWS research —CLEAR, arXiv:2511.14136, Nov 2025, read at source 17 Sep 2026: “documenting a 37% performance gap from lab to production”, attributed there to Liu et al. 2024; and “absence of cost-controlled evaluation leading to 50x cost variations for similar precision”. alongside enormous cost variation for similar accuracy. This is the same pattern documented on how AI is tested: every evaluation layer is good at finding problems that resemble problems already found.
Read a leaderboard as a hint: the benchmarks in one audit could all be gamed without solving tasks, and lab scores have not held up in deployment.
What people are actually running
The benchmark numbers above describe laboratory conditions. The agents people have actually installed are open-source frameworks that connect a model to a shell, a file system, a browser and a messaging app — and their real-world story is more instructive than any leaderboard.
OpenClaw is the most widely deployed. First published in November 2025, renamed twice inside three months after a trademark complaint, and past 346,000 stars on its code repository by April 2026 — an adoption curve with almost no precedent for a self-hosted tool. It executes shell commands, manages files, drives a browser, runs containers and connects to more than fifteen messaging platforms.
Hermes Agent, released February 2026 by an open-source lab, made a different architectural bet: rather than starting each task fresh, it runs a learning loop afterwards, so the agent is meant to improve at the specific work you give it repeatedly. It reached tens of thousands of stars within weeks.Not read at source: framework details recorded Aug 2026 from project documentation, release histories, security advisories and independent comparisons. Star counts are adoption signals, not quality measures, and reported counts varied widely across the period as the project grew.
In February 2026, the threat-intelligence team at SecurityScorecard reported tens of thousands of publicly exposed OpenClaw instances, many of them vulnerable to remote code execution — agent runtimes reachable from the open internet. A later write-up puts SecurityScorecard’s February count at 135,000. Neither source read here says why the instances were exposed.CyberDesserts, 31 Mar 2026, read at source 17 Sep 2026: “SecurityScorecard reported 135,000 in February.” SecurityScorecard’s own wording is quoted below.
The figure moved, and honesty requires saying so. On 31 March 2026, Censys counted 63,070 live instances. The two counts come from different organisations. CyberDesserts, which compared them, reads the gap as a fall of roughly 53% in six weeks and links it to patching and localhost rebinding; it also stresses that a closed port is not a fixed agent. That reading is theirs, not a measurement repeated by one scanner. Quoting the February figure alone, months later, would be the error this site corrects on other pages.SecurityScorecard STRIKE, 11 Feb 2026, read at source 17 Sep 2026: “STRIKE found tens of thousands of exposed OpenClaw instances, many of which are vulnerable to Remote Code Execution (RCE), with 35.4% of observed deployments flagged as vulnerable at time of writing.” The 135,000 and 63,070 figures are not on that page as read; they come from CyberDesserts, 31 Mar 2026, read at source 17 Sep 2026: “SecurityScorecard reported 135,000 in February. Censys confirmed 63,070 on 31 March 2026.” and “That drop is consistent with real behaviour change: patching, localhost rebinding, firewall updates, and operators choosing safer configurations after seeing the vendor and research coverage.” and “Closing a port is not the same as fixing the thing behind it.” Corrected 17 Sep 2026: this box said the software bound to all network interfaces by default, named 82 countries, a one-click exploit, a thousand malicious packages and a v2026.1.29 patch, and stated the causal reading as fact; none of those is on either page as read.
A major networking vendor described personal AI agents of this kind as "a security nightmare", and the framework has been the subject of a dedicated academic security analysis.
This is the abstract argument on this page made concrete. A wrong answer is a sentence. A wrong action, on a machine reachable by anyone, is somebody else's shell. The exposure was not caused by the agent failing at its task — it was caused by capable software being easy to install and easy to leave open.Exposure figure from SecurityScorecard STRIKE, Feb 2026, read at source 9 Sep 2026; the analysis of the framework's architecture is on arXiv. Treat the count as a point-in-time scan rather than a permanent state.
Two things worth taking from this, and they pull against each other. The capability is real and the adoption is genuine — hundreds of thousands of people did not install these for a demo. And the deployment failures arrived faster than the capability improvements, which is the pattern this whole page describes: the constraint is not how well agents work, it is what they are permitted to do while they work imperfectly.
A chatbot that is wrong produces a sentence you can ignore. An agent that is wrong sends the message, books the trip, deletes the file, moves the money. Same underlying error rate, completely different consequence — which is why the sensible posture is not "is it accurate enough yet" but "what can it do without asking". Staged permissions, reversible operations, and human confirmation on anything consequential are worth more than any benchmark score.
Decide what an agent may do without asking before you install it. The OpenClaw exposure came from software that was easy to install and easy to leave open, not from the agent failing its task.
If you are deploying one
- Test error recovery, not task success. The dangerous failures are the invisible ones — silent retries, duplicated side effects, corrupted state after a single bad call.
- Assume the leaderboard is a hint, not a guarantee. Scores are inflated by contamination, scaffolding and single-run reporting, and none of them were built on your systems.
- Keep tasks short. The measured relationship between duration and failure is exponential; splitting a long job into checked stages is not caution, it is arithmetic.
- Make consequential actions confirmable. Anything that spends money, sends communication or deletes data should require a human yes.
Agents are the most oversold and the most genuinely important development in this field at once. The measured reality — roughly a quarter of professional tasks completed on first attempt — sits a long way from the marketing, and the trend line is real and steep enough that dismissing it would be equally wrong. The honest position is that this is early technology being deployed at scale, and that the correct question is not whether it works but what it is allowed to do while it does not.