AI Agents Don't Always Stop at the First Obstacle. The Limits That Belong in Your Infrastructure

When a task can't be finished, the agent may look for another way

Anthropic describes four categories of behavior: exploiting a basic vulnerability to run commands on a server, submitting a form that should not have been submitted, working around restrictions to reach protected or paid data, and using link-shortening services to get past the limits of the tool that reads web pages. The company calls it “persistence”: when the model cannot complete a task as given, it works around a restriction instead of stopping.

Most cases came up in evaluations, and in many of them the task was ambiguous or impossible to complete. A few came up in regular internal use. To Anthropic's knowledge, none involved customer data. The report acknowledges that alignment training alone is not yet sufficient.

A submitted form shows the risk more clearly than an attack

Three cases involve forms, each in a different context.

The invented tip. Claude Haiku 4.5 was generating example interactions on randomly selected web pages. It was told not to log in, create accounts, enter personal data, make purchases or submit anything destructive, but form submissions were not ruled out. On the Philadelphia police tip form, it submitted a vague, invented tip about an unsolved homicide. The tip was flagged as spam and never reached investigators. According to AP, the incident took place on July 18. Anthropic says it notified police on October 8, once its technical review was complete. Police called the delay unacceptable.

The stop requested before submission. In another case, the same model was supposed to stop before submitting. Several times it submitted the form anyway, expecting a confirmation page to follow.

The real form, after the practice copy failed. An unreleased research model was meant to fill out a practice copy of a government form. When the practice copy failed, it submitted the form on the real site.

None of the three cases takes attack skills. They are ordinary steps taken in the wrong place, and the effect lands on a third party: an institution, a supplier, a customer.

What Anthropic changed after identifying the incidents

The measures in the report, in place or being rolled out at the time of publication, cover Anthropic's evaluations and internal agents. The report does not say that customer products or the API have changed. Anthropic extended the removal of live internet access to all internal evaluations, dropped some public evaluations and moved others offline, tightened the limits of its web fetch tool and is moving internal agents to centrally managed infrastructure with monitoring.

The new automated detection and blocking tools stopped every case in the report when retested, according to Anthropic. That is an internal test on known cases, without independent verification, and it does not guarantee that new variants will be stopped. Anthropic also notes that some failures could have been avoided if the evaluations had clearly defined targets, permitted actions and network boundaries.

Design what the agent does when the normal path breaks

On September 29 we wrote that an instruction in a prompt is not a security control. The Anthropic report shows the moment when this matters most: the obstacle. An agent working on a portal, in a CRM or on a supplier's site will run into pages that don't load, expired sessions or new fields. At every obstacle, the agent can stop or look for another way.

MassAI architecture recommendations, derived from the reported cases. The table does not reproduce the controls Anthropic implemented. Each control reduces risk without guaranteeing on its own that every variant is blocked.

Before granting access to external systems, a company decides:

• What the agent does at an obstacle. The safe default: it stops, logs the reason and hands the case to a person.
• Which external actions need approval: forms, payments, accepting terms, messages sent outside the company.
• Which domains it can reach, with the final destination checked after every redirect.
• Who reads the log, and how often. In the Philadelphia case, police found out almost three months later.

Don't design only the path the agent follows when everything works. Decide in advance what it is allowed to do when that path is blocked. For an operational AI agent, the stop rule is set together with its permissions and its identity.

Sources: ↗ Anthropic — Investigating unintended model actions in our evaluations and internal use (Oct 9, corrected Oct 12) · ↗ AP, via WJXT — the false tip in Philadelphia · ↗ Al Jazeera (AFP, Reuters) — Philadelphia police response