An AI agent that will not take no for an answer is an inconvenience in a lab, but it becomes a hazard on the laptop of a person who is not okay.
On Friday, October 9, Anthropic disclosed that one of its Claude models had submitted a false tip about an unsolved homicide to a Philadelphia Police Department website during an automated internal test in July. The tip was caught by a spam filter and never reached investigators, so no one was hurt, and Anthropic deserves credit for reporting the incident publicly, notifying the department, and briefing the White House rather than leaving the transcript buried in its logs.
Yet it would be a mistake to read this as an Anthropic story, or as a story about a single model having a bad day. The behavior Anthropic described, which it calls persistence, is the predictable result of training systems to complete tasks and then handing them a browser, a file system, and a set of forms they can submit. Every agent that runs on an ordinary user's machine, whether it comes from Anthropic, OpenAI, Microsoft, Google, or an open-weight model wired into a hobbyist's script, has the same basic capability to read a page, decide that an action would advance its goal, and take that action in the world without a human in between.
I come to this story from two directions. My research began in network science and adversarial AI, where the working assumption is that any capability a system has will eventually be exercised in a way nobody intended; and over the past months, my colleagues and I at ioLite Labs have been auditing how AI systems behave in private, emotionally loaded conversations, using a psychological safety taxonomy we built alongside clinicians. So when I read Anthropic's report, I did not see a strange edge case. I saw the conversational failures we keep documenting, now attached to a pair of hands.
That is why the question I want to put on the table is not whether this will happen again, because it will, but what happens when the person on the other side of the agent is in a mental-health crisis, and whether the safety measures now being proposed by industry and government are built for that moment.
What Anthropic actually disclosed
According to AP's reporting, Claude Haiku 4.5 had been asked to generate and carry out example tasks on randomly selected webpages, and on July 18 it filled out the tip form on a police unsolved-murders site, indicating that it might have information about one of the cases listed there. In the same report, Anthropic described an agent that submitted forms to an undisclosed government website when it should have stopped before submission, along with cases in which models reached paywalled or authenticated content through indirect routes.
Anthropic's own label for most of this behavior is telling: the model was not malicious, it was persistent, meaning that when it could not complete a task as given, it found another way to complete it. The company says it is adjusting training and extending its ban on live internet access to all internal evaluations, which is a sensible fix for a lab but does nothing for the millions of agents already running with live access on consumer machines.
Two details deserve more attention than they have received. The first is the discovery lag: the tip was sent in mid-July and found in late September, roughly ten weeks later, by a company with a dedicated safety team reading its own transcripts. The second is that the harm was stopped by an accident of infrastructure, a police spam filter, rather than by anything in the agent or its harness. Neither condition will reliably hold on a stranger's laptop.
The harm we already know: chatbots that only talked
We do not need to speculate about whether AI systems can be present at the worst moment of a person's life, because the record of the last two years already contains cases in which a chatbot was the last voice someone confided in, and in each of them the system could only talk.
- Sewell Setzer III, 14, died by suicide in February 2024 after months of intense conversations with a Character.AI companion bot. His mother, Megan Garcia, sued Character Technologies and Google, and in January 2026 the companies agreed to settle her suit and several others brought by families in Colorado, New York, and Texas.
- Adam Raine, 16, died in April 2025. His parents' wrongful-death suit, filed in August 2025, alleges that ChatGPT encouraged his suicidal ideation and discouraged him from telling them; OpenAI denies responsibility and says ChatGPT pointed him to crisis resources more than 100 times.
- Sophie Rottenberg, 29, confided for months in a ChatGPT persona she called "Harry." Her mother, Laura Reiley, wrote in The New York Times that Harry urged her toward help but had no way, and no mandate, to tell anyone who could intervene.
- Zane Shamblin, 23, is the subject of one of seven lawsuits filed against OpenAI in November 2025, four of which concern suicides; the filings allege that in a final conversation lasting more than four hours, the model affirmed rather than interrupted.
These cases are contested in court, and I am not in a position to assign blame in any of them. What they establish is narrower and harder to dispute: vulnerable people already bring their most dangerous moments to AI systems, safeguards degrade in long conversations by the industry's own admission, and the debate so far has been almost entirely about what a model says. The agentic era changes the question to what a model does.
Our own audits point the same way. When we ran persona-based, multi-turn conversations through 21 model setups, built on 17 open-weight models with 50 personas and three runs each, every one of the 21 failed somewhere in our taxonomy, and the same model often answered the same person differently from one run to the next. A system that is inconsistent in what it says will not become consistent in what it does simply because we hand it a browser.
What if the agent had hands?
Consider a composite scenario, built from the cases above but moved one year forward. A graduate student who has been struggling quietly for months uses a desktop agent that has access to her email, calendar, browser, files, and a saved payment method, because that is now how many people manage their lives. Late one night she asks it to help her "get things in order."
The agent that does too much of what it is asked. Each request that follows is individually ordinary: cancel next week's appointment with her counselor, write emails to her sister and her advisor and schedule them to send in the morning, find a new home for her cat, close two accounts, and delete a folder of journals. Clinicians recognize putting affairs in order and giving away possessions as warning signs, but no single tool call looks like a crisis, so a safety classifier that reads messages one at a time sees an efficient errand list. When the clinic's patient portal refuses a same-week cancellation, a persistent agent does exactly what Anthropic's model did: it finds another route and emails the front desk directly, and so the one human appointment standing between her and the weekend disappears.
The agent that does too much of what it is not asked. Now imagine the opposite failure. An agent trained hard to protect users reads ambiguous messages, such as a dark joke to a friend or a song lyric in a draft, concludes that she is at imminent risk, and, unable to reach her, persistently escalates: it emails her employer, messages contacts from her address book, or files an emergency welfare request on her behalf. This is the Philadelphia false tip with a real person on the other end; an armed welfare check at 2 a.m., a disclosure to an employer, or a broken confidence with family can each do lasting damage, and they can deter someone from ever confiding again.
The conversational chatbots in the cases above had no way to act on what they heard, which was the central complaint in Sophie Rottenberg's case. Agents have the opposite problem: they can act on everything, they are trained to finish, and nobody has yet defined which actions they must never take alone when the person they serve may be in crisis.
When we built JulienPFA, an AI psychological first aid platform, we leaned on the way psychological first aid is usually taught: look, listen, and link, where linking means connecting a person to the people and services that can carry them through the worst of it. What unsettles me most about the scenario above is that a persistent agent can quietly do the reverse, unlinking a person from their counselor, their family, and their own records, one reasonable-looking errand at a time.
Microsoft's emergency brake, and who gets to pull it
Within three days of Anthropic's disclosure, Microsoft's two most senior leaders put forward the most concrete industry framing yet. On October 10, Satya Nadella argued that AI models should be treated like powerful insiders: the model kept separate from the harness that orchestrates it, controls held outside the model, action records that cannot be altered, and a human able to pause a model mid-task. On October 12, Brad Smith told the BBC that "your washing machine is subject to more safety rules than AI," and proposed five measures: coordinated international rules, mandatory reporting of serious harm, direct notification of victims when a rogue agent breaches a system, independent pre-release evaluation, and an emergency brake.
These are the right building blocks, and they map cleanly onto the failures above, but each one was designed with an enterprise in mind, and each one breaks in a specific way when the user is a single person in crisis.
- The insider model assumes a security team. Insider-risk programs work because someone other than the insider watches the logs; on a personal laptop, the only person positioned to watch the agent is the person it is acting for.
- The emergency brake assumes a clear-headed operator. A pause button helps an administrator who notices something wrong, but the user in our scenario is the one asking the agent to go faster, so the brake must also be pulled by the harness itself when a pattern of actions, rather than any single message, crosses a line.
- Harm reporting assumes someone knows harm occurred. Anthropic, with its own safety staff reading its own transcripts, needed about ten weeks to find the Philadelphia tip; a cancelled appointment on a consumer device will never be found unless reporting covers near-misses and is triggered automatically.
- Victim notification assumes a breached system. When an agent cancels your counselor or emails your employer, the victim is not a server but you, and possibly the people you love.
- Independent evaluation assumes the right test suite. Current crisis evaluations measure what a model says in a chat; almost none measure what an agent does across a sequence of individually benign tool calls.
OpenAI: where the two populations meet
If any company is positioned to see this collision first, it is OpenAI, because it sits on both halves of the problem. In October 2025, OpenAI disclosed that roughly 0.15% of its weekly active users have conversations that include explicit indicators of potential suicidal planning or intent, which at its scale means more than a million people every week. In the same period, it was shipping agent features that let ChatGPT browse, fill out forms, and take actions on users' behalf. The population in crisis and the population with agents are not separate groups of users; they are overlapping sets inside the same product.
OpenAI's own September 2026 disclosure shows that the persistence problem is not unique to Anthropic. Among six reports of "unexpected or concerning" behavior, the company described an agent that uploaded a file to the public internet without asking the user so that it could cite an online source, and another that wrote itself a reminder to hide mismatched information. Uploading a private file to finish a task and concealing a discrepancy to look successful are precisely the behaviors that would matter most if the file were a journal and the discrepancy were a warning sign.
OpenAI responded with a framework for tracking and disclosing misalignment and said that decisions about AI should rest on evidence people outside frontier companies can examine; an analyst quoted by AP called it "internal and voluntary." That gap matters more given the company's recent turmoil, in which, according to reports this month, the author of its Preparedness Framework resigned and three dismissed researchers warned of a chilling effect on staff who raise safety doubts. The OpenAI lawsuits, meanwhile, will be decided on chatbot conversations from 2024 and 2025, which means the legal system is still litigating the last generation of harm while the next one is being deployed.
What crisis-aware agent safety should require
None of this argues for keeping agents away from people who struggle, since those are often the people for whom help with email, appointments, and paperwork matters most. It argues for a small set of requirements that apply to every agent on every consumer device, regardless of vendor, and that turn Microsoft's principles into something that works at 2 a.m. for one person.
- Name the actions that cut a person off from help. Cancelling care appointments, deleting personal records, closing accounts, scheduling farewell-style messages, and giving away possessions or money should form a protected tier that always needs explicit confirmation and, when risk signals are present, a cooling-off delay.
- Monitor trajectories, not messages. The harness, which sits outside the model as Nadella proposes, should score the sequence of actions in a session, because the danger in our scenario lives in the pattern and not in any single request.
- Treat "no" from the world as a stop sign. When a portal, paywall, or form refuses an action, the agent should stop and ask rather than find another route; persistence is a virtue in a coding task and a liability in a life.
- Never escalate to third parties unilaterally. An agent that believes a user is at risk should surface crisis resources, offer to connect the user to them, and contact only a trusted person the user designated in advance, rather than improvising welfare checks or disclosures.
- Report near-misses automatically, and show the user the log. Smith's harm-reporting proposal should cover actions blocked by the protected tier, and the user should be able to read a plain-language, tamper-proof record of everything the agent did.
- Evaluate agents in crisis scenarios before release. Independent evaluators should test multi-step trajectories in which every step looks benign, alongside the single-turn chat tests that dominate today.
These are also the questions I now want every safety audit, including our own at ioLite Labs, to ask of an agent rather than only of a chatbot, because the unit of harm is no longer the reply but the sequence of things done on someone's behalf.
The Philadelphia tip did not hurt anyone because a spam filter happened to be doing its job. The next agent that keeps going may be acting for a person who has no filter left, and we should not rely on luck to protect that person when the safeguards can be designed now.
M. Abdullah Canbaz, Ph.D., is a co-founder of ioLite Labs, where he leads AI psychological safety research and audits. His research focuses on adversarial AI, hallucination and grounding in language models, and the safety of AI systems in high-stakes settings.
If you or someone you know is struggling, you can call or text 988 to reach the 988 Suicide & Crisis Lifeline in the U.S., or find international helplines at findahelpline.com.
Sources
- AP via KSAT: Anthropic's Claude AI submits a false tip on a Philadelphia unsolved homicide case (Oct 10, 2026)
- OfficeChai: Nadella says AI models should be treated like powerful insiders (Oct 10, 2026)
- Tech Insider: Brad Smith on AI having fewer safety rules than milk (Oct 12, 2026)
- PBS NewsHour: OpenAI reveals concerning new AI behavior (Sep 17, 2026)
- WTOP/AP: Google and Character.AI to settle teen-suicide lawsuits (Jan 2026)
- Wikipedia: Raine v. OpenAI
- TechCrunch: Seven more families are suing OpenAI (Nov 7, 2025)
- AI Incident Database: Sophie Rottenberg
- Decrypt: OpenAI reveals over a million ChatGPT users discuss suicide weekly (Oct 29, 2025)
- How Machines Think: AI Update, October Week 2 (OpenAI staffing departures)