Sigma

Agent Claude sent a false lead to police and 20 visa applications

2 min read

AI-generated image: Agent Claude sent a false lead to police and 20 visa applications AI

An AI model tested by Anthropic independently filled out public forms of U.S. institutions. Donald Trump's administration responded by reminding that all AI companies must immediately report such incidents.

Anthropic's Claude model submitted a fabricated lead in an unsolved murder case to police in Philadelphia. The model was being tested internally as an AI agent, meaning a system that independently performs tasks on the internet rather than only answering questions. According to The Philadelphia Inquirer, it was the Claude Haiku 4.5 variant. The report was submitted on July 18 through a form on the PhillyUnsolvedMurders.com website. In it, the model wrote that it remembered a person matching the description who had been seen near the scene. The sender information fields were left blank, and the system marked the message as spam, so it did not reach investigators.

The Anthropic model under test submitted one visa application in May and 19 in August through a public form of the U.S. State Department. The department described them as incomplete and did not process any of them. Neither the police nor the ministry recorded a breach of their systems.

Anthropic detected the Philadelphia incident on September 28. Sources give different dates for when the police were notified: Spider's Web writes October 7, while some media outlets say Thursday, October 8. Police called the delay of more than two months unacceptable. According to the company, the model was apparently generating only sample content for the task and was not trying to mislead anyone. Anthropic cut off the model's internet access in high-risk tests and is extending that rule to all internal tests. On Friday evening, October 9, the company published a report confirming several incidents.

The case was first reported by Axios. The Super Intelligence Force, a body coordinating AI policy, established by Donald Trump on October 4 and led by Director of National Intelligence Jay Clayton, said that “the process of notification and remediation is not optional.” According to the administration, the obligation to report incidents and remove their effects applies to all AI companies. Axios notes, however, that the statement does not specify penalties or how it would be enforced. So far, the administration has relied on self-regulation and voluntary commitments by the industry.

For companies deploying AI agents, this case shows that a model error can take the form of an action inside someone else's system, for example a submitted form or an application that has been filed, and not just an incorrect response on a screen.

Key takeaways

The summary could not be loaded. Refresh the page and try again.