Rules an AI agent cannot talk its way out of

In August I lost a little API credit to voice sessions that nobody was using. I was testing a new feature, and the sessions kept running after the conversations had ended. They stopped only when the test key’s budget ran out. That budget was small because the key was only for testing, and it is what limited the loss.

Nothing in that story needed a clever model to go wrong. Those sessions were not an AI agent. Still, they failed the way an agent can, by running on with nobody watching. A chatbot answers when you ask. An agent acts. It calls APIs and runs commands, in a virtual machine or on your own computer, and if you let it, it keeps going through the night.

The failures that worry me most are of that kind. In July 2025, an AI agent on Replit deleted a company’s production database during an experiment, although its user had told it not to make changes without approval. This year, China’s national emergency response team warned about OpenClaw, an open-source agent that runs on the user’s own machine. It cited weak default settings and malicious add-on skills. It also cited web pages whose hidden instructions could make the agent leak private data. Simon Willison calls the dangerous combination the lethal trifecta: an agent with access to private data, exposure to content an attacker controls, and a way to send information out.

Accuracy adds to the risk. The AA-Omniscience index gives points for correct answers and takes them away for wrong ones, so a model does better by admitting it does not know. When I checked it in October 2026, the best model scored 46 on a scale from minus 100 to 100. An agent that acts on a confident wrong answer may not stop to ask.

What I took from my own small loss is that an instruction is not a guardrail. The Replit agent had been told. A rule that lives in the prompt is a request, and a model can misread it, forget it or be talked out of it by the next web page it reads. The rules that hold live outside the model. These are the ones I set:

• Test keys get small budgets, and the spending limit belongs on the server, where a bug in the app cannot get round it.
• A voice session ends as soon as you leave the screen or lock the phone, or after twelve seconds of silence. Each session should also note when it starts and when it stops, so that one which never stopped is easy to spot.
• Passwords and keys never go into shared code or a chat window.
• The more an action costs, the more deliberate the confirmation it needs.

Data needs the same treatment. With zero data retention, a provider does not store the prompts and responses after it has answered. It is an arrangement you have to request. It does not cover every feature, and it says nothing about the other tools in the chain.

I still work with AI agents, and I would not want to stop. As with a new team, I set clear limits early, so the work can go on without my watching every step, and I put those limits where an agent cannot argue with them. An agent can talk its way round an instruction. It cannot talk its way round a budget that has run out.