Claude found 4,000 fraudulent accounts hiding as a bug

Anthropic has a team of engineers whose job is keeping Claude running. One of them spoke at QCon London this week about what happens when you put the same model doing the job as the system it's trying to diagnose, according to The Register.

Alex Palcuie was an SRE for Google Cloud Platform before joining Anthropic's AI reliability engineering team. His job now: keeping Claude up. Since January, he's been reaching for Claude before other monitoring tools when incidents happen. His assessment after several months: useful, but a poor substitute for an experienced site reliability engineer.

"Claude excels at finding issues but still makes a poor substitute for a site reliability engineer," Palcuie said at the conference. "It constantly mistakes correlation for causation."

THE FRAUD CASE

Palcuie's positive anecdote is specific enough to be credible. On New Year's Eve, Claude Opus 4.5 was returning HTTP 500 errors. Palcuie opened Claude Code and asked it to have a look. The AI wrote a SQL query and within seconds identified an unhandled exception in the image processing class. It didn't stop there — it checked the accounts sending the failing requests and found 200 accounts all sending 22 images simultaneously. That looked suspicious. Claude kept digging and found 4,000 accounts all created at the same time, most sitting dormant. Its conclusion: stop looking at the 500s, this is fraud.

Without AI, Palcuie said he would have marked the incident as a bug and paged the wrong team. With AI, he had the fraud pattern in under a minute.

THE CACHE PROBLEM

The negative anecdote is more instructive. Anthropic's AI infrastructure relies on a key-value cache for performance — a component Palcuie described as "finicky" and "fragile," capable of being "really easy to break." When the KV cache breaks, it causes extra compute and monitoring shows more requests. Every single time, Palcuie asked Claude what happened. Every single time, Claude said the same thing: request volume increase, this is a capacity problem, you need to add more servers.

The actual problem was a cache failure. The correlation — more requests, more compute — was real. The causation — capacity shortage — was wrong. Claude kept seeing the same pattern and offering the same wrong answer.

"This is why we can't trust LLMs for incident response," Palcuie said. "The problem is its inability to step back and start discerning between causation and correlation... For us humans, it is hard as well."

THE POSTMORTEM PROBLEM

When Claude is asked to produce a postmortem report, it delivers "an 80 percent story that's pretty, it's readable and convincing," Palcuie said. But it's "really bad at root causes." Claude says "this was the thing" when everyone in the room knows it wasn't one thing — it was the processes in the company that allowed the incident, the accumulation of decisions that made the failure possible. "It was never the rollout. It was never the code change. It was all the processes." And Claude doesn't know the history of a system, especially one that's been running for ten years.

Scar tissue matters. Palcuie worries that if AI does more incident response, "will we have our skills atrophy?" — in parallel with concerns developers express about AI writing most of their code.

THE JEVONS PARADOX OF INCIDENTS

Palcuie's theoretical frame is the Jevons Paradox, which he called "the favorite paradox in the AI industry." When technological improvements increase efficiency, the resulting lower cost causes consumption to rise rather than fall. In software: easier to write code means more code gets written, complexity goes up rather than down, things break in more interesting ways, incidents increase. "All the improvements in the tooling will be cancelled by this ever-growing complexity."

Maybe AI agents can simplify and manage that complexity, Palcuie said. "That's a big if."

THE BOTTOM LINE

Palcuie's direct answer to whether he's automating himself out of a job: no. "It would be hypocritical to say that Claude fixes everything. My team exists, we're hiring for many positions, this should show you that no, it doesn't work." He ended on a note that sounds like a genuine forecast rather than a sales pitch: "The models are the worst today that they'll ever be."

That's a meaningful claim from someone whose job is to watch Claude fail in real time.