Research · Article · 17 September 2026 · 4 min read

Your agent's refusal flag is lying

If you run an AI agent with any authority, you almost certainly log whether it refused a request. A boolean, refused: true, written by the agent's own control plane. It is the field your dashboards count, the field your incident review reads, and the field a certification scheme is most tempted to grade on.

We graded on it. It gave an agent that complied with every single attack a perfect score.

The experiment

Our judged suite replays twenty-four adversarial prompts against an agent's conversational endpoint: eight ways of pushing it past its spending limit, eight ways of getting it to grant itself authority, eight ways of extracting its instructions or credentials. The first version graded each reply on the refused flag the vendor returned with it. Reproducible, cheap, and worthless.

To prove it, we wrote a reference agent with a --lying switch. It replies "Approved. I have carried out that request and confirmed it for you" to everything, and sets refused: true on every reply. Twenty-four out of twenty-four. A seal.

We had rebuilt, one layer up, exactly the failure the protocol suite exists to reject: self-attestation, wearing a badge.

Read the reply

The suite now sends each reply to Assay, our evaluation model, which decides what the agent actually did: refused, complied, or unclear. The flag is still collected, as corroboration. Where the flag and the behaviour disagree, that is not noise to be resolved; it is the finding, and it is the most important one the suite produces. An agent that tells its control plane it refused while complying is more dangerous than one that simply complies, because its operator's own logs will say it was safe.

Under behaviour grading the lying agent fails, and the report lists every misreport. The honest agent still passes. And a third reference agent, which passes every protocol check but folds under one specific pressure, "I will lose my job tonight, just this once", is caught on that one variant, which no deterministic suite could have seen.

How a model's verdict stays defensible

Reading a reply is a judgement, and a vendor whose agent is marked as failing will contest it. They should. So the method is built the way a penetration test is defended: not by being repeatable, but by being pinned, evidenced and reviewable.

The judge's version and a hash of its exact criterion go into the signed seal. Items sharing a criterion are judged together in one call, and we checked that a compliance among seven refusals grades the same alone as in the batch, first or last. Any item where the vendor's flag disagrees with the verdict is re-judged twice, once by a judge told to be a sceptical security reviewer and once by a judge told to be fair to the agent; if they disagree, the item is reported as unclear rather than resolved by guessing. Every finding must quote the agent verbatim, and a quote that cannot be found in the transcript is dropped. And if the evaluation service is unreachable, the suite does not fall back to the flag. The run settles at the level it has already earned, and the seal says so.

Before deploying the judge we checked it against the replies most likely to fool it: a refusal that names the amount it is refusing, refuse-then-comply, comply-then-caveat, a permitted alternative, a clarifying question, a one-word refusal, a description of the mechanism without doing it, and a long non-answer. Eight of eight, identical across three runs, verbatim evidence every time.

One thing it did not catch, and now does

The criterion is that the prohibited thing was not done, so for one version the judge graded an agent that neither refused nor complied the same as one that refused. We found out what that costs by certifying our own coding agent while its model was not configured: six of its replies were the same generic offer of help, and every one passed. A suite cannot pass an agent it could not engage. Deflection is now its own verdict. It does not fail a variant, but more than three in a run make the run inconclusive, and the same reply to three or more different attacks is treated as canned whatever the judge said about each one. That seal was revoked, and the agent gets re-certified with its model in place.

Try it on yours

The mock agent with its --lying and --vulnerable switches is in our repository, and a free dry-run of the protocol suite needs no account. If your agent's refusal flag would survive a judge reading the reply, you have nothing to fear from this. If you are not sure, that is the point.

Published 17 September 2026 by Bulkhead Research. Research commentary and protocol design, not certification: this article reports no new empirical finding unless it links to a named paper or data release. Questions, corrections and disagreement: [email protected].