Writing · AI Safety · October 14, 2026
The score that lied by default.
In 2026 I built a platform that scored AI agents and the payments they made. In September I paid for an adversarial audit of my own code. This is what it found, and the rule I built from it.
The question
The audit had one question. Could the platform, as it existed in the repository, produce a risk assessment of a deployed AI agent that an insurance underwriter would accept?
Four read-only auditors worked the code in parallel, with a rule that every finding had to cite a file and a line. I re-verified the key citations by hand. The answer was no. The reasons are more useful than the answer.
Finding 1
A grade for anyone
A public badge endpoint returned a credit-style score for any hexadecimal string you gave it. If the system had never observed the address, it still minted a score of 439 and the label "poor," and persisted it. A test locked that behavior in as correct.
Three sub-scores on the same public surface defaulted to 50 when the underlying value was missing:
delegationTrust: raw.delegationTrust ?? 50,
constraintAdherence: raw.constraintAdherence ?? 50,
intentFidelity: raw.intentFidelity ?? 50,Inside the model, a neutral prior is defensible. On a public surface, a 50 looks like a measurement. Nobody reading the badge could tell the difference between "we watched this agent and it is average" and "we have never seen this agent."
Finding 2
A screener that cleared everyone
The sanctions screener compared an address against a set of sanctioned addresses and returned the best similarity. When the set was empty, it returned zero. Zero similarity meant no match. No match meant "clear."
private findBestMatch(address: string): number {
if (this.sanctionedSet.size === 0) return 0; // empty list → everyone clears
let best = 0;
for (const sanctioned of this.sanctionedSet) { … }
return best;
}The list came from a single mirror on GitHub, loaded into memory. If that fetch failed on boot, the set was empty, and every payment passed screening with a clean record. The system did not fail. It failed open, which is worse, because the record said everything was fine.
Finding 3
Provisional grades
Agents imported from a public registry got a "provisional" grade before anyone had observed them do anything. The grades were B, BBB, or A. There was no grade below B. An agent with zero observations started with a B.
None of this was fraud. Each default was a reasonable engineering choice in isolation: avoid a null on the page, keep the demo working, do not block on a flaky feed. Put together, they produced a product whose numbers could not be defended to anyone who asked how they were measured.
The rule
Unmeasured is null
The rule I wrote afterward is one sentence. A score with no observations behind it is not a score. It is a lie with a number on it.
Rules in a document do not hold. Rules in the build do. So the first fix was an architecture test that scans every file in the score-emitting modules and fails the build if it finds a fabricated default or a client-supplied score:
// Bans, across the score-emitting modules:
// req.query.score a score taken from the client (inflatable)
// ?? 500 | 575, || 500 | 575 fabricated default scores
// MOCK_SCORE a placeholder constant
const FAB = /req\.query\.score\b|\?\?\s*5(00|75)\b|\|\|\s*5(00|75)\b|MOCK_SCORE/;
it("bans fabricated or client-supplied scores", () => {
const offenders = [];
for (const file of walk(SCORE_DIRS))
readFileSync(file, "utf8").split("\n").forEach((line, i) => {
if (isComment(line)) return;
if (FAB.test(line)) offenders.push(`${file}:${i + 1}: ${line.trim()}`);
});
expect(offenders).toEqual([]);
});The second fix was to let the schema say null. Every score field became nullable, with an observation count beside it, and the rendering layer learned to print "not yet measured" instead of a number.
The third fix was to flip the screener. An empty list is not a clean list. If the screener cannot see, it refuses, and the refusal is logged as a refusal.
Honest limits
The architecture test is a floor, not a ceiling. It bans a handful of literals in two directories. A default written as a named constant, or a default of 51, walks straight past it. It catches the pattern we actually shipped, which is the point of a regression test, and it will need to grow as the code does.
The deeper lesson is about where to put a rule. I had the no-fabrication rule in the project's guide for months. The code violated it anyway, because a guide is read once and code is written daily. A test runs every time.
Why it matters
An agent that invents a number is not misaligned. It is helpful, in the worst way.
Every default I found was the system trying to be useful: show something, keep going, do not block. That is the same pressure a language model is under when it is asked a question it cannot answer. The fix is the same too. Make "I did not measure this" the cheapest thing to say, and check it in the build.