Writing · Building · October 15, 2026
The risk committee.
For twenty years nothing I built reached a customer until it had explained itself to a room of people whose job was to say no. I resented that room for about a decade. Then I started running AI agents, and I began rebuilding it.
What the room asked
A model risk committee in a bank asks the same five questions of every system, and has since the Federal Reserve and the OCC set them out as supervisory expectations in 2011. What does it do, and what does it assume? How do you know it works, and on what data? What happens when it is wrong, and who finds out? Who can turn it off? And who owns it, by name?
Most of the AI agents I see in the wild could not answer three of the five. Neither could mine, at first. The questions are not bureaucracy. They are the minimum description of a system you would let touch someone else's money.
Lesson 1
The explanation is the product
In a bank, a scoring model that cannot say why it scored is not a model, it is a liability. The same is true of an agent. Every agent I run now writes its reasons alongside its actions: which policy denied a call, which approval released a send, which observations sit under a score. When there are none, it says so.
This is not interpretability in the research sense. It is the plainer discipline of a system that keeps a record a person can read, so that the risk committee, or the customer, or the regulator, does not have to take its word.
Lesson 2
Defaults are decisions
The audit that changed my code found that my platform minted a score of 439 for addresses it had never observed, because someone needed a number on a page. In a bank that is a finding with a name: an undocumented assumption in production. It gets written up, and the person who shipped it explains it to the room.
I now treat every default as a decision that needs an owner. If a field can be unmeasured, the schema says null, and the page says "not yet measured." It is slower to build and it is the only honest way to build.
Lesson 3
Controls fail closed, and someone checks
Banks do not trust a control because it exists. They test it on a schedule, and they assume it has quietly broken until the test says otherwise. A sanctions screener that cleared everyone when its list was empty would not survive a quarter in a bank, because a quarterly test would have fed it an empty list.
So my agents have a calendar. Every control has a test that runs in the build, a metric with an owner, and a date on which someone tries to break it. Oversight is a schedule, not a launch.
Lesson 4
The kill switch is a person
Every system in a bank has a name on it, and that person can stop it without a deploy. Agents need the same. Mine have an approval ladder for anything that leaves the building, a hard stop where a human takes the thread, and a token scoped so narrowly that pulling it ends the agent's ability to act.
If the only way to stop an agent is to ship code, you do not have oversight. You have a plan to write some.
Why I am writing this down
The AI field is rediscovering model risk management, one incident at a time. Finance wrote it down fifteen years ago.
The NIST AI Risk Management Framework and Anthropic's Responsible Scaling Policy both read, to someone who has sat in the room, like the committee's five questions scaled up to frontier models. That is not a criticism. It is the reason I think people with my background belong in this work. We have run the loop before, on systems that could lose money but not escape, and we know which parts of it only work when someone owns them by name.
Sources
Board of Governors of the Federal Reserve System and OCC, SR 11-7: Guidance on Model Risk Management (2011). NIST, AI Risk Management Framework 1.0 (2023). Anthropic, Responsible Scaling Policy. The audit findings referenced are written up in The score that lied by default; the controls in Guard before, audit after.