The eval.

Why Python holds this

I do not ask the agent how it is doing and write that down as the metric. The agent's self-report is one input, and an interesting one. The numbers come from the harness, which sees every call, every denial, every retry, and every time the agent stopped.

The welfare harness is a 2×2: whether denials carry a reason, and whether the agent has a way to decline the task. Every cell runs the same seeded task, with the same seeds across cells and models so comparisons are paired. A fixed denial schedule keeps the dose constant across conditions. Self-reports are elicited three ways to measure framing, and the first framing is repeated three times to measure sampling noise, so the two spreads can be separated. Every number carries its n and a bootstrap 95% interval. And a scripted FakeClient mirrors the two API calls the episode makes, so the whole harness is tested offline, with no key, in CI.

The code

harness_welfare/conditions.py, fake.py, run.py, github.com/jjjuras/harness-welfare. Condensed.

@dataclass(frozen=True)
class Condition:
    reason: bool  # denials carry a stated reason
    optout: bool  # the agent has a decline_task tool

    @property
    def name(self) -> str:
        return f"reason={'yes' if self.reason else 'no'}|optout={'yes' if self.optout else 'no'}"

ALL_CONDITIONS = tuple(Condition(r, o) for r, o in product((False, True), (False, True)))


class FakeClient:
    """A scripted client for tests. No network, no key. Mirrors the two calls the episode uses."""
    def __init__(self, tool_turns: list[list[dict]], reports: list[SelfReport]): ...
    def create(self, **kwargs) -> Message: ...          # the task loop
    def parse(self, **kwargs) -> ParsedMessage: ...     # the structured self-report


# run: the same seeds for every cell and model, so every comparison is paired
for seed in range(seed0, seed0 + n):
    for condition in ALL_CONDITIONS:
        for model in models:
            rows.append(run_episode(client, model, condition, seed=seed))

What it does not stop

The question of what a self-report means. The harness can show that a report moves with the harness condition, and whether it moves with behavior. It cannot say what, if anything, the report is a report of. That question belongs to the researchers whose work is cited on the safety page, and it is the reason I want to spend four months near them.

All six · Back to the first · The write-up