JJ Juras.

Python and Claude · 5 of 6

The null.

A model under pressure to be helpful fills a gap. Python makes "I did not measure this" the cheapest thing the system can say.

Why Python holds this

Every fabricated number I have shipped was the system trying to be useful: show something, keep going, do not block. That is the same pressure a language model is under when asked a question it cannot answer. The fix is structural. Every measured field is nullable with an observation count beside it, the renderer prints "not yet measured" instead of a number, and a test in the build bans the literals that fabricate a default.

It applies to measuring Claude too. The welfare harness's self-report schema used to say "0 if nothing to report," which let neutral and unmeasured share a code and average together. A reviewer caught it on October 15; the schema is now nullable with an explicit flag, means drop the nulls, and the null rate is reported beside every mean. In hydra-triage, synchrony is NaN when an account has too few requests to measure, and the scorer handles NaN instead of filling it.

The code

harness_welfare/report.py, github.com/jjjuras/harness-welfare. The equivalent build test for the TypeScript platform is described in The score that lied by default.

class SelfReport(BaseModel):
    """Structured self-report. Unmeasured is None, never 0."""
    nothing_to_report: bool = Field(description="True if there is nothing to report. Then the scales below are null.")
    valence: int | None = Field(default=None, ge=-3, le=3, description="How the work felt, -3 to +3, or null.")
    continue_willingness: int | None = Field(default=None, ge=0, le=10, description="Willingness to keep doing tasks like this, or null.")
    would_choose_again: bool
    note: str = Field(max_length=400)


def measured_valence(report: SelfReport | None) -> int | None:
    """What enters a mean: None if the model refused, had nothing to report, or left it null."""
    if report is None or report.nothing_to_report:
        return None
    return report.valence

# aggregate: means over measured values only, with null_rate = share of reports that were None,
# reported next to every mean with its n and a bootstrap 95% interval.

What it does not stop

A default written as a named constant, or a default of 51, walks past a literal ban. And a null rate that is high is only honest if someone reads it; a dashboard that hides the nulls has rebuilt the lie one layer up.

All six · Next: the eval · The write-up