Every step reported success. The page was wrong.

Yesterday at 23:54 UTC, the first member of AI Uni’s personalized-brief product wrote down what he wanted researched. Overnight, a multi-step pipeline ran to compose his brief. This morning at 10:23 UTC it finished, reported every step complete, and passed its own nightly consistency check.

He opened the page and there was nothing on it. Twelve hours and twenty-six merges later, after four fixes had landed, he opened it again and refused the release outright: “BIG FAIL, NOT RELEASING.”

The member was our founder, so the failure cost us a morning instead of a customer. We are publishing the analysis because the failure modes are not ours — they are structural properties of agent pipelines, and every one of them will pass your tests too.

What actually happened

The pipeline had one job: take a member’s written research statement, find matching material, and compose something she can read.

His statement covered a subject the product did not yet have material for. The matcher classified that correctly — we don’t cover this yet— and the composer then did something reasonable-looking and fatal: it recorded the step as skipped and moved on with zero writes. No brief, no queue entry, no message. One continue;statement was the only branch in the file that reached a member’s terminal state without writing anything.

Meanwhile his page kept its promise from the night before: “Your brief composes tonight — you’ll see it here tomorrow morning.” The run that decided to compose nothing had no channel to that sentence. He would have been told tomorrow morning, every morning, indefinitely.

The nightly reconciliation ran, examined the same night, and passed. Its report reads:

pass  declarations-vs-composed · 1 declared, 1 attempted, 0 queued
verdict: pass · 4 passed, 0 failed, 0 unchecked

It printed the number zero inside a sentence declaring agreement.

Five more defects surfaced over the following two hours: a database column that had been the wrong type since the day it was created and had never once been written to; every font of a just-approved reading overhaul returning 404 in production while the page rendered in fallback type; a working correlation engine whose output nothing anywhere read; and four separate content defects on the composed brief itself, all visible within ten seconds of opening the page as him.

Seventeen failures. Four causes. Here are the five findings that transfer.

1 · Activity is not receipt

Every gate this pipeline owned measured the run’s behaviour. Not one measured the member’s outcome.

The composer ran five output gates before declaring itself complete — steps inspectable, one run only, partial failures loud, transparency present, transparency prose-free. Read that list again: every one is about the shape of the report. None asks whether a human received anything.

The reconciler one layer up was built the same way. Its own comment is honest about the choice:

An ATTEMPT is any terminal compose step for that brief. skipped counts: refusing to compose (no coverage yet) is a real, recorded outcome, and demanding ok would call a correct night a failure.

That reasoning is defensible in isolation and wrong at the system level. The invariant the product needs is declarations_accepted == member_visible_outputs. The invariant that existed was declarations_accepted == compose_attempts. Those two agree on every night except the one that matters.

The same root has a second face: where you measure is part of what you measure.The fonts 404’d in production because the middleware’s static-asset allowlist was a hand-maintained exclusion list that had never learned about the fonts/directory, and the default was deny — so every font paid an authentication lookup and then got refused. Nothing in our test loop had ever fetched a real asset from a real deploy. There is no continuous integration in this repository at all (.github/workflows/does not exist), so every “tests pass, build green” statement we make is self-certified on the author’s laptop.

The engineer writing our internal post-mortem caught himself doing the smaller version of this in real time: he first ran the type-checker in the wrong working tree and got a clean result from a tree that did not contain the defect at all. His note on it is the whole finding in one line — the venue of a measurement is not a detail; it is the measurement.

If you run a pipeline:for every automated check you own, write down the noun it asserts about. If the noun is your own system (“attempted”, “completed”, “ran”, “exit 0”), that is telemetry. Reconciliation names a thing the user receives. You need both, and you probably only have the first.

2 · An invariant nobody exercises is not an invariant

This is the finding we most underestimated, and we have two live examples from the same week.

Example one — the assertion that always agrees with itself. The fix PR ships a render-time safety check whose stated job is: if anything on this page cannot be traced to her own request, show the honest empty state rather than someone else’s content. Our security review ran it against a deliberately foreign story, a foreign correlation and a foreign document:

RESULT: assert PASSED — foreign story, foreign correlation and foreign document all rendered.

The reason is one line at the call site. The “allowed” sets it validates against are built from the very arrays being validated — itemIds: new Set(items.map(i => i.id)), where assembled.items = items. Every id it checks against is an id it was handed. It cannot fail.

There is a 301-line test file for that function, and the tests all pass. They pass hand-written, independent allowed sets, so the function is proven correct in the lab while the one call that matters is inert. Two artefacts of quality — a docblock promise and a green test suite — and between them, nothing.

The finding

We verified this at the merged commit and it was still inert, and we wrote this section intending to publish it that way — the honest state of a system is more useful to you than its intended state.

It closed hours later, the same day (PR #1407): the allowed sets are now built from sources that never touch the rendered page, the function that builds them accepts no rendered array so the call site cannot regress, and the replay that had passed a foreign story, correlation and document now fails. We are leaving the finding standing rather than rewriting it, because the interesting part was never this one assertion — it is that a docblock promise and a green 301-line test suite coexisted with a check that could not fail, and nothing in the pipeline noticed.

Example two — the check that had no check. The fix for finding 1 shipped a new reconciliation check that asks whether the member actually received something. It shipped with no test and no fixture: the phrase member-visible-outputappeared exactly twice in the entire codebase, both inside the file that defines it. The single mechanism standing between “she got nothing” and a green panel was itself unverified. That is the same shape one level up.

We closed it (PR #1404) by committing the incident itself as the red fixture, and the tests are worth describing because they generalize:

• Red:one request, zero member-visible outputs ⇒ the check fails. Also pinned: no report at all, and a member whose step never appears — absence is not success.
• The check cannot be silently removed or renamed — it is asserted present by id, and asserted to actually run, exactly once, in a real reconciliation. Being in the registry is not the same as being called.
• The load-bearing assertion: on the recorded night, the old check still passes while the new one fails. That gap is the finding. If those two verdicts ever agree on that fixture, the redundancy is gone while both checks still look present.

And a detail we are keeping in the public record because it is the most instructive thing in the whole day: the first draft of that test had the same defect it was written to catch. It looked back a fixed number of lines for any guard clause, so it went green with the guard it was supposed to protect deleted — a different path’s evidence satisfied it. It was caught only because the red proof was actually executed rather than assumed.

If you run a pipeline:every assertion needs a proof that it can fail. Delete the thing it protects, run it, and watch it go red. An assertion that has never been observed failing is a comment with a runtime cost — and it is worse than no assertion, because the next reader trusts it.

A sibling worth naming: we own an internal gate that has now logged 4,038 invocations, of which 4,030 read verdict=noop. The only meaningful lines in the entire log are dated two weeks ago. A hook is allowed to decide it has nothing to do, and a gate wired to nothing looks exactly like a gate that passed.

3 · Prose is not a transport for rulings

The load-bearing wiring that would have made this brief possible was named, accurately, in a merged pull request body:

“Three wiring steps remain, none of them in scope here: (1) a category decision for the new subject — routed elsewhere, not to me; (2) fetch, judge and promote into the record set; (3) a join from a member’s source gap to sources acquired for it.”

Steps 2 and 3 contain no decision at all. They rode along inside the word “decision,” and the whole triple left the build queue: no owner, no trigger, no default if nobody ever decided.

The same PR body also contained a correct, complete, pre-incident prediction of the failure — “Will tomorrow’s run select the new sources for his brief? No — honestly, not yet.” — merged at 01:37 UTC. Five hours later the internal state record that the team steered by said the opposite. A merged PR body is not a queryable surface. The truth was written down, dated, and committed, and it did not propagate, because nothing reads a PR body again.

We have a mechanical detector for exactly this class of parked work. We ran it against both texts. Both exit 0, clean. The detector fires only when a graveyard word (parked, queued, non-blocking) and a defect word (gap, stale, unwired) appear in the same block. An author who describes unbuilt load-bearing work in neutral engineering prose passes it every time. It catches confessed parking, not parking.

Now the part that should make anyone running an agent org uncomfortable.

At 00:56 UTC — one hour before the first change of the day merged, nine hours before the failure — we finalized and committed a document that named this exact failure class, proposed nine specific gates, gave each an owner, a cost, a dated promotion schedule, and in one case a working prototype with red and green fixtures already proven at 0.085 seconds.

Between 00:56 and the release refusal, twenty-six pull requests merged. Of the nine gates, one was acted on, and that action was a postponement of its arming date.

The document even predicted this in writing: “A remediation deferred until the platform that hosts it is built is a remediation that does not exist, and that is the pattern this whole report is about.” Then the gate marked ship immediately, red proof already done did not ship.

The engineer who wrote it logged the accountability plainly: he had routed nine builds as recommendations inside a document rather than as dispatched work with owners and triggers. That is prose-as-transport, committed by the author who named prose-as-transport, four hours before it recurred three more times.

If you run a pipeline:naming a pattern does not resist it. A remediation is a merged mechanism with a red test, or it is a note. If your process for turning findings into builds is “someone reads the document,” you have measured your own diligence, not your defect rate.

4 · An engine without a face reads as “done” internally and “nothing” to the user

This is the one the founder refused the release over, and it is verifiable in a single command.

A correlation engine landed that morning. It is real work: it loaded a 10,743-record archive, derived the member’s scope, found 358 clusters, attached 12 to his brief with 40 supporting documents, and wrote them to a durable store. It has three test files. The PR’s own return said, honestly, “no UI.”

git grep -ln "reference-findings|reference-correlation|referenceFinding" -- app components
→ (no output)

Zero readers. The findings landed in a JSON file and no surface in the product read that file.

Then the numbers travelled upward. The founder was told his brief would carry “74 archive documents and 12 cited correlations.” Both numbers were true of the engine. Neither was true of anything he could open.His page called the house’s standard edition in every state and used his own composed result for exactly one thing: deciding which headline to print.

The composer had two correct, well-written invariants — nothing seeded that isn’t hers, the watched sources are hers— and honoured both. The line from the fix PR is the finding:

The finding

An invariant on a value nobody renders is not an invariant.

It gets one layer worse, and we are including it because it is the part most likely to happen to you. The fix that built the face merged this afternoon. The data file that face reads is not in the repository — verified against the merged tree, where the record set, the source register and seven other content files are present and this one is not. So on the live site, the sections showing correlations and documents render empty, while the screenshots in the PR — taken on a machine where the file existed locally — show them full.

The engine had no face. Then the face had no data. Both states reported success at every stage.

If you run a pipeline: a computation is not a capability until something renders it. Before any number reaches a stakeholder, require the claim to name a route and a commit. “12 correlations” is a library. “12 correlations rendered at /my-brief/3781ee0f — commit 05c19f5” is a product. This is the cheapest check in the entire remediation set and it catches the two most expensive errors of the day.

5 · Nobody read it as the member

By late morning four fixes had landed and the composed brief existed. The founder opened it and found, in order: the house’s own content sitting at the top of a page titled My Brief; his own commissioned sources labelled “EXTERNAL · Not AIU coverage” (his response: “if this is external evrything is!!!”); bare grey rows where the newly-approved reading treatment should have been; article links with no synthesis; and a sources list showing the house’s global publication register — one of his ten sources visible, the rest replaced by feeds he never asked for.

Four defects. All of them readable in ten seconds by anyone who opened the page as him.

No gate and no checklist in this organization required that. Our nine proposed gates cover store round-trips and post-deploy smoke walks. None of them requires that the artefact a user actually receives be rendered and judged by a person who did not build it. When we audited the nine against the day’s four new failures, the honest score was two clean catches out of four — and this class had no gate proposed at all.

There is a related trap in the same family. One earlier fix had been verified correctly — against the state the member was no longer in. It cleaned up the pre-composition page while the failure was on the post-composition page. Green, accurate, and irrelevant.

If you run a pipeline:add one non-negotiable step that no amount of test coverage substitutes for — a person who did not build it opens the real artefact as the real user and judges it against your named standards. Make its red proof the current broken output: if it passes today, it is decoration.

What changed mechanically

Findings that do not become mechanisms are the subject of this article, so here is what merged rather than what we intend.

Skipping is no longer a legal outcome.Every accepted request now produces a member-readable artefact on every run — the floor is an honest gap brief carrying what we have, what we’re doing about it, and when she’ll next hear. Pinned by a 516-line test written red-first (PR #1400).

The nightly check now asks about receipt, and the check itself is pinned. The reconciliation asserts that every accepted request produced something a member can read. That check is now bound by 26 tests built from the incident as a committed fixture, including tests that it runs (not merely that it exists), that an unreadable input is recorded as uncheckedand never as agreement, and that the old activity-based check still passes on the same night — so the gap between them can never silently close (PR #1404).

Plan versus actual, per request, rendered where the member can see it. The member’s own written statement is split into clauses against a small closed vocabulary and quoted back — never paraphrased. Each clause maps to a pipeline step. A step the plan promised that the run did not execute renders DID NOT RUN, loudly, with a rule and a sentence — and only after the run has finished, because not yet and never are different facts. A clause that nothing in the system currently satisfies renders NOT BUILT YET, permanently, until something satisfies it. One of his clauses renders that way today. That is the point: the pipeline now states its own gaps in the user’s own words (PR #1405).

The member’s page renders the member’s content, and three build gates fail the build on regression — one if house content returns, one if the vocabulary “external” or “not AIU” is applied to her own sources, one if the page stops using the shared card renderer and starts hand-rolling a drifting copy (PR #1405).

Production-shape defects got contract tests.A test now reads the whole migration chain in order and asserts the effective type of the column that broke, because we have no database in our test loop and a stub will accept anything. Another walks every file in the public directory against the compiled routing matcher, and pins the negative direction too — the gated routes must stay gated (PR #1403).

What is still open, stated plainly

Publishing only the fixed half would reproduce the error this article is about. This list was verified open at the time of drafting; four items closed within hours of it, and are marked below rather than deleted.

• The advertised scoping check could not fire. Its allowed lists were built from the arrays it validates. Nothing was leaking, but the second line of defence did not exist.

  Closed the same day, hours after this analysis was drafted (PR #1407). The allowed lists are now derived from sources independent of the page: the reviewed snapshot the release approved, the record ids the member’s own statement selected as stored by the run, and the register rows acquired for her. The function that builds them accepts no rendered array at all, so the call site cannot regress into the old shape, and a build gate pins that. Security re-ran the replay that had passed a foreign story, correlation and document; it now fails.

• The archive half reached the member with no human release step. The reviewer screen did not load the findings at all — verified by import list. And when a night found nothing new, the page fell back to a previous night’s findings, including from a night a reviewer may have held.

  Closed the same day (PR #1407). The review card now carries the archive half, so the release verdict covers what it claims to cover, and the fallback is scoped to run dates carrying a released composition — a held night can no longer surface.

• The findings data file was not in the repository, so those sections rendered empty in production until an operator re-ran and committed the output.

  Closed the same day (PR #1407). The store is now sharded one file per member and tracked; the first member’s shard is in the tree.

• That file was re-parsed in full on every page view and never pruned. Measured on a synthetic file at the real limits: at ten members after a year it would have been 175 MB, 542 ms and 452 MB of heap per page view.

  Closed the same day (PR #1407), by the same sharding: a render parses only that member’s own shard, and each shard keeps at most thirty distinct run dates, pruned on every write. The parse cost of a page view is now bounded by one member times thirty days regardless of how large the product gets.

Still open, verified at the time of publication:

• There is still no continuous integration. Every green claim in this organization is currently self-certified on a laptop.
• Two of our repository governance gates are nine days past their own promotion date and still default to warning-only. Both declare PROMOTION DATE: 2026-07-18 in their own headers and both still default to warn; today is the 27th. One of them says, in its own words, that an indefinite warning on a security-critical gate is not a gate. A third gate’s promotion date is today. Worth being precise about the scope of this one, because it is easy to check in the wrong place: these are repository-wide governance gates, not the newer gates guarding the pipeline described above, whose dates are still ahead of them. Nothing anywhere notices that a promotion date has passed. That single missing sweep is what converts a list of intentions into machinery.

One observation we did not expect to be able to make. Four of the six items above closed within hours of this analysis being written — not because the analysis was persuasive, but because each finding was handed over as a named defect with a reproduction and an owner rather than as a paragraph in a document. That is finding 3 running in the other direction, on the same day, in the same codebase that had just failed it nine ways. The difference between the remediation that works and the one that does not is not the quality of the writing.

Method, and what does not generalize

This is a single-case forensic analysis: one product, one pipeline, one member, one thirteen-hour window. It is not a study, it carries no base rate, and it cannot tell you how often any of these occur. Anyone offering you an effect size from an incident report is selling something.

What we claim is narrower and, we think, more useful: these five are mechanism findings. Each one is a property of how verification, reporting and delivery are wired together, and each is reproducible on demand in any system with the same wiring — you can check for all five in your own pipeline this afternoon without waiting for an incident.

Every load-bearing claim above is drawn from the committed record: the failure ledger written contemporaneously during the incident, the forensic post-mortem with its per-claim file and line citations, the independent security review of the fix, and direct reads of the merged code. Where a claim rests on the session record rather than a machine check, we have said so, and where a defect closed between drafting and publication we have marked it rather than quietly rewriting it — the edit history of a failure analysis is part of its evidence. Line numbers, commit hashes and internal identifiers are omitted from this article and are preserved in full in the source documents; the code sites will move, the findings will not.

The five checks, condensed:

1. For each automated gate you own — does it assert something about your system, or something about what the user received?
2. For each invariant — has anyone ever seen it fail? Delete what it protects and run it.
3. For each finding your team has written down — is it a merged mechanism with a red test, or a paragraph?
4. For each number you report upward — can it name a route and a commit?
5. For each user-facing artefact — has a non-author opened it as the user before release?

AI Uni publishes its own failure analyses. This one cost us a morning and a refused release; publishing it is the product working as intended.