Every Suite Was Green: What Running My Own Governance Tool Actually Broke

Five weeks ago I published an argument that an unenforceable architecture decision record is a wish, and a tool built on it.

Since then I've done the thing that actually tests a claim like that: I made two real projects depend on it. Not a demo repository. A production Next.js application and a plugin marketplace, both of which now fail their pull requests if the tool misbehaves.

It misbehaved. This is what that found.

• The bug that logged success
• The fix is a reframe, not a patch
• Exactly one, not at most one
  • Why "exactly one" and not "at most one"
  • Three cross-checks, because one assertion is a story
  • What it deliberately does not cover
• Running the verification found four more bugs
• The claim I had to withdraw
• What real dogfooding looks like
• The honest ledger
• What I'd take from this

The bug that logged success

The @adrkit/ci Action comments on a pull request with the decisions governing the files that pull request changes. It's supposed to post once and then update that same comment on every subsequent push.

It didn't. It posted a new comment every push. It did that for two releases, in every repository pinned to the moving v0 tag, all of which are mine.

Every test was green the whole time.

Green is doing less work in that sentence than it looks like it is. The unit test covering this path asserted that an unresolved identity adopts nothing, which is correct behavior and is exactly what shipped. Nothing asserted that the identity was ever resolvable in the path that actually runs. And the end-to-end job invoked the CLI with contents: read, so it never constructed a GitHub client, never resolved an identity, and never posted a comment. The surface the bug lived on had no coverage at all.

Which leaves the job log as the only witness, and this is where it gets interesting. The Action's job log for a healthy run reads:

adrkit: created the governing-decisions comment.

That is also exactly what it prints when it has just created the fourth duplicate comment on a pull request that already had three. The log line is the same. The exit code is the same. A reviewer reading the output of a broken run sees a tool reporting that it did its job, because it did do its job. It just did it again.

The underlying cause was an identity check. The upsert logic was correct, but it depended on knowing whether the Action itself had authored the existing comment, and that identity was unobtainable.

The default GITHUB_TOKEN is a GitHub App installation token, and an installation token cannot call users.getAuthenticated. There was a fallback for exactly that case, gated on recognizing the token as the default one by comparing it against process.env.GITHUB_TOKEN. But action.yml defaults the token input to ${{ github.token }}, and Actions does not export GITHUB_TOKEN into a step unless the workflow explicitly asks for it.

So the comparison had no right-hand side. The identity resolved to "unknown." An Action that cannot prove it wrote the previous comment correctly refuses to adopt it, and correctly creates a new one. Every single time.

The fix is a reframe, not a patch

The insight that closed it: a permission-shaped refusal is not missing information.

An app installation token is always refused by users.getAuthenticated. That refusal is not an error to route around. It positively establishes that the caller is a bot, which is most of what the identity check needed in the first place. Identity is now classified by what the token actually proves rather than by what it failed to return.

That's paired with a stricter ownership rule. The marker comment has to be exactly the body's first line, not merely present somewhere in it. That's the same test the ARB queue Action already applied to its managed issue, so the two surfaces now agree.

An identity that genuinely cannot be resolved still adopts nothing, but it now says so as a job-log warning instead of passing silently. The fix also repaired repositories using a custom GitHub App token, which had never had a working upsert at all.

Exactly one, not at most one

Fixing a bug that hid inside a success message raises an obvious question: how do you keep it fixed?

The answer is a job called action-dogfood, which runs the real Action twice against the repository's own pull requests and then asserts, over the API, that exactly one comment leads with the <!-- adrkit:ci --> marker and that a Bot authored it.

Because an empty comment list satisfies "at most one."

That's the blind-pass shape. A gate that passes when there is nothing to look at isn't a gate. If the Action had silently stopped posting entirely, "at most one" would have gone green forever, and the check built to catch a duplicate would have missed a disappearance.

The comment list is compared against GitHub's own reported comment count, so a dropped --paginate cannot hide a duplicate on page two. The surviving comment's id is compared across both dispatches, which is a stronger claim than "there is one comment now." And ids that existed before the run are recorded up front, so a pre-existing comment can't be mistaken for one this run created.

Ownership requires a bot author and a leading marker. Both, not either.

After a pull request's first push the comment already exists, so a dispatch that writes nothing satisfies both assertions. The inverse of the original bug is therefore caught on every newly-opened pull request's first run, and not on later pushes within one pull request.

That gap is written down rather than implied. Adding an Action output to close it was considered and rejected.

Running the verification found four more bugs

There's a second layer: a reference repository that calls the Action the way a consumer does, pinned to a commit rather than a branch, covering two scenarios the in-repository job structurally cannot. A fail-closed dispatch against an invalid corpus directory that must write nothing, and a degrade under a read-only token that must stay green and post nothing.

Actually running it found four defects in the verification artifact itself, all invisible to static review:

• path: .adrkit deleted the contents of a directory the reference repository tracks.
• A second workflow posting the same marker silently weakened a run instead of failing it, because a foreign comment satisfies the "absent" rule. The first attempt went green having observed two updates and no create.
• The degrade scenario never echoed its outcome, so it proved nothing.
• --paginate combined with join(",") inside --jq emits one line per page, which would hand a multi-line value to a flag expecting one, past thirty comments.

A fifth defect was in the instructions rather than the code. The README said to delete the marker comment before pushing, which removes the accidental protection a pre-existing comment provides. Both the verification workflow and the reference repository's own workflow listed an empty set and both created, within the same second. The order is now push, settle, delete, re-run, which excludes the race structurally.

That last one is my favorite, because it's the first time the duplicate rule has been observed firing outside a fixture. A rule you've only ever seen pass is a rule you haven't tested.

The claim I had to withdraw

While measuring all this I noticed that GitHub didn't bump a comment's updated_at across either in-place update, and I generalized it: GitHub does not bump updated_at for a byte-identical PATCH.

That generalization is wrong.

It doesn't hold on openleague, where three same-commit re-runs with a SHA-256-identical body advanced updated_at every time. The obvious second explanation, that the actor differs, is also wrong: the unchanged rows and the advanced rows are the same actor. Endpoint and elapsed time are both ruled out by measurement.

So the mechanism is not established, and it's recorded as not established rather than given a third plausible story. The gate asserts id stability instead, which is the property it actually needs.

I'm including this because "I measured something, generalized it, found a counterexample in my own second repository, and could not explain either" is a normal week in software that almost never appears in a project's public writing. The alternative to writing it down is a confident sentence in a README that someone eventually depends on.

What real dogfooding looks like

The two repositories doing the work are worth describing concretely, because "we use our own tool" usually means a badge.

openleague is a Next.js and Prisma platform for managing sports teams and leagues. It pins @adrkit/cli and @adrkit/mcp at exact versions with no caret. It runs the governing-decisions Action on every pull request, plus two more adrkit workflows including a monthly sweep that opens and closes a single tracking issue for records whose review date has come up. It wires the MCP server into three separate agent surfaces. And it has a couple hundred lines of integrity checking whose entire job is to make sure four separate adrkit version pins in that repository never drift apart.

That last one is the signal I'd point at. Someone writing supply-chain guards around your tool is treating it as infrastructure.

context-kit went a different direction and built a bridge plugin around the corpus, with commands to promote a decision into a record and to index existing ones. Its AGENTS.md makes corpus consultation a standing rule: before proposing a change to a governed path, check what already binds it.

Both of those are mine. That matters, and I'd rather say it than let the paragraph imply otherwise.

The honest ledger

adrkit has eight stars, three forks, and two contributors from outside the project, six commits between them out of two hundred and thirty-four. There is, as far as I can find, no third-party writeup of it anywhere. The npm download numbers look respectable and are substantially my own continuous integration, which you can see in the shape of the curve: real spikes on days I ship, and days with none at all.

The project grades every capability on a three-rung evidence ladder. Unit and contract coverage, then a maintainer-owned reference repository, then external validation. The CI comment path reached rung two this month. Rung three is open on everything, and rung three is the only rung that means somebody other than me found this useful.

That's the accurate status, and the README says the same thing. A governance tool that overstates its own evidence is refuting itself in public.

What I'd take from this

Test suites cannot see identical outputs. If your success path and your failure path print the same line, coverage is not the control you think it is. Find the assertion that distinguishes them, or accept that you're flying on vibes.

"At most one" is not a gate. Any assertion satisfied by an empty result set will pass forever once the thing it watches disappears. Write the lower bound too.

Run the harness before you trust the harness. Four of the defects above lived in the verification artifact, not the software. Static review found none of them.

Depend on your own thing, in something you'd be embarrassed to break. Every bug in this post came from that, and none of them came from reading the code again.

• Site: adrkit.dev: quickstart, commands, MCP setup
• Source: github.com/mbeacom/adrkit, Apache-2.0, schema additionally CC0

Sources. The #107 root cause, the action-dogfood assertion design, the four reference-artifact defects, and the withdrawn updated_at claim are all recorded in the project's CHANGELOG under v0.7.0 and v0.8.0, and on ADR-0026. Repository and contributor counts come from the GitHub API, and download figures from the npm registry, both checked on 26 August 2026. The openleague and context-kit integration details come from those repositories' committed workflows, package.json, and MCP configuration.