Nestor G Pestelos Jr · Writing · Print

What I Check Before Accepting Agent-Written Code

Published September 8, 2026

TL;DR

A test can reject the fix you need when it preserves a known limitation. Check what the test accepts, whether policy uses trusted inputs, and which commit a review covers.

A refund operation in a client application returned success and changed the record's state without publishing an event. The test required that absence. Publishing the missing event would fail its assertion.

This was one of three different acceptance problems in my agent-assisted work. The refund needed a different expected result. My guest-agent demo needed trusted data before a memory write. My development tooling needed to detect commits added after review.

A zero amount still changed state

There was no money left to return. In the refund method, an early return sat inside a transaction block. It exited the method, so execution never reached the event publisher after the block.

The fix replaced return with next. Execution could leave the block and continue to the publisher.

Snippets are simplified illustrations with generic names, not client source code. Supporting paths and helpers are omitted.

def finish_refund
  result = transaction do
    mark_refunded!
    next { success: true, refunds: [] } # Previously: return
  end
  publish_event if result[:success]
  result
end

The revised test still expected a successful result and an empty list of refunds. It now also required the event, recorded for that order and carrying the refund reason.

The change needed both the control-flow fix and a new expectation. More tests around the old expectation would not settle whether the limitation stays.

The failure case also mattered. A separate test continued to require no event when the transaction failed. Publishing after a successful zero-amount operation and withholding publication after failure were both requirements.

The policy needed trusted data

My guest-stay agent demo gained a path that could persist memory. The demo uses synthetic fixtures; its earlier proposal endpoint had no persistent side effect.

The new endpoint loads stay data and test fixtures from the server's own assets. Its policy code evaluates the proposed choice against that data. It then builds a memory record from the verdict, validates the record, and attempts persistence.

const trustedData = await loadServerData();
const verdict = evaluatePolicy(trustedData, proposal);
const record = buildMemory(verdict);
const validated = validateMemory(record);
if (validated) await persistMemory(validated);

If the server accepted client-supplied fixtures, a request could change the permission fields used to evaluate its own proposal. The server therefore loads its own copy of that data.

The policy code and the record validator also have to agree about what they exchange. A hand-built test record can pass validation even while the policy produces a different shape.

For an escalation record, the policy adds a field identifying its stage. A validator built only around the original fixture could reject that legitimate output. The tests pass actual policy-produced memory into the validator, including the escalation path. They also check that unexpected fields are rejected.

That is a limited claim about the record's shape. A string in an allowed field can still contain inappropriate content.

New commits need another review

Suppose a review covers commit A. Then another commit, B, is added to the pull request. The earlier review may still appear complete, but it does not cover B.

My development tooling records the commit that was reviewed. Before merging, its verifier checks whether the code still matches that commit. If the code has changed, it marks the change not ready to merge and requires another review.

if live_revision != reviewed_revision:
    state["merge_ready"] = False
    state["next"] = "review"
    save(state)
    raise ReviewRequired()

The recorded review stays attached to commit A. Replacing its commit ID with B would make B appear reviewed without anyone examining it.

The regression test reproduces this sequence: record a review, add a commit, then run the check. It verifies that the tool refuses to mark the change ready and preserves the original review record. A separate test checks that matching commits pass.

This catches changes present when the check runs. Another commit could still arrive before the merge.

Before accepting a change

For one proposed change, inspect what its passing checks accept. Does a test preserve a limitation you mean to remove? Can the request supply data used to authorize itself? Does the review cover the current commit? Record the answer and its evidence before calling the change ready.

Files (demo)

The public demo's source at the referenced commit:

Back to top