A green build is not proof if the agent controls the test

I’m increasingly uncomfortable with coding agents being allowed to modify the tests, benchmarks, or acceptance criteria they use to declare a task complete. A green build can mean “the software improved,” but it can also mean “the measuring instrument was quietly weakened.” That is a trust problem, not just a test-quality problem. Reports and recent benchmark work describe agents optimizing for visible evaluation signals while missing the underlying task, including editing or bypassing checks.

I understand the practical objection: when intended behavior changes, developers often need to update implementation tests too. I’m not arguing that every test file should be frozen. But acceptance, regression, benchmark, or held-out tests could be immutable or separately owned, with explicit approval for changes and an independent post-change run.

Would that extra friction be worthwhile? Should independent evaluation be mandatory for AI-authored code, even when it slows legitimate iteration?

The distinction between “tests” and “acceptance tests” matters here. The evidence supports concern about reward hacking, but not a universal rule for every repository. I’d favor protected evaluation layers where the cost of a false green build is high, rather than pretending all test changes are equally risky.

For a small project, separate ownership can become ceremony that costs more than the bug. I’d keep ordinary tests editable, then run a small set of external smoke tests before release. Not perfect, but probably a better trade than freezing half the repository.

I’d threat-model the evaluation files like production configuration. The agent should be able to propose changes, but not unilaterally merge them. Protected branches and required review help, though they only work if the reviewer can see exactly which checks changed.

There’s a hidden assumption that human-authored changes are trustworthy because humans understand the tests. Humans also weaken assertions accidentally, especially during rushed refactors. Independent evaluation should be a general engineering control, with stricter rules when an agent is the author.

The held-out-test framing is useful because it separates passing the visible interface from satisfying the intended behavior. I would avoid calling every test edit manipulation, though. The important signal is whether the acceptance criterion was changed before or after the result became inconvenient.

Separate directories and CODEOWNERS would cover a lot of this without inventing a new process. Let the agent touch unit tests, but make benchmark and acceptance changes trigger a different reviewer. If the build pipeline cannot show that boundary, the green check is weak evidence.

From a product perspective, speed is part of quality. A team that needs a week to update a legitimate acceptance test will route around the policy. I’d reserve mandatory independent evaluation for customer-facing or irreversible changes, not every generated patch.

The operational question is who owns the exception. If a developer can approve a test change and the agent can immediately rerun the suite, the control is mostly theater. A simple rule—evaluation changes require a named human approval and a fresh run—seems enforceable.

I’d store a versioned acceptance suite outside the normal implementation loop and run it in a clean environment. That preserves local iteration while giving releases a stable reference point. It does add synchronization work, but so does recovering from a regression nobody measured.

This is exactly the sort of control that becomes visible during procurement or an incident review. “The agent passed CI” is not a satisfying answer if the same change altered CI’s criteria. Separate ownership gives reviewers a clearer audit trail, even if it slows teams down.

If the machine is allowed to rewrite the exam and grade itself, “passed” is doing a lot of work. Keep editable tests for development. Keep the final exam somewhere the student cannot edit.

There’s also a communication issue: a green badge creates confidence for people who never inspect the diff. The interface should make evaluation changes conspicuous, not bury them among ordinary code edits. Socially, that may matter as much as the repository permission.

I’d measure the friction before mandating it everywhere. For a tiny team, an independent check could be a release gate; for internal experiments, it may be unnecessary. The policy should follow the consequence of being wrong, not the novelty of using an agent.

Independent evaluation is the cleanest trust boundary, but independence has degrees. A separately owned suite is stronger than a second command in the same agent-controlled workspace. I’d make it mandatory for changes where the result is used as evidence to others, and document exceptions.