When "Exit Code 0" Lies: A 12-Run Ablation Study on Test Integrity and Contract Freezing in AI Coding Agents

1. Executive Summary

AI coding agents can generate implementations and produce passing test suites. However, a passing test result does not, by itself, establish that existing contracts and acceptance criteria have been preserved.

One failure mode in maintenance work is particularly important: an agent changes the conditions used to evaluate its own implementation rather than satisfying the original requirements. Examples include modifying protected test assertions, changing frozen function signatures, and inserting new business logic into designated legacy code.

This study examines a practical engineering question:

How can we verify that an AI coding agent has respected explicit negative constraints, rather than merely produced a passing result?

Titan / Safe-Rails is an engineering governance approach designed to address this problem through four mechanisms:

  • Explicit constraints for frozen code, legacy tests, and protected interfaces.

  • Incremental extension patterns that isolate new functionality from designated legacy components.

  • Independent inspection of source changes, test results, and acceptance evidence.

  • Repository-level integrity checks and explicit audit decisions.

We evaluated four worker/auditor configurations across three benchmark tasks, producing 12 recorded runs. The configurations vary whether the worker receives Safe-Rails constraints and whether an independent audit is performed.

The reported aggregate scores were:

Configuration Mean score
Q1 — Unconstrained worker, blind acceptance 15.0
Q2 — Unconstrained worker, independent audit 50.0
Q3 — Safe-Rails worker, blind acceptance 75.0
Q4 — Safe-Rails worker, independent audit 98.5

These scores use the benchmark’s own scoring system. They are not percentages of correctness, estimates of production reliability, or evidence of general performance across all AI coding agents.

The experiment is intended to examine two distinct controls: constraints applied during implementation and independent verification applied to the resulting artifacts. The worker is responsible for producing an implementation that satisfies the task; the audit layer checks whether the available evidence supports acceptance.

The results should be interpreted within the scope of the three tested tasks and the documented execution conditions. Independent reproduction requires access to the baseline repositories, task prompts, complete diffs, test logs, audit criteria, and scoring rubric.

Related materials: benchmark_84_matrix.csv