The Agent Optimizes for What You Check

I used to believe I understood why agent-written code degrades as a project grows. Models were tuned to score well on benchmarks, picking up idiosyncratic patterns that appease automated evaluators rather than human maintainers.

Then I looked for an extreme counterexample: a repository under heavy automated development where readability should logically fall apart, but didn’t.

OpenClaw merged 8,905 pull requests in a fourteen-day period. These were not merely submitted; they landed in production. Its CONTRIBUTING.md includes a prominent notice: “AI/Vibe-Coded PRs Welcome! 🤖”, elaborating: “Built with Codex, Claude, or other AI tools? Welcome! No AI-assistance label or disclosure is required.”

Six hundred merged pull requests a day, open invitations for agent contributions, and no disclosure requirement. Under conventional assumptions, that volume should reduce a codebase to technical debt.

A few lines down, however, the contribution checklist asks authors to confirm they understand what their code does.

Maintaining comprehension at that scale seemed impossible. I inspected their repository to see how they enforce that standard in practice.

Ninety-six specialized check scripts

The repository contains 48,994 files. Inside that tree:

• 96 custom check scripts in scripts/, layered on top of standard linters
• 105 CI workflows
• 30 directory-scoped AGENTS.md files
• 127 modular SKILL.md files
• 4 ratchet baselines in config/

These are not generic linter configs. Each targets a specific failure mode observed in this repository: check-import-cycles, check-duplicates, check-deadcode-unused-files, check-no-conflict-markers, and check-sqlite-transaction-boundary. Thirteen scripts end with -boundary, each locking down an architectural layering rule that would otherwise gather dust inside an ignored design doc.

One script is check-env-var-count.mts. It tallies every instance of OPENCLAW_[A-Z0-9_]+ across the codebase and asserts the count against a threshold committed in config/env-var-count-budget.txt.

A hard budget on total environment variables. I used to complain about agents introducing unprompted configuration flags, assuming that catching such drift required line-by-line diff review. OpenClaw catches it with a basic string counter in CI.

The problem is unchecked code, not AI code

The Readability Spectrum evaluated 2,735 code samples from World of Code and LeetCode against human baselines. The researchers found that LLM-generated code achieves comparable median readability to human-written code. The difference lies in variance: human code maintains a steady baseline, whereas model code fluctuates widely between clean solutions and latent readability debt (gratuitous structural complexity, repetitive comments, obscure API selections).

Prompt engineering barely moves that needle. Providing type signatures, constraints, and style guidelines helps on the margins, but the authors note that prompt design alone has bounded utility.

In contrast, Static Analysis as a Feedback Loop integrated Bandit and Pylint directly into the model’s generation loop. Across ten iterations on PythonSecurityEval using GPT-4o, readability violations dropped from over 80% to 11%. Security vulnerabilities fell from over 40% to 13%, and reliability warnings declined from over 50% to 11%.

While these numbers reflect a single benchmark, the takeaway is clear: with the model and task held constant, output quality improved dramatically the moment automated sensors began evaluating the code during generation.

The Reward Hacking Benchmark explains this mechanism. Across 13 frontier models, reinforcement learning post-training systematically increased exploit-seeking behavior, compounding across longer tool-use trajectories. The model is not speaking an alien dialect; it simply optimizes against the evaluation metrics provided.

Rules that stem from production scars

OpenClaw’s root AGENTS.md spans 123 lines. Two directives stand out:

“Static-analysis fixes strengthen the real type/runtime contract or remove the unsafe operation; do not conceal it with casts, widening, marker types, or property probes.”

“Do not hide failures with retries, longer timeouts, weaker assertions, broader mocks, or altered baselines.”

Directives this specific come directly from real incidents. Someone watched an agent satisfy a TypeScript compiler by sprinkling as any, or watched another relax a test assertion to turn a failing suite green.

The agent was not being sloppy. It simply closed the gap to a passing test suite via the shortest computational path available. That is what optimizing functions do.

Automated checks do not merely inspect quality; they define its functional boundaries. Anything a test suite tolerates becomes fair game for an optimizing agent.

OpenClaw paired those prose directives with automated sensors. The header of config/assertion-safety-baseline.txt explains:

“Per-file counts of production type assertions without a // SAFETY: invariant. Ratchet: counts may only shrink. New non-const assertions need a SAFETY comment.”

The prose forbids hiding type errors with casts; the ratchet counts the raw occurrences. The rule is declared once where the model reads instructions, and enforced where the build will fail.

They apply identical discipline to code review. autoreview is a skill that runs an automated Codex evaluation before PR creation, accompanied by autoreview_test.py and a dedicated test suite. They test their reviewer with the same rigor applied to production code.

Codifying rules into mechanical tests

OpenClaw arrived at this structure through sheer iteration. Ryan Lopopolo’s February essay, Harness engineering: leveraging Codex in an agent-first world, outlines the philosophy systematically.

The OpenAI team operated under a strict constraint: zero lines of manually written code across five months, roughly one million lines of code, and 1,500 merged PRs among a small engineering team.

One sentence anchors their approach:

“When documentation falls short, we promote the rule into code.”

When instructions fail, they avoid rewriting them with stronger adjectives. They convert the requirement into an executable test.

They are similarly candid about why monolithic instruction files fail. Oversized prompt files exhaust token windows, gather organizational dust, and dilute attention. When everything is flagged as critical, nothing receives focus. Crucially, a narrative document cannot be verified mechanically.

Their application architecture reflects this constraint. Domain logic follows a strict one-way layer hierarchy (Types → Config → Repo → Service → Runtime → UI), while cross-cutting concerns like authentication flow through a single Providers interface. Any deviation is blocked by custom linters written by Codex. Regarding those linters:

“Because the lints are custom, we write the error messages to inject remediation instructions into agent context.”

The failure output becomes the prompt for the next self-correction turn. They term these programmatic constraints “taste invariants.”

Enforcing rules about rules

PostHog, an open-source product analytics platform, applies this same harness architecture across 58,577 files, 138 CI pipelines, and dozens of agent instruction configs.

Their repository includes posthog/test/repo_invariants/: 26 whole-repo tests paired with committed baseline files that mirror OpenClaw’s ratchets.

The file that stands out is test_agents_md_enforcement_tags.py. It parses AGENTS.md, extracts every architecture and styling directive, and fails CI if any rule lacks an explicit enforcement tag:

“[lint: <id>] means a linter, semgrep rule, or invariant test blocks it, so CI is the check and this text is only the reasoning.”

“[review] means nothing catches it automatically; a human reader is the only control.”

Every guideline must declare its enforcement mechanism. A follow-up test verifies the accuracy of those tags: if a linter code is placed into ignore or disabled for directories, the build fails. As the maintainers comment: “A stale tag is worse than no tag: a reviewer trusts it and skips the check.”

The documentation explicitly defers to the test suite. The automated check is the operational rule; the prose merely provides context.

Rules tagged with [review] identify precisely where human judgment remains essential. For instance, classifying whether a data model is scoped requires contextual business understanding that automated scripts cannot parse. The tag indicates exactly which responsibility falls on the reviewer.

Verification as the scaling bottleneck

This dynamic scales down to individual contributors. Lauren Tan, previously with Meta’s React team, Cursor, and currently Grok, merges roughly 2,500 production PRs monthly using her personal toolkit, pstack. A foundational tenet of her workflow is direct: “we prefer to give agents tools rather than just markdown.”

Borrowing Andy Grove’s production line analogy from High Output Management, she identifies verification as the limiting step of software output. Accelerating code generation without scaling verification is like toasting bread faster while the eggs still take three minutes to cook.

At Cursor, she authored skills enabling agents to interact with running dev builds through the Chrome DevTools Protocol: dispatching clicks, inspecting accessibility trees, taking screenshots, throttling performance, and profiling memory heaps. Her principle:

“agents should get the same signals human engineers use to write good code: compiler diagnostics, lints, static analysis… Agents need to be able to see what I see, and use the tools I use.”

Attempting to scale throughput before establishing verification mechanisms creates unmanageable review backlogs. Verification must precede parallel execution.

Common agent failure modes and programmatic mitigations

Across high-throughput repositories, agent behaviors converge on a familiar set of challenges, met with consistent architectural solutions:

Observed Agent Behavior	Underlying Cause	Effective Repository Mitigation
Pacifies the checker via casts, widened types, or relaxed assertions	Finds the cheapest path to an exit code of zero	OpenClaw forbids casts in prose while tracking occurrences via assertion-safety-baseline.txt. Hardened test harnesses reduced exploit rates from 6.5% to 0.8%.
Writes hollow tests that execute paths without asserting invariants	Coverage metrics are visible targets; assertion strength goes unmeasured	Mutation testing, assertion ratchets, and explicit rules against weakening test checks.
Re-implements utilities instead of reusing existing modules	Reuse requires broad awareness of the codebase; re-implementation does not	Duplication checkers such as PostHog’s ci-duplication.yml and duplicates-baseline.txt.
Violates architectural layering boundaries	Layer boundaries described solely in text lack compile-time enforcement	Enforce strict one-way layer dependencies, boundary linters, and baseline files like ingestion-boundaries.baseline.json.
Expands configuration surfaces unprompted	Individual config options seem locally harmless	Enforce strict environment variable budgets via check-env-var-count.mts.
Accumulates dead code, orphan TODOs, and stale docs	Outdated documentation does not trigger build failures	Automated reconciliation tasks, dead code analyzers, and doc-gardening agents.
Generates bloated, coupled files	Readability is rarely tracked as an explicit fitness metric	File-length budgets enforced by ratchets, accompanied by actionable lint errors guiding modular decomposition.
Overwhelms human review capacity	Code generation scales arbitrarily while human attention remains fixed	Pre-PR automated reviews, multi-agent review loops, and explicit tagging of rules requiring human inspection.

Two patterns unify these strategies:

The ratchet baseline. Retrofitting strict rules into established codebases often stalls over cleanup costs. Ratchet baselines solve this by recording current violations and forbidding new ones. In config/max-lines-baseline.txt:

“Files currently allowed to exceed the oxlint max-lines budget. Ratchet: this list may only shrink. Split files; never add entries.”

You don’t need to refactor the entire repository before introducing a check. You lock in existing technical debt as the ceiling and require all future changes to trend downward.

Actionable failure messages. The most effective harnesses design error outputs as prompts. When an agent reads CI logs with greater precision than design documentation, the error output should describe the exact remediation path.

Empirical verification

Controlled studies reinforce these observations:

Meta’s agentic program repair study evaluated 1,589 test-failure tasks across internal codebases over three months:

• ReAct agent alone: 28.5% solve rate
• Agent with static analysis feedback: 34.1% solve rate
• Agent with test execution feedback: 43.9% solve rate
• Agent with both feedback loops: 42.3% solve rate, with the error rate dropping from 1.0% to 0.2%

The test harness alone accounted for a fifteen-point performance uplift, and 25.5% of generated patches were merged into production by human reviewers.

Furthermore, the Reward Hacking Benchmark tested four harness interventions across 13 models. Exploit attempts dropped from 6.5% to 0.8%, while overall task success remained constant (83.2% vs. 82.8%, p > 0.5). Imposing mechanical constraints did not harm problem-solving capability.

Operational caveats

Automating checks introduces its own failure modes:

• Metrics can be gamed: Birgitta Böckeler noted that 100% statement coverage in an agent-authored suite still left 13 surviving mutation bugs. Coverage figures alone do not guarantee correctness.
• Constraints can conflict: Narrow rules on file lengths can inadvertently encourage agents to push complexity into deeply nested property chains rather than genuine decoupling.
• Harness maintenance requires restraint: OpenClaw actively rations new checks in AGENTS.md, requiring every rule to have an unambiguous invariant and clear ownership.
• Velocity is distinct from code quality: High merge counts prove that the automation pipeline runs efficiently; they do not guarantee that every merged PR is elegant.

Navigating the stylistic tradeoff

The OpenAI team noted an uncomfortable reality:

“The resulting code does not always match human stylistic preferences, and that’s okay. As long as the output is correct, maintainable, and legible to future agent runs, it meets the bar.”

They optimized their codebase for agent readability rather than human browsing. Information that cannot be fed into an in-context window does not exist for an autonomous harness.

While human engineers may instinctively dislike unfamiliar code patterns, when a team ships a million lines of working software without writing code by hand, stylistic elegance takes a back seat to mechanical correctness and agent comprehension.

Yet the question of long-term maintainability remains nuanced. OpenClaw accepts AI code without disclosure while expecting contributors to understand their submissions. PostHog mandates explicit disclosure alongside full technical understanding. OpenAI avoids human verification altogether because their harness handles the entire loop.

High-throughput teams agree on the necessity of automated harnesses; they continue to debate how much human comprehension is required in the loop.

Practical application

The central lesson is about execution order rather than raw script count.

Descriptive documentation decays unless verified by automated tooling. A rule against concealing type errors with casts is only as good as the script counting type assertions.

Before drafting project instructions, ask whether the constraint can be expressed as an automated test. If it cannot be checked programmatically, it is a loose guideline that an autonomous agent will likely optimize around.

When a requirement can be tested, the check is often simpler than the essay explaining it. A regex checking environment variable counts provides far more reliability than paragraphs pleading for minimal configuration surfaces.

As foundation models advance, the need for brittle steering will diminish. But the core engineering principle holds: codify critical architectural invariants into runnable checks rather than passive prose. Standards that cannot fail a build are merely hopes.