What a Merged PR No Longer Proves
In February I wrote Where Did the Learning Go? after a day of shipping real features with an agent and learning nothing. My answer then was to build the measurement: turn decisions into hypotheses and check the numbers later.
I still think that is right, but it only answers half the question.
A cost dashboard tells me whether a decision worked. It does not tell me whether I understand why it worked, or whether I could predict what happens when the conditions change. Those are different things, and I keep running into the second one.
Code used to be scarce, and the PR was a receipt
Terence Tao’s Math 2.0 talk gave me better words for this than I had.
His framing is that mathematics ran for a long time on proof scarcity. Proving something was hard and rare, and a lot of the field’s habits and incentives grew up around that fact. AI is moving math into proof abundance. And problem solving, the most visible goal of research, was always a proxy for something deeper. The proxy was accurate while proofs were scarce. Tao argues it drifts out of alignment once they are abundant.
The part that stuck with me is his description of open problems as lighthouses. You do not sail toward them just to arrive. You use them to explore the area around them, and what you learn on the way, including the failed attempts, is often worth more than the answer.
Software ran on the same kind of scarcity. A merged pull request bundled several things: someone understood the problem, worked through the constraints, picked an approach, and made it run. The code was the visible part. A lot of the learning happened on the way to it.
Chasing a slow request could end in a one-line fix. It could also teach the team where work piles up, which assumptions break under load, and which measurement actually tells two explanations apart. That second output is what shaped the next design decision.
Figure 1. The task produces a change and a better ability to handle the next task. This is my adaptation of Tao’s diagram for engineering.
The link between the PR and the understanding was never perfect. You could always ship code you did not fully understand. But when implementation took real effort, the output usually carried some evidence of that effort.
With agents, a working implementation can show up before anyone has built the understanding that writing it used to require. The code is just as useful, but its existence now says much less about whether anyone understood the problem.
Figure 2. The failure mode I worry about: the change arrives faster, and the understanding behind it quietly stops renewing. It can happen, and nothing forces it to.
The PR count is impressive and still not enough
Lauren Tan reports merging roughly 2,500 production PRs a month. Anthropic reports a 67% increase in PRs merged per engineer per day as Claude Code adoption grew internally. These are self-reported numbers with different units and contexts, so neither works as a ranking or as a controlled estimate of what AI does to productivity.
I also do not want to wave them away. There is a lot of skill behind that kind of throughput: picking the right tasks, building verification, keeping parallel work from colliding. I wrote about some of it in The Agent Optimizes for What You Check.
But the count cannot answer the questions I actually care about. Did users get something better? How much review and rework did it take? Is the system easier to operate now? Can the team reason about the next failure?
Even PR size changes what the number means. Split one change into ten PRs and the count goes up tenfold while the outcome stays the same.
Anthropic says this directly: pull requests alone are an incomplete measure, and the metric is meant to sit next to things like DORA metrics. I agree, and what I would do is match the evidence to the claim:
- For throughput, compare work of similar scope.
- For outcomes, look at behavior users or operators actually feel.
- For understanding, ask someone to predict an unfamiliar case and explain the mechanism.
- For cost, count the human attention and compute it took to reach an acceptable result.
A passing check has a boundary
Software makes this sharper, because each of our checks proves something different. A compiler checks encoded rules. A test checks the behavior you chose to test. A formal proof establishes a property under a model. Production tells you how the system behaved under the conditions you happened to observe.
All of it is evidence. None of it tells you that the model matches reality.
Tao has a thought experiment along these lines. An AI finds a cancer treatment, its model’s prediction is confirmed in Lean, and it passes a stage 3 trial. Nobody knows how it was found. Before it goes into your bloodstream, would you want at least one human expert who understands the mechanism?
Here is a smaller, made-up software version. An agent designs an inventory protocol and proves it can never oversell under its stated model. The proof is valid. But an older writer in production changes inventory without going through that protocol. The proof never covered it.
Noticing that mismatch is engineering work, and producing checked artifacts faster does not make it go away. Adding an approval button does not either. Approval means something only when the person approving has enough context to judge the consequences. They do not need every line memorized, but they do need to understand the mechanisms and assumptions the decision rests on.
Use the cheap code to build the understanding
This is where I got more optimistic, because the same abundance that broke the proxy also makes it cheap to build understanding on purpose.
Karpathy suggested asking models to explain things in ASD-STE100, the Simplified Technical English spec originally written for aerospace maintenance documentation. The full spec is strict, so he sometimes asks for something like “80% of the way to ASD-STE100.” It changes the style and guarantees nothing about correctness. What I like about it is that plain, short sentences leave less room to hide a step you did not follow.
His bigger point is about the medium. Ask for a diagram instead of prose. Ask for an interactive HTML page. Ask for a custom explainer video. When code is this cheap, you can build a tool to answer one question, use it, and throw it away. A tool that would never have been worth building before is now a normal part of an investigation.
Tao lands in a similar place. His examples of good “Math 2.0” uses of AI include what he calls open exposition problems: finding motivated explanations for results whose existing proofs are opaque. The software version of that is obvious to me: we have plenty of working code whose explanation is the missing piece.
Thariq Shihipar’s Learn Quiz prompt covers the other half. It asks the model to teach a session incrementally: have the human restate their understanding first, fill the gaps from there, and quiz on motivation, design decisions, business logic, and edge cases. It can pull up code or have them use the debugger when needed, and it does not end the session until they have shown they understand.
Plain language and diagrams make an explanation easier to take in. Restating it and answering a new question show whether you can actually use it. Running something checks whether the explanation matches the system.
Figure 3. AI can help at every step of this loop, including building the tools that test what we think we know. A fluent explanation on its own does not close it.
That is the version of “Software Engineer 2.0” I find useful: a workflow that ships working software and also deliberately builds the ability to understand and change it. (Karpathy’s older “Software 2.0” is a different idea, about neural networks replacing code.)
Try it: the response that never arrived
Here is a small example. A server commits a payment, but the response never reaches the client. The client times out and retries.
Before reading on, predict what happens. Does the retry charge the customer again? What assumption makes your answer true?
Figure 4. A timeout does not tell the client whether the first request committed. This is a simplified model, not any particular payment system.
Here is a toy model so the question has an answer. The server stores the idempotency key and the result together, atomically. It keeps that record for 120 seconds. Nothing else deduplicates requests. The first request has already committed and its response was lost.
How many charges does the customer end up with in each case? Commit to an answer before you open it.
Retry with the same key after 30 seconds
One. The record still exists, so the server returns the saved result without charging again.
Retry with the same key after 300 seconds
Two. The record expired at 120 seconds. As far as the server can tell, this key is new, so it runs the payment again.
Retry with a new key after 30 seconds
Two. A new key has no saved result. The idempotency layer is working exactly as designed, and it still cannot help.
Retry with the same key at exactly 120 seconds
It depends on whether the record expires at 120 seconds or after it, and my description above does not say. “We use idempotency keys” sounds like an answer until someone asks about the boundary, and then you have to go read the real code.
A tool like this is something an agent can build in a few minutes. Let people vary the key, the delay, and the retention period, and make them predict before they reveal. A changed key or an expired record pushes the conversation past “we use idempotency.”
It is still only a model, though, and a polished interface reproduces a wrong assumption just as faithfully as a right one. To connect the lesson to a real service, you need an isolated reproduction against the actual implementation, real traces, or some other authoritative evidence.
If the prediction fails, you fix the explanation. If it holds, you keep the assumption, its scope, and a regression test. The UI can go away afterward; the understanding should stay.
This is one more reason PR counts miss part of the picture. Some of the most useful AI-written software may never ship as a feature. It exists so someone can understand something else.
What I want to change in my own workflow
For a consequential change, I want the handoff to include the assumption that makes the approach valid, the evidence behind it, and a case that would break it. The depth should match the risk. A typo fix does not need a tutorial.
For a part of the system that people find hard to reason about, I want to ask an agent to build an explanation around one concrete question: a request timeline for retries, a dependency map for a migration, or a small experiment matrix to separate competing latency theories.
Then I want someone who has to maintain that part to predict a new case. Where the prediction and the result disagree is where the next gap in understanding is.
This also changes what deserves credit. A useful contribution might be shipping a feature. It might also be finding out that a feature is unnecessary, deleting a mechanism, or producing a checked explanation that makes the next ten changes safer. Each of those needs evidence that fits what it claims.
Cheap implementation is usually framed as more patches, faster. I think the bigger opportunity is that investigations and explanations that used to cost too much are now affordable.
The question I want to carry into the next task: once the change works, what will we understand well enough to make the next decision better?
The inventory and retry examples are hypothetical, and the workflows here are proposals, not measured evidence that they improve learning. This continues an earlier thread: where did the learning go and how humans learn when agents scale.