Agents in the Workshop, Not in the Product

Agent capability keeps going up.

And yet, looking back at what I’ve actually shipped, there is an uncomfortable admission: I have never built a product with an agent running inside it.

That gap is what I want to explore here.

Two very different meanings of “agents in the loop”

When people talk about “agents in the loop,” they usually conflate two distinct problems:

1) Agents in the loop of how the product gets built. This is the world I live in every single day. Claude Code writes code while I review diffs. An LLM drafts translations for this blog. Another helps me evaluate database schema decisions. The construction process has agents threaded throughout. Claude Code remains one of the best agent-in-the-loop tools I’ve used, precisely because the agent stays inside my personal developer feedback loop.

2) Agents in the loop of how the product actually runs. This is the production world: after deployment, agents remain active in the system, reasoning, deciding, and acting on behalf of live users.

The first category is already transformative. The second, at least in my own engineering practice, has barely started.

What I’ve actually shipped

To be concrete about my own track record:

• ai-sota-feed-bot: This was the first product I built using Claude Code. It’s a static feed generator. While the pipeline was created with agents, users only interact with pre-rendered static content. There is no agent in the runtime.
• This blog: Every article is polished, translated, and adapted with AI help, but readers land on plain static HTML.

My agent usage has centered almost entirely on the workshop rather than the live product.

Why this won’t stay this way

The future will inevitably involve multiple autonomous agents acting on behalf of humans: scheduling, filtering, negotiating, summarizing, tutoring, and intervening. They won’t just be experimental demos; they will become load-bearing components of products people rely on every day.

We are already seeing early signs. AWS recently detailed a DevOps agent that autonomously investigates production incidents and coordinates incident response. That agent sits directly in a live operational loop, not in an offline dev environment.

The surface area entering production is massive: agents making decisions, allocating budgets, and communicating with peer agents. Most of us, myself included, are still figuring out how to build reliable software around those primitives.

Why it remains hard to ship an agent inside a product

The common failure pattern is familiar:

You wire up a model to a tool. The initial prototype looks magical: it generates sensible reasoning steps, and everyone in the demo room is impressed.

Then you attempt to harden it for real traffic, and structural cracks appear immediately:

• Outputs become overly complex and vague. Responses sound plausible on the surface, but drift away from what the user actually requested.
• Verification is missing. The system cannot determine whether its action was correct, and neither can the operator once volume exceeds ten requests a day.
• Cost stays invisible until the invoice lands. Prompt growth, retries, expansive context windows, and multi-turn tool loops cause spending to scale non-linearly.
• Performance defies standard monitoring. Tail latencies, partial failures, timeouts, and silent model drift rarely register clearly on traditional APM dashboards.

These aren’t optional polish items. They represent the dividing line between having a novelty AI demo and running software that customers can actually depend on.

Right now, assembling that stack is tedious. Teams stitch together evaluation frameworks, prompt registries, distributed traces, token meters, and guardrails, hoping the seams hold under pressure. Without those components, an agent system is not production-ready.

Agents must become predictable machine systems

Every complex system previously adopted as societal infrastructure (payments, search, messaging, maps, and recommendations) had to become trustable and consistent before it became standard utility.

Raw intelligence was never sufficient. Systems had to be:

• Bounded: clear limits on what they will and will not execute
• Observable: clear visibility into real-time actions and cumulative costs
• Recoverable: sensible degradation paths when failures occur
• Attributable: clear audit trails explaining why a decision was reached

Agents aren’t quite there yet. This isn’t because models lack capability, but because the systems surrounding them rarely meet production standards.

If agents are to operate alongside us the way payment processors or search engines do, this surrounding harness is what must be built: consistent outputs, predictable behaviors, visible costs, and resilient error recovery. The goal isn’t merely being smart; it is being dependable.

Why this section exists

Agents in the Loop documents that transition.

Rather than focusing on using agents to write code faster, this section explores what happens when an agent moves from the workshop directly into the running product: what breaks under load, which constraints are non-negotiable, what production-grade harnesses look like, and what I learn trying to build them.

I don’t have all the answers yet. I have a clear view of the gap, a handful of working hypotheses, and a strong hunch that this is where modern product engineering will be defined.