Claude Code Agent Harness Engineering

Last updated: Oct 4, 2026

Confidence: Tier 1 for the architecture framing; Tier 2 for cross-product performance claims. Controlled studies show measurable harness effects, but do not establish that the harness always matters more than the model.

Reading time: ~35 minutes

The core claim

A raw LLM is not an agent. It becomes one when connected to a harness. The useful unit of evaluation is therefore the model-harness pair, not either component in isolation.

The 2026 Agent System and Harness Design survey decomposes an execution harness into observation, context, control, action, state, and verification. Code as Agent Harness adds a code-centric view: code is not only an output but the executable substrate for tools, memory, control, coordination, and verification. These are useful taxonomies, not performance proofs.

Controlled evidence supports a narrower claim. In The Scaffold Effect in Coding Agents, two fixed models were tested across three harnesses on 50 Terminal-Bench Pro tasks. Harness choice changed tokens per solved task by up to 40 times, while paired pass-rate differences stayed within 0 to 8 percentage points and were mostly not statistically significant. The harness can dominate cost and failure behavior without dominating task accuracy. Conversely, model quality or model-harness compatibility can remain the binding constraint. Report the pair, the task set, and the budget.

Tokens per solved task are not a billing multiplier: input/output prices, caching and human review require separate accounting. A non-significant success-rate difference also does not establish equivalent quality. Keep the study’s resource and outcome measures distinct when selecting a local baseline.

This page uses agent harness in its runtime sense: the system that owns the agent loop, tools, context, state, and permissions. A repository can also provide a repository harness around that runtime: its instructions, setup, task state, and verification gates. The distinction matters because a project can improve its repository harness without replacing Claude Code, and a team can switch runtime harnesses without discarding every project practice.

This page covers what is inside the runtime. For the repository layer, see Repository Harness Engineering. For feedback loops, executable workflow graphs, stopping rules, and judgment boundaries, use Loop & Graph Engineering. For a dated comparison of specific products across CLI, IDE, and cloud, see the Agent Harness Landscape.

A user goal moves through six harness stages: context building, LLM reasoning, policy gating, guarded tool execution, verification, and an accepted result. Observability spans every stage, constraints govern policy and runtime, and feedback returns accepted results to the context builder.

The LLM is one stage in the reliability loop. Context, policy, guarded execution, verification, observability, constraints, and feedback determine whether the model’s proposal becomes an accepted result.

Choose the right entry point

Question	Canonical page
What does a runtime harness contain, and how do its controls work?	This Agent Harness Engineering page
How should loops, workflow graphs, state transitions, stopping rules, and judgment be designed?	Loop & Graph Engineering
How should independent Claude Code sessions exchange evidence without spreading drift?	Cross-Session Messaging: Coordination safety
Which runtime, orchestrator, framework, or adjacent project should I compare?	Agent Harness Landscape
What does a specific coding-agent product support?	Agent Tools: Beyond Claude Code
What distinguishes a runtime, repository, evaluation harness, and orchestrator?	Glossary
Which Claude Code release introduced a behavior?	Claude Code Releases and its machine-readable history

The pages remain separate on purpose. Engineering concepts change more slowly than product inventories, licences, feature evidence, and popularity signals. Combining both would make the architectural reference inherit the catalog’s dated snapshot and maintenance cycle.

Four Layers, Four Responsibilities
Three Foundational Properties
The Nine Components
The Lethal Trifecta: Security Model
CI/CD Agentic Patterns
Digital Twin Testing
Observability Stack
Test Distribution and Component-Stacking Anti-patterns
Creator-Verifier Pattern
Reference Architecture
Practitioner Video Evidence
Harness Optimizers and Meta-Harnesses

0. Four layers, four responsibilities

The word harness is overloaded. This guide uses four layers so a model, a runtime, a repository setup, and a fleet manager are not treated as interchangeable products.

Layer	Responsibility	Examples	What it does not replace
Model	Generates and reasons over tokens	Claude, GPT, Gemini, DeepSeek, Qwen	Tools, policy, durable state, or execution
Runtime harness	Runs the agent loop and mediates tool use, context, permissions, and recovery	Claude Code, Codex, Gemini CLI, DeepSeek Harness, OpenCode	Project-specific instructions and delivery gates
Repository harness	Makes one codebase legible and verifiable to a runtime	`CLAUDE.md`/`AGENTS.md`, `init.sh`, lockfiles, task state, tests; Liza’s installed contracts and guardrails	The runtime’s model routing, tool loop, or sandbox
Orchestrator	Coordinates multiple runs, workspaces, or harnesses	Symphony, Vibe Kanban, AgentBox, Proliferate, Liza	The underlying runtime agent loop

The boundary is practical. If a product can only schedule or inspect Claude Code and Codex sessions, it is an orchestrator. If it supplies an iterative model-and-tools loop itself, it is a runtime harness. If it is committed with the repository and tells any compatible runtime how to work safely, it is a repository harness.

A product can span two layers without owning all four. Liza installs behavioral contracts, skills, settings, and guardrails into a repository, then coordinates external coding-agent CLIs through worktrees, durable task state, leases, doer/reviewer roles, recovery, and merge gates. Claude Code, Codex, or another selected CLI still owns the inner tool loop. The Landscape profile records the pinned code evidence and security limits.

The term meta-harness is overloaded. In the optimizer research covered by this page, a harness optimizer or meta-harness proposes changes to a target harness, evaluates candidates, and promotes or rejects versions. Some products and practitioner articles use meta-harness for a different role: a common client that dispatches tasks to Claude Code, Codex, Cursor, Pi, or another runtime. Under this guide’s taxonomy, a system that selects or coordinates existing harnesses without modifying and evaluating them is an orchestrator or control plane, not a harness optimizer.

Loop, graph, harness, and orchestrator are different views

These terms answer different questions. Treating them as competing product categories hides the boundary that matters.

View	Primary question	Core artifact
Loop engineering	What feedback repeats, and what stops the repetition?	Goal, action, observation, verification, stopping rule
Graph engineering	Which nodes, edges, state transitions, joins, and checkpoints are executable?	Versioned workflow graph and shared state schema
Harness engineering	What context, tools, policy, state, verification, and recovery bound execution?	Runtime and repository controls
Orchestration	How are runs, workspaces, queues, budgets, and people coordinated?	Scheduler or control plane

Loop Engineering is a useful practitioner label for replacing repeated manual prompts with a system that finds work, dispatches it, checks it, and decides what happens next. It is not a formal standard. Graph Engineering in the Era of LLM Agents and What Makes Prompts a Graph propose a broader graph-engineering vocabulary, but both are recent preprints. The mechanisms are older than the label: the official LangGraph Graph API already models shared state, nodes, fixed or conditional edges, and parallel super-steps, while its persistence layer provides checkpoints, interruption, replay, and fault recovery.

A loop can be encoded as a graph. A graph can coordinate several loops. A harness can execute either while adding permissions, context, tools, verification, and recovery. An orchestrator can schedule several harnesses without owning their inner loops. Compare products inside the same layer before comparing feature counts.

1. Three foundational properties

arXiv 2605.18747 (“Code as Agent Harness”, May 2026) formalizes three properties that distinguish a harness from a simple LLM wrapper:

Executability: the harness can verify what the agent actually did, not just what it said it would do. A harness that only logs prompts and completions is not executable, because it cannot distinguish a successful tool call from a hallucinated one.

Inspectability: when something fails, the harness produces actionable diagnostic output. Stack traces pointing to the prompt assembly step. Friction events tagged with which rules and skills were active. Token consumption by component. Without inspectability, debugging an agent failure requires reconstructing the session from logs, which is expensive and often incomplete.

Statefulness: the harness maintains continuity between sessions. Session state is externalized, not held in the model’s context window. When the agent resumes after a context reset, it can reconstruct where it was without the human providing a full briefing. Anthropic’s own telemetry shows the 99.9th percentile session duration passed 45 minutes in January 2026, up from 25 minutes in October 2025. At that length, statefulness is not optional.

2. The nine components

These nine components appear across Claude Code, Anthropic SDK, OpenAI Agents SDK, LangGraph, AWS Bedrock AgentCore, and Factory.ai Missions. No single tool implements all nine in exactly the same way, but the structure is consistent enough to use as an evaluation checklist when assessing a harness against a new tool or framework.

2.1 While-loop engine

The main loop: perceive (read context, tool outputs, latest user instruction), plan (call LLM with assembled prompt), act (execute tools). This is the heartbeat. Anthropic SDK, OpenAI Agents SDK, and LangGraph implement it differently (Anthropic is streaming-first, LangGraph is graph-based), but all three have this loop as the core abstraction.

Where problems emerge: loops that don’t cap iterations, loops that don’t handle LLM refusals or ambiguous tool calls, loops that let the context grow without summarization until they hit the window limit and abort.

Three loop horizons

A useful practitioner model separates three feedback horizons. The terms are not a formal standard, but they expose where a correction should become durable.

Horizon	Boundary	Typical controls	Where it usually lives
Inner loop	One agent run	Tool results, targeted tests, skills, local checks	Runtime plus repository harness
Outer loop	Task or pull request	Agentic QA, full test suites, deeper review, independent verifiers	Repository harness plus CI
Meta loop	Across many runs	Failure mining, recurring review analysis, harness search, rule and verifier updates	Team process, optimizer, or meta-harness

The AI Native DevCon interview on harness engineering describes the inner loop as fast feedback while the agent works, the outer loop as slower checks around the pull-request boundary, and the meta loop as continuous improvement across failures. This model turns repeated human correction into a maintenance signal. A review comment that recurs should become a test, rule, skill, hook, or verifier at the earliest loop that can catch it.

Build one bounded loop before adding autonomous layers. In a 2026 context-engineering interview, Dex Horthy describes a lights-off software factory that removed human testing and review, accumulated failures, and was shut down. That is a practitioner retrospective, not a controlled study, but it identifies a concrete failure mode: autonomy expanded faster than verification and recovery. Addy Osmani gives the same rule a name, “parallelize last”: copy a factory’s parallelism only after one unit has a dependable judge, a recovery path, and a review format people can absorb, because “parallelism multiplies the bottleneck you already have”, which is usually human review capacity (Brownfield Agentic Engineering, September 2026). He also narrows what the meta loop should produce: move a recurring correction into a lint rule, hook, type, test, or skill, and “keep prose for constraints that cannot be enforced mechanically”.

The meta loop can now be automated. Meta-Harness searches harness code using prior source, scores, and execution traces. Agentic Harness Engineering represents editable components as files and couples every change to a prediction that the next evaluation can falsify. Automation does not remove the need for held-out tests or budgets. It makes those boundaries more important.

2.2 Context management

What goes into the prompt on each loop iteration: conversation history, tool outputs, retrieved memory, current task state, rules from CLAUDE.md. The challenge is that context is finite and expensive. Strategies:

Compaction / summarization: replace earlier turns with a compressed summary. Claude Code compacts automatically near its context threshold and also exposes /compact for an explicit compaction. PreCompact and PostCompact hooks let teams inspect or react to that boundary.
Sliding window: keep the last N turns verbatim, summarize everything before.
Retrieval-augmented context: retrieve relevant chunks from long-term storage rather than carrying everything in the window.

The ACE pipeline (see context-engineering.md §6) is the Config-Persistence layer above context management: it governs what rules and skills are loaded across sessions.

Agentic Context Engineering treats context as an evolving playbook rather than a repeatedly rewritten summary. The paper reports gains of 10.6% on agent benchmarks and 8.6% on finance tasks through incremental generation, reflection, and curation. Cite those as method results, not proof that adding more context is better. The operational principle is smaller: preserve useful detail, attach feedback to an inspectable artifact, and test the evolved context against held-out tasks.

Context beyond the repository

Repository files are only one source of truth. Product decisions may live in Linear, interface constraints in Figma, component behavior in Storybook, and operational history in incident systems. Marc Sloan’s 2026 talk on product and design constraints argues that this external context must become accessible to the agent without silently drifting away from its source.

Access pattern	Strength	Failure mode
Live tool or MCP access	Fresh data and direct attribution	Availability, permission, and prompt-injection risk
Versioned repository snapshot	Reproducible and reviewable with the code	Snapshot can become stale
Synchronization bridge	Combines freshness with a local artifact	Sync failures can create two conflicting truths

Record the source, retrieval time, and freshness expectation for external context. If an agent cannot tell whether a design decision is current, the context builder has moved uncertainty into the prompt instead of resolving it.

2.3 Tool registry

The catalog of available tools: name, schema, description, permissions, cost estimate. A static tool registry loads all tool schemas on every call. A dynamic registry (“tool search on demand”) loads only what the current task plausibly needs.

Anthropic’s internal data (cited in the Fowler article, source: practitioner post) cites a 37% reduction in token usage from dynamic tool dispatch versus static loading. This number is not independently replicated, but the directional claim is credible: giving the model 40 tool schemas when it needs 4 adds noise and cost.

For sensitive MCP (Model Context Protocol) tools, put authentication and authorization at the execution boundary. An identity-aware gateway is one production pattern, but it is not a default property of MCP or Claude Code. The required control depends on the tool’s data, side effects, and deployment model. See MCP Servers Ecosystem and Security Hardening.

2.4 Sub-agent management

Delegation to specialized sub-agents with their own context windows and task scope. The orchestrator spawns a worker, provides a bounded task description, and receives a structured result. The worker does not share the orchestrator’s full context; it receives only what it needs.

Factory.ai Missions formalizes this: an orchestrator agent decomposes requirements, delegates implementation to workers, and routes completed work to adversarial validator agents. On a documented Slack clone project, independent validators caught 81 problems before any code was merged, generating 34% of the implementation work as “fix features.”

Claude Code subagents inherit the parent session’s permission mode by default. That is runtime behavior, not proof of least privilege. Narrow each agent with tools, disallowedTools, and permissionMode, and use isolation: worktree when parallel writers must not share a checkout. See Tools Reference, Agent Teams, and the worktree isolation definition.

2.5 Native tools and loadable skills

Claude Code exposes native tools for file access, search, shell execution, web access, delegation, and other runtime operations. Skills are different: they are loadable instruction and resource modules invoked through the Skill tool. A skill can include scripts and deterministic checks, but the skill itself is not inherently deterministic because the model still interprets its instructions and chooses actions.

Test deterministic scripts and validators with ordinary assertions. Test the model-mediated part of a skill with behavioral tasks, expected boundaries, and human or calibrated evaluator review. The Tools Reference documents the runtime tool surface; the skills examples show the separate packaging model.

2.6 Session persistence

State that survives context resets and session interruptions. Not the same as long-term memory (which is a higher-level concept). Persistence at the harness level means: the agent can reconstruct its current task state from externalized artifacts rather than from the in-context conversation history.

Factory.ai Missions uses a shared artifact layer (validation contracts, feature lists, skill definitions) to survive the context limits of multi-day missions. E2B and Northflank provide this at the infrastructure level via persistent sandbox state. Anthropic Claude Managed Agents provide it as a product feature with checkpointing.

2.7 Dynamic prompt assembly

The step that turns the current state (task description + relevant context + tool definitions + memory + rules) into the actual prompt sent to the LLM. This is not standardized across frameworks. LangChain, LangGraph, and the Anthropic SDK each have different abstractions.

The places where assembly goes wrong: rule injection that conflicts with the user instruction, tool schemas that overlap in ways that confuse the model’s selection, memory retrieval that surfaces outdated context. Harnesses that make assembly visible (logging the final assembled prompt, not just the response) are dramatically easier to debug.

Ryan Lopopolo’s 2026 harness-engineering talk frames the repository as an environment that must make the team’s standard of good work legible and surface the right context just in time. The operational test is simple: if the same review feedback appears twice, decide whether it belongs in a formatter, linter, test, architecture decision record, skill, or verifier. Prompt text is only one possible control.

2.8 Lifecycle hooks

Injection points that fire at defined runtime events. Claude Code does not expose generic pre-LLM and post-LLM hooks. Its event model includes instruction loading, permission requests, tool use, compaction, subagent and teammate lifecycle, worktree changes, session lifecycle, and failure events. The exact list and blocking semantics live in Hooks Events Reference.

Hooks are where you insert observability instrumentation, permission validation, rate limiting, output sanitization, and audit logging. Only events with documented blocking semantics can stop an action. Async hooks are appropriate for telemetry that does not need to interrupt the loop.

2.9 Permission enforcement

Every consequential action needs an enforcement boundary before execution. Depending on risk, that boundary can combine explicit approval, permission rules, sandboxing, network policy, scoped credentials, and post-execution verification. Human review remains useful for high-impact decisions, but repeated approval prompts are not a complete isolation strategy.

Two useful structural mechanisms are:

Sandbox isolation: the agent runs in an environment where destructive actions are physically impossible, not just disallowed by policy but literally impossible given the network, filesystem, and process constraints. Kubernetes agent-sandbox, E2B microVMs, and Northflank BYOC runners implement this at the hardware level.
Identity gateway: sensitive tool calls are authenticated at call time with a scoped session credential rather than a broad static API key. Strata Maverics and Microsoft Entra Agent ID implement this pattern with OAuth OBO (On-Behalf-Of) flows that scope permissions to the current task context.

Claude Code now supplies additional primitives for this layer: sandbox.network.strictAllowlist denies non-allowlisted sandbox traffic without prompting, worktree isolation separates concurrent edits, and --restricted removes command and code execution plus WebFetch unless explicitly restored. These controls reduce exposure, but they do not establish that every MCP integration uses per-session identity or that every action is sandboxed.

A 2026 interview with Claude Code creator Boris Cherny describes the permission system as a layered combination of classifiers, static analysis, pattern allowlists, conservative defaults, and a human decision when the system is unsure. This is useful design rationale from a product creator. Current behavior still needs to be checked against official documentation and the Claude Code release history.

Claude Code implementation checkpoint

The nine-component model is product-independent. The table below maps it to Claude Code behavior through v2.1.250 without turning this page into a second release log.

Harness concern	Current Claude Code mechanisms	Release evidence
Context and state	Automatic and explicit compaction; `PreCompact` and `PostCompact` hooks; automatic memory	v2.1.247, v2.1.105, v2.1.76, v2.1.59
Tool and prompt assembly	Deferred tool loading, `alwaysLoad` MCP option, `InstructionsLoaded` hook	v2.1.121, v2.1.69
Delegation	Background subagents, inherited permission mode, concurrency and nesting limits, conversation forking, prompt-cache inheritance	v2.1.198, v2.1.212, v2.1.217, v2.1.219, v2.1.232
Isolation and policy	Worktree isolation and later hardening, strict network allowlist, `--restricted` launch mode	v2.1.49, v2.1.50, v2.1.222, v2.1.219, v2.1.248
Recovery and coordination	Cross-session messaging, per-agent cache TTL, subagent model fallback	v2.1.224, v2.1.248, v2.1.247
Inspection and control	Permission, tool, compaction, instruction, failure, agent, teammate, and worktree hook events	v2.1.69 to v2.1.219

Use the human-readable release history for interpretation and claude-code-releases.yaml for version-level lookup. reference.yaml provides stable topic routes into this page and the surrounding guide.

3. The Lethal Trifecta: Security model

Simon Willison coined this term in 2025 (see martinfowler.com/articles/202508-ai-thoughts.html):

Private data + untrusted content + external communication = documented exfiltration vector.

Any two of the three are manageable. All three together, without structural isolation, create a path where an attacker plants instructions in data the agent will read (a document, a code comment, a Jira ticket), the agent processes those instructions using its access to private data, and then uses its communication capabilities to exfiltrate.

This is not theoretical. GitHub Security has documented prompt injection attacks in Copilot via malicious repository content. The defense is not better prompt engineering; it is structural isolation.

AgentDojo makes the threat measurable with 97 realistic tasks and 629 security test cases for agents that consume untrusted tool data. CaMeL demonstrates a structural response: trusted control flow is separated from untrusted data flow, and capabilities enforce policy when tools are called. CaMeL reports 77% task success with provable security versus 84% for its undefended system. Security has a utility cost, but the boundary is enforceable outside the model.

Defense layers

Layer	Mechanism	Implementation
Network isolation	Agent cannot reach external endpoints except an explicit allowlist	GitHub Agentic Workflows: Squid proxy on allowlist; Northflank: egress firewall
Filesystem isolation	Agent writes to a temporary workspace, not the host filesystem	Kubernetes agent-sandbox, E2B microVM
Identity scoping	Every tool call carries a per-session credential with minimum required permissions	Strata Maverics, Microsoft Entra Agent ID
Output validation	Agent-generated content passes through a threat detection step before reaching any production surface	GitHub Agentic Workflows: Safe Outputs pattern (Semgrep + TruffleHog + LlamaGuard)
Read-only execution context	Agent reads the codebase but cannot write directly; writes go through a PR/review gate	GitHub Agentic Workflows default posture

Why human review alone fails

Anthropic’s internal data (shared responsibility model documentation, April 2026): 93% of agent permission requests in production are approved without adequate review. This is not a criticism of the humans involved; it is a structural consequence of volume and cognitive load. When a human approves 50 permission dialogs per hour, individual approval becomes a formality.

The defense that scales: structural isolation (makes certain actions impossible) and second-agent validation (creator-verifier pattern, Section 8). Human review remains appropriate for high-stakes, low-frequency decisions, not for routine agent actions.

4. CI/CD agentic patterns

Three platforms have productized agents as a CI/CD primitive. The choice between them is an architectural decision, not a feature comparison.

Test selection as an agent primitive

Before any of the three platforms below, one layer decides what those agents actually run: deterministic test selection, a service that picks which tests run on a given change based on past results and package relevance, instead of running the full suite on every PR. This is not specific to any platform; it is infrastructure the three CI/CD agent patterns below sit on top of.

It matters specifically for agents, not just for CI cost. A human reviewer can look at a failing test and judge, from experience, whether it applies to their change. An agent needs that judgment made for it: a specific, current set of relevant tests is what lets it self-verify and iterate instead of drowning in irrelevant failures (Agentic coding is straining CI, Anthropic engineering, September 2026; see full evaluation).

Anthropic’s own service illustrates the failure mode to design around. Its v0 split a listener, which records test results from every CI run, from a selector, which reads that history to decide what runs on new PRs. Both ran as a single process, because a running per-test history needs a single writer, and that single-writer constraint is what blocked horizontal sharding. Under concurrent CI load the listener fell behind the PR queue: a bad merged change failed a test for everyone, a flaking dependency turned into flaky reds that blocked merges, and a newly fixed test simply didn’t run until the listener caught up. Three sequential patches, a bigger machine, then sharding, then daily restarts, each bought less time than the last before the team rewrote the service around a stateless in-memory journal that any worker can append to, making it horizontally scalable at the cost of running more expensive infrastructure than the original singleton.

The limits: no product is named in the source, no latency SLO is published, and the distributed design is explicitly more expensive to run in exchange for being easier to scale and to profile. Treat this as a pattern to plan for, not a specific tool to adopt.

GitHub agentic workflows

The central concept: gh aw compile takes an agent workflow definition in Markdown and produces a .lock.yml, a hardened GitHub Actions file that executes the workflow with enforced isolation. The compilation step is where security properties are baked in, not added later.

Execution model: agent runs in a read-only context. It can analyze and generate. Any write (commit, comment, deployment) goes through “Safe Outputs”, a separate job that validates the generated artifact before it touches production surfaces. Safe Outputs runs Semgrep (SAST), TruffleHog (secrets detection), and LlamaGuard (harmful content) on agent-generated diffs before they are applied.

Code review integration: GitHub Copilot Code Review has processed 60M+ code reviews as of 2026. On Code Review Bench (Martian, March 2026, 200,000+ open-source PRs, 17 tools evaluated), Augment Code leads at 62.8% recall, Copilot at 53.3%. Graphite leads precision at 75% but recall at 8.8% (high precision means few false positives, low recall means many real bugs missed). No tool dominates both metrics.

c-CRAB benchmark: Claude Code reaches 32.1% on tests derived from selected human review comments. A separate coding agent applies the comments before scoring. Of 234 tests, 192 are structural and 42 behavioral. These results are neither a production ceiling nor general defect recall. The repair agent, test construction and selected concerns constrain the interpretation.

Best for: organizations that want GitHub-native audit trails and a straightforward integration with existing Actions pipelines.

AWS Bedrock AgentCore

A managed runtime for production agents: versioning of agent definitions, Memory for cross-session state persistence, native Observability (OTel-compatible), and continuous evaluation on 1-2% of live traffic. The eval-on-traffic feature catches silent degradation: a model upgrade that regresses quality without triggering any explicit alarm.

Best for: organizations already invested in the AWS ecosystem that need managed state persistence and want continuous quality monitoring without building their own eval infrastructure.

GitLab Duo

Fix CI/CD Pipeline reached General Availability in GitLab 18.8. When a pipeline fails, Duo reads up to 150 KiB of logs, diagnoses the root cause, and proposes a fix as a Merge Request. CI Expert Agent is in beta as of 18.11 for broader pipeline assistance.

Key constraint: 150 KiB log limit. Pipelines with verbose output beyond this threshold get truncated context, which degrades diagnosis quality before routing failures through this path.

Best for: GitLab-centric organizations that want agent-assisted CI diagnosis without adopting a separate platform.

5. Digital twin testing

Agents cannot be tested safely in production on the first pass. The standard practice for testing non-AI software is staging environments. For agents that call external services (Slack, Jira, Okta, Google Drive), staging means either burning real API quota or using behavioral mocks that simulate the service accurately enough to surface integration bugs.

The behavioral mock distinction: a static mock returns a fixed response. A behavioral mock maintains internal state that evolves logically through sequences of interactions, replicates rate limiting, delayed state propagation, and conditional dependencies. An agent that retries after a 429 response will behave differently against a mock that accurately replicates Slack’s rate limit window versus one that just returns 200 for everything.

Current coverage by service

Service	Best available mock	Coverage
Slack	Slack-Mock (github.com/Skellington-Closet/slack-mock)	7 interaction channels: Web API, RTM, Events API, Slash Commands, Webhooks, Interactive Buttons, message delivery. State management included. Most complete.
Google Drive	Mockoon pre-configured sample	REST API surface. Limited behavioral state.
Okta	Community patterns, DevForum	Authentication flows and identity lifecycle. No official mock. Custom build required.
Jira	Atlassian-recommended staging environments	Separate app keys for isolation. Not a behavioral mock.
Generic HTTP	WireMock (stateful), Beeceptor (AI-powered, multi-protocol)	No behavioral state for specific services, but configurable for arbitrary HTTP.

Materialize (“always-current digital twins”) takes a different approach: real-time sync with operational systems, with logical isolation. Closer to a managed staging environment than a mock. Useful when agents need authentic data distributions rather than plausible-but-fake test data.

LangWatch Scenario SDK (langwatch.ai/scenario) is the only attempt at systematic behavioral agent testing without requiring a real running service: it simulates multi-turn conversations against an agent-user that generates realistic inputs, while an agent-judge evaluates whether the system agent met its success criteria.

6. Observability stack

The open-source baseline that works in production, documented by independent organizations:

OpenLLMetry (Traceloop)       ← instrumentation layer (Python + TypeScript)
      +
OpenInference (Arize)         ← semantic schema for LLM/agent attributes
      ↓
Langfuse or Arize Phoenix      ← tracing backend + eval storage
      +
DeepEval or LangWatch Scenario ← quality evaluation

For enterprise with governance requirements, add Strata Maverics or Entra Agent ID in the identity layer.

OpenTelemetry GenAI conventions (August 2026 status)

The canonical OpenTelemetry GenAI registry defines shared fields for conversations, system instructions, tool definitions, tool-call IDs, arguments, results, workflow names, token usage, and operations such as execute_tool, invoke_agent, and invoke_workflow. The GenAI conventions are still evolving and several definitions have moved to the dedicated semantic-conventions repository, so pin the version used by production instrumentation.

Do not record full content by default. System instructions, retrieval queries, tool arguments, and tool results may contain secrets or personal data. Prefer identifiers, sizes, hashes, allowlisted fields, and explicit redaction. Enable payload capture only for a bounded diagnostic purpose with retention and access controls.

What to instrument first

Every LLM call: latency, input tokens, output tokens, model name, finish reason.
Every tool call: tool name, arguments (redacted if sensitive), result success/failure, latency.
Session-level: total tokens per session, session duration, task completion (binary).
Eval scores: task completion rate, tool correctness rate (correct tools used / total tools used).

Operational measures beyond telemetry

Token and latency traces explain resource use. They do not establish that the agent produced acceptable work. Add outcome and control-flow measures that connect a run to its reviewed result.

Measure	Why it matters	Source prompt
Turns to accepted completion	Reveals friction hidden by a final pass/fail score	Patrick Debois discusses turn count as an agent-enablement signal
Manual takeovers and human review comments	Measures where the harness still depends on rescue	The AI Native DevCon interview proposes tracking takeovers and pull-request comments
Requirement coverage	Prevents a broad success score from hiding one missed contract	Executable-spec verifiers can report one verdict per requirement
Evidence completeness	Requires commands, outputs, tests, screenshots, or traces for claims of completion	Simon Willison demonstrates a Markdown proof artifact built from commands and outputs
Latency distribution	Captures slow tails that a mean or median hides	Amit Kushwaha argues for distribution-aware agent benchmarks
Controlled recovery	Tests whether state survives an interruption without duplicated or lost work	Interrupt the run at a defined point, then inspect its resumed trajectory
Repeated-run reliability	Separates a task solved once from behavior that succeeds consistently	Towards a Science of AI Agent Reliability defines consistency, robustness, predictability, and safety as separate dimensions
Harness-model pair	Prevents a model name from absorbing harness-specific cost and failure behavior	The Scaffold Effect reports harness-specific cost and failure fingerprints under fixed models

Report denominators and distributions. “Eight accepted tasks out of ten” is inspectable. “The agent usually succeeds” is not. For latency and cost, include at least the median and a tail percentile when the sample supports it. For small pilots, publish every result instead of implying statistical precision.

Datadog, Honeycomb, New Relic, and MLflow all support OTel GenAI conventions. Arize Phoenix processes 1 trillion spans per month across DoorDash, Instacart, Reddit, Uber, and Booking.com, making it the most documented production scale for an open-source option in this space.

LLM-as-judge limitations

JudgeBiasBench (arXiv 2604.23178, Hongli Zhou et al., April 2026) measured more than 50% error rates for frontier models on advanced bias detection tests. Style bias is the dominant pattern: scores of 0.76-0.92, versus position bias below 0.04. The practical consequence: an LLM judge approves stylistically polished but incorrect outputs significantly more often than it should. True negative rates for identifying invalid outputs typically sit below 25%.

No commercial platform (Arize, LangWatch, Openlayer, Langfuse as of May 2026) publicly documents how it neutralizes style bias in its evaluators.

The working solution: use LLM-as-judge for qualitative dimensions that cannot be evaluated deterministically (tone, explanation quality, context sensitivity). Use code-based checks for anything that can be evaluated mechanically (tool selection correctness, structured output schema compliance, regression tests on known examples). Do not use LLM-as-judge alone as the quality gate for production traffic.

7. Test distribution and component-stacking anti-patterns

An empirical study across 39 open-source agent frameworks and 439 agentic applications (arXiv 2509.19185) found that more than 70% of testing effort in agentic systems targets the deterministic components (tools, APIs, workflow logic), while less than 5% targets the Plan Body, the LLM reasoning core. Adoption of dedicated LLM evaluation tools (DeepEval) was below 1% despite those tools’ high marketing visibility.

This is structurally backwards. The deterministic components are the most testable with standard unit tests and will fail loudly when broken. The LLM reasoning core is where the non-obvious, non-deterministic failures live: the ones that produce plausible-looking but wrong outputs, miss edge cases in tool selection, or hallucinate capability claims.

Recommended rebalancing: give the Plan Body its own behavioral and repeated-run tests instead of assigning an arbitrary percentage target. The pass^k pattern addresses non-determinism: run a critical test several times and report how many executions pass. Promptfoo’s --repeat flag implements this. LangWatch Scenario SDK does it at the multi-turn simulation level.

For deterministic components: standard unit tests, schema validation, explicit tool call checks. For LLM reasoning: behavioral simulation (LangWatch Scenario), LLM-as-judge with bias awareness, regression suites on known-good examples from production.

Tests are necessary, but a green suite is not a complete proof bundle. Simon Willison’s 2026 engineering-practices talk pairs tests with captured commands and outputs, then identifies sandboxing as the main execution boundary. For an agentic coding task, retain the requirement map, relevant commands, outputs, test results, review verdict, and any visual or runtime evidence needed to reproduce acceptance.

Do not assume that stacking planning, tools, memory, reflection, and retrieval is monotonic. Cross-Component Interference tested all 32 subsets of five components on HotpotQA and GSM8K. A single-tool configuration exceeded the all-in system by 32% on HotpotQA; a three-component subset exceeded it by 79% on GSM8K. This is a preprint over two benchmarks, but its design supports a strong engineering rule: start with the smallest sufficient harness and require each added component to pass a paired ablation and regression test.

Test review constraints separately

SWE-Gate separates functional tests from constraints derived from code reviews. With those constraints supplied, 221 of 644 functionally passing model-instance repairs still violated constraint tests. The denominator spans four backends over 303 synthesized Python repair instances; it is not 644 independent production PRs. The replication repository is available, but was not run for this guide.

For a local repair, test the reported bug and relevant compatibility, error-handling or resource constraints separately. For example, a parser fix can pass its new input case while changing the exception callers rely on.

Test the control and its permitted path

Marmelab’s September 2026 survey recommends testing the harness, observing actual use and retiring obsolete components. Its repository inventory is discovery evidence, not a controlled estimate of effectiveness.

For each preventive control, retain the incident it addresses, an input that must be blocked, a valid nearby input that must pass, and the expected result when its adapter cannot interpret the action. Test the policy core, the host adapter and the observed client behavior separately. A deny-all implementation can pass a negative-only suite; a hook invocation log can coexist with an ignored response.

Give the control an owner and a reconsideration condition. Low usage can indicate a discovery failure or a rare protected action. Before removal, replay the motivating incident and permitted alternatives, then assess task outcomes on held-out cases. A static inventory or a successful installation does not establish that the control works in a running session.

Lean countermeasure loop for a repository harness

Treat a recurring defect as a reason to inspect the work station, not as an automatic request for another skill or reviewer. Toyota’s jidoka and kaizen supply the analogy: detect an abnormality close to its source, stop affected work, correct its cause, and check for recurrence. A software team must define its own stop and acceptance policy. The following is a test design, not a measured improvement.

Observed failure	Candidate control at the point of work	Proof before adoption
A cited source line exists, but an agent misreads its behavior	Keep a mechanical citation check, then add independent semantic review or a characterization test for that behavior	The known false claim is rejected, a nearby true claim passes, and the test runs against the relevant revision
An editor changes a legacy file’s encoding or line endings	Route edits through a format-preserving path and check the resulting bytes before commit	The unsafe edit is caught even through an alternate write path; a valid edit retains its original format
Agents submit changes faster than people can verify them	Admit a new change only when the named verifier and required checks have capacity	Queue age and accepted output improve or stay within the owner’s limits without more escaped defects or hidden work

For each candidate, retain the failure example, a permitted neighboring case, the exact host and repository revision, observed execution, false blocks, recovery effort and owner. Compare accepted changes of the same class over comparable windows. A source-level test, an installed hook, a green CI run and a released user outcome are different observations. Use the countermeasure worksheet for one failure; the admission worksheet handles the shared queue; context engineering handles information loaded into one task. If the control adds more waiting or maintenance than the failure it prevents, revise or retire it after the owner reviews the evidence.

Structural scores and passing suites need behavioral evidence

The Walkinglabs evaluation records a synthetic repository that scored 100/100 structurally while verification failed, and a simulated review accepting not approved. The course is useful teaching material; these cases do not justify treating its scaffold as a production control.

The AutoHarness evaluation records 410 passing supplied tests alongside targeted counterexamples involving skill ownership, fabricated evidence, lost updates and mixed file generations. A suite can pass while a stated property lacks coverage. Use these cases to design failure tests for your own boundary, not to attribute those defects to another project.

Keep viewed, invoked, executed successfully and useful as separate observations. Preserve the producer, relevant revision, scope, terminal state and acceptance policy for each result. The runnable control exercise illustrates receipt validation, durable budgets and effect recovery on a simulated service; it does not measure skill activation or usefulness.

Each observation needs its own evidence; none implies the next. Open the image for full size. Acceptance also requires policy authority. French version and sources.

8. Creator-verifier pattern

The creator-verifier pattern assigns production and evaluation to separate steps or agents. It is a useful design candidate, not a guaranteed accuracy multiplier.

Natural-Language Agent Harnesses, RQ2, reports lower aggregate scores after adding its verifier on sampled coding and computer-use tasks, alongside some additional rescued cases. In a recorded SymPy case, local verifier acceptance disagreed with the benchmark. This bounded counterexample motivates evaluating final acceptance after review, rather than counting successful intermediate verdicts.

The evidence is mixed. Self-Refine reports gains from iterative same-model feedback across seven tasks. Multiagent Debate reports improvements on selected reasoning and factuality tasks. Conversely, Large Language Models Cannot Self-Correct Reasoning Yet finds that intrinsic self-correction without external feedback can fail or degrade answers. Scalable oversight experiments find task-dependent rather than universal gains, and LLM-as-a-judge bias research documents systematic evaluator biases. Anthropic’s evaluator-optimizer guidance therefore recommends the pattern when evaluation criteria are clear and iterative refinement produces measurable value.

A fresh context is only one dimension of independence. A verifier can still share the creator’s blind spots when both depend on the same model family, specification, tools, retrieval corpus, reward signal, or missing runtime evidence. Assess at least five dimensions:

Independence dimension	Failure when shared	Stronger design
Context	The reviewer inherits the creator’s framing and assumptions	Give the reviewer requirements, artifact, and evidence without the creator’s chain of reasoning
Model or provider	Correlated blind spots and self-preference	Stratify results by model and test provider diversity where risk justifies the cost
Evidence and tools	Both agents inspect the same incomplete signals	Add deterministic tests, runtime traces, external sources, or a different inspection tool
Role and incentives	The reviewer optimizes for agreement or style	Require per-requirement pass/fail verdicts and evidence, not a generic quality score
Escalation authority	A probabilistic verdict silently becomes final	Define appeal, timeout, human checkpoint, and fail-closed rules for high-impact decisions

Measure rescued failures, false accepts, false rejects, regressions, latency, and cost. Counting reviewer calls or using a different role name does not establish independence.

Read-only permissions constrain authority; they do not demonstrate independent judgment. Preserve those controls while comparing review treatments, and record which evidence each reviewer obtains independently.

Cross-session messaging can carry the handoff between independently launched Claude Code sessions, but the transport does not make the verifier independent. Bind the handoff to a commit SHA, send the original requirements, artifact, evidence, and uncertainty instead of the creator’s rationale, then let the verifier inspect its own current state. Apply the cross-session threat model separately for sender trust, inbound policy, permissions, and cross-machine exposure.

Liza implements this pattern as doer/reviewer pairs backed by a deterministic supervisor. Its state machine and merge authority can reject invalid transitions even when an agent proposes them. That is stronger than a role name in a prompt, but it still needs outcome evaluation: a structurally separate reviewer can share the same model, incomplete specification, or missing evidence as the doer.

The pinned Liza configuration makes the boundary visible. Some high-impact planning stages require a quorum of two and mark provider diversity as preferred, while the coding pair uses a quorum of one. Liza’s architectural issues ledger records unenforced provider diversity at verdict submission and unmeasured reviewer accuracy. Manual inter-phase checkpoints apply when auto_resume is disabled; the pinned profile also documents autonomous progression when enabled. Human governance does not imply a human in every execution cycle. The code enforces workflow legality; it does not prove that a legal reviewer verdict is correct.

Judgment allocation: where responsibility actually sits

A stopping condition answers when a loop ends. It does not by itself establish that the result is good enough. Every production design should record who sets the quality bar, what evidence is admissible, who decides that the evidence is sufficient, and who handles exceptions.

Decision	Default owner	Evidence	Escalate when
Product intent and acceptable risk	Accountable human or product authority	Requirements, policy, risk classification	Intent is ambiguous or impact is high
Task decomposition and routing	Orchestrator or planning agent	Typed task graph, dependencies, budgets	Fan-out is large, irreversible, or crosses domains
Execution	Runtime and repository harness	Tool results, sandbox events, changed artifacts	A permission, policy, or environmental boundary is reached
Mechanical acceptance	Deterministic gate	Tests, schemas, linters, policy checks, reproducible commands	Coverage is incomplete or a check is flaky
Semantic sufficiency	Reviewer agent for bounded work, accountable human for high-impact work	Requirement-level verdicts plus inspected evidence	The verifier is uncertain, conflicted, or outside its evidence boundary
Release or policy exception	Human or external policy authority	Signed approval and auditable rationale	Never silently auto-approve an exception

This is the useful insight from the loop-engineering debate: human judgment does not necessarily disappear when prompts are automated. It moves to loop design, quality criteria, exception policy, and final accountability. Own the Outer Loop presents this as a practitioner design principle. Treat the exact placement as a risk decision, not a universal rule.

Practical implementation: spawn a second agent with the output artifact and the original requirements. Ask whether the output satisfies each requirement. Do not ask “is this good?” Ask “does this satisfy requirement X?” with explicit pass/fail for each.

An executable-spec variant makes that contract more granular:

A planner extracts explicit requirements and acceptance evidence from the task.
One verifier evaluates each requirement independently, in parallel where isolation allows it.
Each verifier inspects the relevant codebase state, not only the produced diff.
Runtime behavior is checked in an ephemeral browser or sandbox when static inspection cannot prove it.
The harness aggregates per-requirement verdicts and preserves the evidence behind each one.

Shachar Azriel presents this architecture in Executable Specs: Building a Verification Layer for Agentic Coding. The talk documents an implementation approach. It does not independently prove the resulting product’s accuracy, so treat the pattern as a design candidate and test it against real repository tasks.

This does not eliminate hallucination; it catches the subset of hallucinations that are inconsistent with the stated requirements. For catching hallucinations that are internally consistent but factually wrong, you need domain-specific test cases.

Human oversight needs enough information and capacity to challenge or resume the work. A compact decision dossier should retain sources, unknowns and disagreements; approval speed alone cannot establish its quality. Learning with AI owns the proposed human exercise and its evidence limits.

9. Reference architecture

User instruction
      ↓
┌─────────────────────────────────────────────────────────────────┐
│                       HARNESS                                    │
│                                                                  │
│  ┌─────────────┐      ┌──────────────┐      ┌───────────────┐   │
│  │   Context   │      │  While-Loop  │      │  Permission   │   │
│  │  Management │◄────►│    Engine    │◄────►│  Enforcement  │   │
│  └─────────────┘      └──────┬───────┘      └───────────────┘   │
│                              │                                    │
│  ┌─────────────┐      ┌──────▼───────┐      ┌───────────────┐   │
│  │   Session   │      │   Dynamic    │      │   Lifecycle   │   │
│  │ Persistence │◄────►│   Prompt     │◄────►│    Hooks      │   │
│  └─────────────┘      │  Assembly   │      └───────────────┘   │
│                        └──────┬───────┘                          │
│  ┌─────────────┐             │              ┌───────────────┐   │
│  │    Tool     │      ┌──────▼───────┐      │  Sub-Agent    │   │
│  │  Registry   │◄────►│  LLM Call    │◄────►│  Management   │   │
│  └─────────────┘      └──────────────┘      └───────────────┘   │
│                                                                  │
│  ┌─────────────┐                                                  │
│  │  Built-in   │                                                  │
│  │   Skills    │                                                  │
│  └─────────────┘                                                  │
└─────────────────────────────────────────────────────────────────┘
      ↓
┌──────────────┐      ┌──────────────┐      ┌──────────────────┐
│   Sandbox    │      │   Identity   │      │  Observability   │
│  (E2B/k8s/  │      │  Gateway     │      │  (OTel + eval)   │
│  Northflank) │      │(Strata/Entra)│      │                  │
└──────────────┘      └──────────────┘      └──────────────────┘

10. Practitioner video evidence

This ledger connects the architecture to dated practitioner and product-creator testimony. Every quoted sentence below was checked against the local WebVTT transcript and links to the corresponding YouTube timestamp. Capitalization and punctuation are normalized; wording is preserved. The transcript proves what was said at that point in the video. It does not prove that a product still behaves that way, that a reported outcome generalizes, or that an interview claim was independently measured.

Use three evidence levels throughout this page:

Official current evidence for product behavior, limits, licences, and availability.
Creator interviews for dated design rationale and historical product decisions.
Practitioner talks for methods, failure reports, and hypotheses that still require local validation.

Source	Timestamped verbatim	What it supports	Evidence boundary
Harness Engineering: The New Discipline of Agentic Dev, AI Native DevCon interview, 2026-07-08	”Never tell the agent more than once how to do a thing before it gets codified somewhere in the inner or outer loop.”	Inner, outer, and meta feedback loops	Practitioner model, not a formal standard; speaker name is not established by the local metadata
Patrick Debois, The Rise of Agent Enablement, 2026-07-08	”We’re not building the thing, we’re building the thing that builds the thing.”	Agent enablement as system design	Organizational framing, not a benchmark result
Ryan Lopopolo, Harness Engineering: How to Build Software When Humans Steer and Agents Execute, 2026-06-19	”I never want to give the same review feedback twice.”	Codifying recurring feedback into the harness	Practitioner workflow and definition
Marc Sloan, Harness Engineering Beyond Code, 2026-06-20	”Product and design context is important to agents, but it lives outside the code base.”	External product and design context	Practitioner talk; integration trade-offs require local testing
Boris Cherny, Building Claude Code, 2026-03-04	”If you’re not sure, just ask the human and then they can decide.”	Conservative permission fallback	Creator interview; current behavior belongs in official docs and release notes
Dax Raad, Building OpenCode, 2026-05-27	”You need some kind of control plane to like set up all the providers, permissions, budget controls, rate limits.”	Separation between runtime and organizational control plane	Creator interview and dated product claim, not proof of current availability
Simon Willison, Engineering Practices That Make Coding Agents Work, 2026-03-19	”I think tests are no longer even remotely optional.”	Tests, proof artifacts, and sandboxing	Practitioner recommendation
Shachar Azriel, Executable Specs: Building a Verification Layer for Agentic Coding, 2026-06-15	”The code is how we ground that agent to reality.”	Requirement-level verification against actual code	Architecture described by the speaker; outcomes are not independently verified here
Amit Kushwaha, Benchmarking the Agent Era, 2026-07-10	”You can’t talk about mean and medians. Now you have to talk about in terms of distributions.”	Multi-turn, multi-tool, tail-aware measurement	Benchmark-methodology talk; specific performance claims require the underlying study
Dex Horthy, Context Engineering, 2026-07-15	”If you want to do loops engineering, you should build one loop at a time and keep them small and contained.”	Bounded loops before lights-off automation	Practitioner interview and retrospective, not a controlled study
Pavan Belagatti, What Is a Software Factory?, 2026	”Agents do the work, humans provide the gates.”	Separation between the execution loop and the governance loop	Practitioner short and Port-oriented framing; no comparative reliability or productivity measurement

Three additional talks provide practical examples, paraphrased here from checked passages:

Will Bond and Ameya Ketkar, Uber, Building uReview, 07:03, 2026-08-28: team-specific criteria and single-file/multi-file reviewers within a bounded review window. See role separation.
Ishaan Sehgal, Omnara, The Log Is The Agent, 04:57, 2026-06-25: the raw record and its compacted view serve different purposes. See history and compaction.
Michael Hablich, Google, Building Agent Interfaces, 21:56, 2026-06-05: recovery playbooks and tokens per successful outcome. See actionable tool failures.

These additions and three research papers are assessed in the corpus supplement.

The source list is selective. It includes videos that add a distinct mechanism, metric, or failure boundary to this page. Inclusion is not an endorsement of every claim in a talk.

11. Harness optimizers and meta-harnesses

A runtime harness improves one agent run. A harness optimizer improves the code and configuration that govern future runs. The optimizer can edit prompts, context policies, tools, middleware, memory, control flow, verification, or routing, then use an external evaluator to decide which candidate survives.

This section uses meta-harness in that research sense. Databricks and Omnigent use the same term for a unified developer surface that dispatches a task to one of several existing coding harnesses. That dispatcher can reduce switching friction and preserve tool-policy consistency, but it belongs to the orchestrator or control-plane layer unless it also searches, changes, and evaluates the target harness. Source and terminology review: Databricks cost-management resource evaluation.

Work	Optimization surface	Reported evidence	Boundary
Automated Design of Agentic Systems, ICLR 2025	Code-defined agent systems	Meta Agent Search discovers prompts, tools, and workflows that transfer across domains and models	System-level search; not every result isolates a fixed-model harness effect
AFlow, ICLR 2025	Code-represented workflows	5.7% average improvement across six benchmarks	Heterogeneous model-plus-workflow baselines
ACE, ICLR 2026	Context and memory playbooks	+10.6% on agent benchmarks and +8.6% on finance tasks	Context optimization, not whole-harness search
GEPA, ICLR 2026 Oral	Prompts from reflected trajectories	6% average gain over GRPO across six tasks, with up to 35 times fewer rollouts	Prompt optimizer; model control varies by experiment
Meta-Harness, 2026 preprint	End-to-end harness code	+7.7 points with four times fewer context tokens; +4.7 points across five held-out models on 200 math problems	TerminalBench search and final evaluation reuse the same 89 tasks; total search compute is not reduced to one comparable cost figure
Agentic Harness Engineering, 2026 preprint	Prompt, tools, middleware, skills, subagents, and memory	Terminal-Bench 2 pass@1 rises from 69.7% to 77.0% over ten iterations	Coding-focused preprint; external replication remains limited
HarnessOpt-Bench, 2026 preprint	The optimizer itself	Five optimizer models, four downstream tasks, 111 scored runs	Benchmark protocol, not a production optimizer

The evidence is promising but young. Do not promote a candidate because its development score improved. A credible optimization run needs:

Frozen invariants: target model, tool permissions, task contract, grader, and resource limits remain fixed unless the experiment declares otherwise.
Separated data: detailed development traces, bounded validation feedback, and a hidden test partition that the optimizer cannot inspect.
Explicit budgets: evaluation calls, task attempts, tokens, wall time, and human interventions are metered.
Versioned candidates: every change has a diff, rationale, predicted benefit, regression risk, score, and rollback path.
Multi-metric promotion: success, accepted-task cost, latency distribution, repeated-run reliability, and failure severity all participate in the decision.
External evaluation: the optimizer cannot alter the evaluator, reveal hidden cases, or award its own final score.

HarnessOpt-Bench is the clearest current protocol: a trusted execution environment hides the test partition, enforces evaluation budgets, meters resources, and preserves candidate versions. Establishing Best Practices for Building Rigorous Agentic Benchmarks explains why those controls matter: task and reward defects can distort reported agent performance by up to 100% in relative terms, and its checklist reduced CVE-Bench overestimation by 33%.

The practical conclusion is not “self-improving agents solve harness engineering.” It is that harness engineering can become an auditable experiment when the search space, evidence, budget, and promotion boundary are explicit.