What PASS Misses: Lessons from Six Coding Agent Traces

When GPT-6-Astra was released, benchmark scores and coding results appeared everywhere. I had a narrower question: how would it behave inside a real coding agent workflow, compared with Claude-Fable-5.1, the model I use most often? I wanted to compare reliability, cost, and day-to-day engineering behavior on the same task.

Leaderboards could not answer that, so I ran the comparison myself.

The experiment used three coding agents with different designs: Evot, Pi Coding Agent, and DeepSeek Harness. Each model ran once on each harness and attempted the same task, serde-rs/json issue #979. After all six sessions finished, I reviewed every model request, tool call, and validation step.

The three agents provide useful points of comparison:

• Evot is an open source terminal coding agent. It keeps intervention outside the model to a minimum, uses roughly 1K tokens of system context, and exposes four core tools:

  read
  

  ,

  bash
  

  ,

  edit
  

  , and

  write
  

  . Its design pays particular attention to context use and tool-call volume in long-running tasks.

• Pi Coding Agent is another lightweight terminal harness with a similar default toolset. Developers can assemble workflows with TypeScript extensions, skills, prompt templates, and packages. Pi does not prescribe a single optimal workflow.

• DeepSeek Harness, or DSH, follows an “Everything is a plugin” design. Model adapters, tools, sessions, sandboxes, storage, and the agent loop are all replaceable. This makes DSH a useful example of how a plugin-based, event-driven harness affects context management.

Two models, three harnesses, and one bug produced the same patch through noticeably different paths.

Figure 1.

gpt-6-astra

metrics across the three agents.

Figure 2.

claude-fable-5.1

metrics across the three agents.

The patches matched, but the costs did not

All six runs passed the complete test suite. Their core patches were byte-for-byte identical.

Claude used 44% more model requests, produced 9.6 times as many output tokens, and took 2.3 times as long.

These numbers need context. The task prompt already identified the likely root cause and pointed toward the repair: peek at the next non-whitespace byte before branching. With the implementation direction stated so explicitly, matching patches were expected. Once correctness and patch content were tied, the execution process became the more useful source of evidence.

The evaluation removed external shortcuts

A model could discover a fix by reading the code and failing test. It could also search the web or recover the official patch from Git history. The evaluation environment closed off the latter routes.

Agent execution had no external network access. At container startup, outbound rules allowed loopback traffic, established connections, and the in-container LLM proxy process. All other outbound traffic was denied. After the rules took effect, the process lost its

NET_ADMIN

NET_RAW

The Git history was pruned as well. The official fix for serde-rs/json issue #979 landed after the commit used in this experiment. In a complete repository,

git log --all

Validation also remained offline. Network access was allowed only during dependency preparation, before the agent started.

These controls restricted each model to the current source tree and the failing regression test.

Once correctness tied, the cost breakdown mattered

The direction was consistent across all three harnesses: Claude made more model requests in every matched pair.

GPT's completion times ranged from 50.2 to 55.4 seconds, a spread of 5.2 seconds. Claude ranged from 94.0 to 135.4 seconds, a spread of 41.4 seconds. For this kind of production workload, scheduling and timeout policies would need substantially more headroom for Claude.

The slowest run also shows why completion time should be separated into model and tool latency. Claude took 135.4 seconds on Pi, yet its model latency there was 91.8 seconds, lower than the 110.1 seconds recorded on DSH. Tool execution consumed the remaining 43.6 seconds, or 32% of the run, most likely while waiting for compilation. A single end-to-end number would have incorrectly assigned that delay to the model.

The largest gap appeared on DSH, where part of the cost came from the framework itself.

Three differences hidden by the aggregate results

Every run failed on rg first

GPT-6-Astra's first command on Evot used

rg

command not found

grep

The same failure occurred in all six sessions, three times for GPT and three times for Claude.

The Evot and Pi system prompts suggest using shell tools and list

ls

rg

find

This was unrelated to offline execution. The base image simply did not include ripgrep. Network restrictions do not determine which local commands are installed.

Neither the diff nor the PASS result recorded the failure. In a larger evaluation batch, however, one guaranteed no-op call per task becomes a recurring cost.

Tool-name friction consumed 25% of Claude's Pi session

Claude's first four turns on Pi are especially revealing.

It first called

Read

Bash

Read

Bash

Edit

rg

grep

Three of the first four turns made no progress. The session contained 12 turns in total, so tool friction consumed 25% of it.

Evot accepts capitalized tool names, and Claude's first

Read

Claude also incurred a smaller overhead on Evot. It read the same file twice in succession, and both tool responses reported that the file was unchanged. Aggregate metrics did not reveal that repetition either.

Sixty percent of Claude's DSH output added no engineering result

Claude made 14 model requests on DSH. The harness used one to generate a session title, and 10 requests handled the repair. The final three occurred after the engineering work was complete, causing the final response to be generated twice.

• Step 11 produced the final response with 4,070 output tokens.

• Step 12 triggered context compaction, produced 1,980 tokens, and took 25.7 seconds. It was the slowest individual model request across all six sessions.

• Step 13 generated the same final response again, using another 4,070 tokens.

The session produced 10,120 output tokens. The duplicated response and context compaction used 6,050 tokens, or 60% of the total, without adding a new engineering result.

An aggregate table might suggest that Claude was simply verbose. The trace gives a more precise attribution. Those 6,050 tokens came from harness-level context management, so they should be separated from model behavior and useful task work.

GPT's DSH session also included a title-generation request, but it did not trigger compaction. Claude may have crossed a context threshold, or its longer output may have grown the context faster. This dataset cannot establish the cause. It does show that compaction overhead varies with model behavior.

A five-point checklist for reviewing agent traces

The observations above came from the same checklist. I now use it whenever I review a coding agent evaluation.

Item 4 is easy to overlook. Every trace contained a tool failure, yet every run passed. These failures behave like warnings in a codebase: they do not block the current build, but they accumulate cost when left unresolved.

Item 5 determines whether a final response can serve as an audit record. I rechecked the validation commands in all six sessions. Each run did execute the full test suite, and every report accurately described the observed result.

GPT-6-Astra converged quickly on the root cause

GPT-6-Astra followed almost the same path on all three harnesses. It found the target function, read two files, edited the code, ran the full test suite, and inspected the diff. It did not reread files or build an additional test matrix.

Its final reports were 77 to 101 tokens long. They identified the modified location, described the behavior for four input categories, and reported the test result. Each report matched the trace.

Against the five-point checklist, GPT reached the root cause directly, read relatively little context, and still ran the complete

cargo test

rg

The evidence supports a narrow conclusion: on this task, GPT completed diagnosis, editing, and validation in fewer model rounds. It took no detours and performed few confirmation steps.

At first, the short path looked somewhat aggressive to me. The patch and test coverage held up. For this repair, GPT found the right location, ran the tests, checked the diff, and stopped.

Claude-Fable-5.1 behaved more like a conservative code reviewer

Claude-Fable-5.1 produced the same implementation as GPT. Across its three traces, it spent more time checking compatibility before making the edit.

On Evot, Claude searched the existing error tests. Its final explanation also addressed why

MapKey

MapKey

The Pi run performed the broadest validation. After the full test suite, Claude created a temporary test file to check invalid array keys and invalid numeric keys. It also tested an empty object, truncated input, a normal string, and whitespace-prefixed input. It removed the temporary file when validation finished.

The context compaction and duplicate response on DSH were primarily framework behavior, so they say little about Claude's engineering style.

Additional checks can be valuable in parsers, security boundaries, and compatibility-sensitive code. If the same kind of change touched serialization logic in a financial system, I would prefer this level of review.

The additional assurance had a visible cost. Claude read more source, wrote longer explanations, and used more model rounds. On Pi, three rounds were lost to tool friction. Fixing that harness compatibility issue should reduce its completion time.

The harness amplifies model behavior

Tool compatibility created the first source of friction. Capitalized tool names worked on Evot and failed on Pi and DSH. On Pi, the difference cost the same model two turns while it discovered the accepted syntax.

Context management affected the end of the task. DSH compacted Claude's session and then generated the response again, adding 6,050 tokens without additional engineering work.

Tool capabilities may also affect how far validation can go. Pi did not prevent Claude from expanding the test scope, although one session is far too little evidence to attribute that behavior to Pi's design.

The experimental limits matter. The two model batches ran on different days, and the three harness versions changed between them. Each model and harness combination ran only once, so the sample cannot describe variance or support strict causal attribution. DSH showed the largest difference and Evot the smallest. Both observations need follow-up experiments.

The same evaluation platform now contains 50 sessions for this task, covering 14 models, 4 harnesses, and 17 runs. Across that larger set, no harness is consistently the cheapest for every model. A single paired comparison cannot capture those interactions.

Seven practices for more useful agent evaluations

These six sessions changed how I design coding agent evaluations.

1. Run the same task, version, and day. Models change, and so do harnesses and dependencies. Comparability degrades quickly when runs are separated in time.

2. Prevent answer leakage at the environment level. Block outbound traffic, remove answer-bearing Git history, and verify the result. Without those controls, an experiment may measure search ability. Benchmark performance may also reflect exposure to similar tasks during training.

3. Run every combination at least three times and report the distribution. A single run cannot establish stability. This experiment has one run per combination, so its conclusions are limited to behavior observed in these six sessions.

4. Verify token-reporting semantics before calculating cost. In this dataset, one model reported all input usage as zero, while another reported zero output for intermediate requests. Cost calculations are unreliable until those reporting differences are reconciled.

5. Separate model latency from tool time. Compilation waits are easily blamed on the model when only end-to-end duration is available.

6. Preserve the complete trace. A final diff contains no record of context compaction, duplicate responses, or tool-name mismatches. The diff shows what changed. The trace reconstructs how the change happened, which is often the more important engineering question.

7. Segment results by task type. A small patch and a cross-module refactor can produce very different model behavior. This experiment covers one small repair, so its findings may not carry over to a large refactor.

Agent trace analysis becomes a data engineering problem

Every conclusion above had to be recovered from logs.

We first grouped the six sessions by model, harness, and version. To find where 1,980 tokens were spent, we drilled from the session summary into the twelfth span. Matching identical output counts across two spans confirmed that the final response had been generated twice. We then matched tool calls across all 50 sessions and found that

rg

The current dataset is small: 50 sessions, 14 models, 4 harnesses, and less than 1 GB in total. A few JSON files and a script are enough. Trace.evot.ai mainly provides a convenient interface for side-by-side inspection.

A production agent generates much more trace data. In this evaluation repository, the median coding session occupies 5.7 MB and the largest occupies 32 MB. Each trace preserves complete model inputs and outputs, tool calls and results, and raw streamed responses. A typical session contains dozens of requests, with the longest reaching 86.

Those examples are patch-oriented tasks that usually finish in a few dozen spans. Refactoring, cross-module debugging, and multi-turn requirement clarification can run much longer. Their traces may contain hundreds or thousands of spans, with an expanding context and a larger storage footprint. The following estimates should therefore be treated as a lower bound.

At 1,000 sessions per day, trace data would grow by roughly 8 GB per day. At 10,000 sessions, it would approach 80 GB per day, or about 30 TB per year. Storing the data is only part of the requirement. The system must also support several query patterns:

• Aggregate success rate and cost by model, agent version, and task type.

• Drill from a session summary into one specific tool call.

• Search across sessions for patterns such as tool-name mismatches and duplicate calls.

• Preserve the full raw context for replay and audit.

This workload combines large table scans, semi-structured JSON analysis, high-cardinality dimensions, wide records, point lookups, and analytical scans. Scripts work well for 50 sessions. At tens of thousands of sessions per day, storage cost, query latency, and maintenance become material concerns. The system then needs direct support for semi-structured data, multidimensional aggregation, and hierarchical drill-down.

Databend Cloud is designed for this kind of workload. Raw events can remain in object storage while SQL queries operate directly on JSON. Aggregation and drill-down do not require a separate preprocessing pipeline. For a dataset of only a few hundred megabytes, files and scripts remain the simpler choice, as they were for this experiment.

Coding agent differences live in the execution path

Correctness was tied on

FixJsonParsingBug

Their working processes still differed. GPT-6-Astra used fewer model requests, produced shorter output, and finished faster. Claude-Fable-5.1 read more source, added edge-case tests, and explained compatibility risks. Its path resembled a cautious code review. Tool compatibility and context management in each harness further changed the cost.

After this experiment, a strong benchmark score makes me look for the corresponding trace. Many operational issues never reach the result table.

PASS made the six runs look identical. Their traces exposed six distinct engineering processes and showed exactly where model behavior and harness design interacted.

Original traces and reproduction details

• GPT-6-Astra comparison: run-205

• Claude-Fable-5.1 comparison: run-202