Same PASS, Different Paths: Evaluating Agent Traces at the Span Level with Jev

PASS rate misses execution-path quality

A common agent evaluation keeps the agent and task set fixed, then runs the same cases with different foundation models. The visible outcomes often look alike. Tests pass, and the final patches may even match byte for byte.

The traces can still differ substantially. A session often contains dozens or hundreds of spans that record model requests, tool calls, file reads, edits, and validation steps. PASS rate hides when an agent begins to drift, repeats an earlier action, or spends extra calls recovering from a poor decision.

Our experiments compare both models and harnesses. A harness is the execution framework around a model. It supplies the system prompt and tools, manages context, and controls the agent loop. Changing the harness can change the available information and the resulting execution path even when the underlying model stays fixed.

Three agents, one repair task

We selected three terminal-based coding agents with different design choices.

All three agents received the same task: fix serde-rs/json issue #979. The bug occurs during enum-variant deserialization.

EnumAccess.variant

{[true]: null}

The task prompt identified the likely root-cause location and the direction of the fix. It required the agent to inspect the next non-whitespace byte before choosing a branch, update the implementation, add a regression test, and run the full test suite. Validation ran offline, with dependencies prepared before each agent started. This reduced variation from network access and dependency resolution.

Span-level evaluation is still too expensive

Databend already handles the trace data. Large volumes of semi-structured JSON land in the

VARIANT

traces

trace_id

SQL can report which tool a span called and how many tokens it consumed. It cannot directly determine whether the call advanced the task or repeated earlier work. Those questions depend on context.

LLM-as-Judge is a common way to evaluate that context. A general-purpose model receives the relevant history, writes a critique, and returns JSON. Assuming about 2,000 input tokens and 300 output tokens per span, models in the GPT-4.1 mini, Claude Haiku, and Gemini Flash price range cost roughly $0.001 to $0.0035 per span, with latency around one to three seconds. Prices change, so production estimates should use the current model versions and rates.

At one million spans per day, evaluation would cost between $1,000 and $3,500. That often forces teams to sample a small fraction of their traces, and the resulting judgments tend to live outside the system that stores the original data.

The lakehouse can already query traces quickly. Evals still need semantic judgment that is fast and inexpensive enough to run at scale.

Jev turns open-ended reviews into bounded decisions

Daniel Kahneman's Thinking, Fast and Slow distinguishes between fast, intuitive System 1 thinking and slower, deliberative System 2 thinking. A coding model that plans and edits code resembles the latter. Deciding whether a tool call is relevant or whether it repeats the previous step is closer to a bounded classification task.

TypeSafe borrowed this distinction for its System One Models. Its first model, Jev, is named after economist William Stanley Jevons.

Jev returns answers to predefined structured questions. Choice selects from a set of options. Score rates an input against defined levels. Noul returns the probability that a proposition is true. Choice and Score also return confidence values, and the schema constrains the result type. Multiple questions can be attached to the same input and submitted in one request.

At launch, TypeSafe listed Jev's input price at $0.042 per million tokens, with free output. Published end-to-end response times ranged from 70 to 500 milliseconds. Jev was still in early access, so quotas and availability depended on the account.

For span evaluation, this interface is closer to a structured questionnaire than a written expert review. The downstream analysis needs scores and decisions, not a paragraph for every tool call. In this workload, that difference cuts the evaluation cost by more than an order of magnitude.

How a span is evaluated

The pipeline first separates deterministic checks from semantic ones. Code can determine whether a file belongs to the task scope, whether validation happened after the final edit, and whether the current tool call duplicates the previous call.

Jev handles the parts that require context. It assigns a relevance score from 0 to 4 and judges whether the span adds progress or touches the known root cause. Policy violations and missing test evidence at the end of a run can be expressed as separate questions.

The distinction around root cause matters. Jev can judge whether a span touches a predefined root cause. Discovering a new root cause through open-ended analysis still belongs to a general-purpose model or a human reviewer.

If a Jev answer conflicts with a deterministic fact, the pipeline records

unsure

Store the judgments in Databend and keep using SQL

Jev runs on top of the

traces

span_judgments

Once the results are stored, off-task rate, spin rate, and model comparisons become SQL aggregations. The existing trace store also serves as the review system, with every judgment linked to its session and original span.

Measured cost across 1,171 spans

trace.evot.ai contains 50 evaluation sessions and 1,171 spans across several model and harness combinations. This article uses two slices of that data, one comparing models and another comparing harnesses. The complete runs remain available on the evaluation site.

Jev cost about $0.00008 per span in these runs. The median latency for a batched request was about 410 milliseconds. The fast LLM-as-Judge control is estimated at $0.001 to $0.0035 per span, with one to three seconds of latency for an individual span.

The latency figures use different measurement units. The 410-millisecond result is the median for a Jev batch, while the one-to-three-second range estimates a single span evaluated by a general-purpose judge. They show the difference in scale, but a strict end-to-end comparison would need the same batch size and concurrency settings.

At up to one million spans per day, full coverage costs tens of dollars. At larger scales, SQL can select a stratified sample that prioritizes failed and unusually long sessions. Once requests are highly parallel, API quota usually becomes the limiting factor.

The task passes, but the paths still diverge

An earlier experiment compared different models on the same

serde_json

These differences matter when selecting a model or changing a prompt because they expose work that PASS rate cannot measure.

run-199 provides a second view. It holds the

glm-5.3-flash

All three harnesses completed the task. Pi made 41 model requests, 15 more than Evot and 13 more than DSH, and took the longest. The span-level evaluation also identified two Pi steps that made no progress. The final result was still PASS, so this difference only appears when the trace is evaluated.

Once the judgments are in

span_judgments

relevance

necessary

SELECT t.model, t.harness,
       avg(j.relevance) AS avg_relevance,
       countIf(j.relevance < 1.5) / count() AS off_task_rate,
       countIf(j.necessary < 0.5) / count() AS spin_rate
FROM span_judgments j JOIN traces t USING (session_id)
WHERE j.kind = 'judged'
GROUP BY 1, 2 
ORDER BY off_task_rate DESC;

Where this method fits

Jev works best when the objective is clear and the possible answers are defined in advance. Open-ended root-cause analysis and complex reasoning still require a general-purpose model or a human. Question wording and scoring criteria also affect the resulting metrics.

Facts that code can determine should remain in code. Jev covers semantic decisions that are hard to express as rules and only need bounded outputs. Low-confidence and

unsure

This split makes it affordable to evaluate more spans and keeps each judgment next to the source trace. Judge errors remain, so teams still need sampled human review.

The measurements in this article cover cost and latency. Agreement between Jev, human annotators, and a general-purpose judge remains unmeasured. The data establishes affordability, while comparative accuracy remains an open question. A production deployment should calibrate questions and thresholds on its own labeled data and account for early-access quotas and availability.

Making span-level evaluation affordable

Databend stores and analyzes the traces. Deterministic code computes the facts, and Jev handles bounded semantic judgments. Returning the results to the same lakehouse keeps model comparison, harness comparison, and stratified sampling in SQL. The data can also support later training-data selection.

PASS tells a team whether the task finished successfully; span-level evaluation shows how it got there and where the cost accumulated.

Related resources

• run-199 comparison

• serde-rs/json issue #979

• Evot on GitHub