From Million-Token Agents to Eval-Ready Data

A leading foundation model company known for its long-context processing and advanced reasoning capabilities needed to capture and process complete execution traces from its open-source, trillion-parameter reasoning model built for agentic workloads. With individual tasks spanning thousands of tool calls and millions of context tokens—and online Trace ingestion reaching terabytes per hour—the company built a unified Agent Trace data pipeline on Databend Cloud, connecting raw ingestion, incremental transformation, Eval-ready datasets, analysis, and historical replay.

When a single Agent run becomes a data-engineering workload

A conventional chat session may end after a handful of turns. A complex Agent task can run for hours: the model creates a plan, calls tools, receives results, updates its context, and decides what to do next.

With a 1M-token context window, the execution history becomes a substantial data workload in its own right. One long-running task may issue thousands of tool calls and process millions of context tokens. Model inputs and outputs, tool responses, execution states, latency, token consumption, and intermediate artifacts continuously create new spans that eventually form a large, deeply nested Trace.

For the engineering team, retaining these events is only the starting point. They also need to answer questions such as:

• Why did a task succeed or fail?

• At which step did the Agent deviate from the expected path?

• Did a change to the model, prompt, tool, or agent harness improve the outcome?

That makes Agent Trace data more than an observability record. It must continue into Evals and attribution analysis, where it can support the next round of model and Agent development.

Why Agent Traces need a data pipeline—not only observability

Traditional distributed traces describe service calls with relatively stable schemas. Their analysis often centers on latency, error rates, and call graphs. Agent Traces are less predictable.

Every model or tool release may introduce new JSON fields. A long-running task can produce spans across a much wider time range, while a single execution path may be distributed across a large volume of records. Enforcing a fixed schema at ingestion time would cause upstream changes to propagate into table definitions, consumers, and transformation jobs. Keeping only raw JSON preserves flexibility, but pushes repeated parsing and transformation costs into every downstream query and Eval.

Raw spans also do not arrive as analysis-ready datasets. They must be cleaned, important fields must be extracted, and newly arrived events must be incrementally transformed into structures suited to Evals and root-cause analysis. At the same time, traces may contain user input, tool responses, or other sensitive fields, so each analytical workload should see only the data it needs.

At terabyte-per-hour ingestion rates, these steps cannot be allowed to interfere with the write path. JSON processing, incremental modeling, ad hoc analysis, and historical replay must run without blocking the continuous arrival of new traces.

The requirement is therefore an end-to-end data pipeline connecting:

• Continuous ingestion

• Lossless raw-data retention

• Incremental cleaning and transformation

• Production of Eval-ready datasets

• Interactive analysis and troubleshooting

• Governed access and historical replay

Databend Cloud provides the primitives for this pipeline. Stage supports data arriving through object storage, while

VARIANT

Architecture: from Kafka events to Eval-ready Trace tables

In this deployment, behavioral events from web and application clients, together with Agent execution events, are represented as Traces and spans and streamed into Kafka before entering Databend Cloud.

The pipeline separates the immutable source of truth from the structures optimized for analysis:

1. Append raw events to

kafka_raw

Incoming events are appended to

kafka_raw

VARIANT

2. Process only newly arrived data

Stream continuously identifies new records. Task then cleans the JSON, extracts relevant fields, and performs incremental modeling. Because the pipeline processes changes rather than rescanning the full history, transformation work grows with new data instead of repeatedly paying for data already processed.

3. Write query-oriented records to

traces

Processed data is written to the

traces

kafka_raw

traces

Running ingestion, transformation, and analysis in production

A terabyte-per-hour Trace pipeline carries three concurrent workload classes:

• Continuous writes from production Agents and applications

• Incremental processing through Stream and Task

• Interactive Trace queries and Eval analysis from engineers

If they compete for the same resources, a complex investigation can disrupt real-time ingestion. Databend Cloud stores persistent data in the customer's own OSS object-storage bucket and allows ingestion, processing, and analytics to run on independent Warehouses. Each stage can scale according to its workload, so teams do not need to reserve an entire compute stack for peak demand at all times.

For long execution-path queries, analytical tables are organized around time and Trace identifiers, reducing the amount of data that must be scanned. Because ingestion, incremental processing, modeling, and analysis remain in one platform, the architecture also avoids additional data movement and orchestration across separate systems.

Governance is part of the same pipeline. Masking Policy can apply fine-grained masking rules to subpaths within

VARIANT

Why the team chose Databend Cloud

The company moved the pipeline into production within days. As a fully managed service, Databend Cloud brought ingestion, incremental processing, compute management, and day-to-day operations into one platform. The team did not need to build and maintain separate storage, scheduling, transformation, and analytical systems—or spend engineering time repeatedly configuring clusters and debugging cross-system data flows.

This shortened the path from initial integration to production and reduced the ongoing learning and maintenance burden. Engineers could spend more time on the work that directly affects Agent quality: improving Evals, refining the harness, and validating changes to models, prompts, and tools.

As raw Traces move into analysis-ready datasets, the feedback loop between Agent execution, issue diagnosis, and evaluation becomes shorter. Databend Cloud helps the customer's engineering team turn continuously growing Agent runtime data into durable assets for analysis and Evals—and use those assets to guide the next model and product iteration.

From Agent history to the next model iteration

Million-token contexts and complex tool use allow Agents to take on longer, more open-ended tasks. They also make the data produced by every run more valuable—and more demanding to manage.

For teams taking Agents into production, the data platform must do more than retain traces. It must ingest them continuously, process them incrementally, govern access, support fast analysis, and convert raw execution history into reliable evaluation feedback.

This deployment offers a practical blueprint: connect Agent Traces and Evals through one managed data pipeline, so every production run can contribute evidence for the next improvement to the model, harness, and product.