# Agent Trace Data Pipeline | Databend Cloud

> Ingest, transform, retain, and analyze production-scale Agent Traces in one S3-backed lakehouse. Turn evolving JSON into eval-ready data.

Canonical: https://www.databend.com/solutions/agent-trace/
Language: en

## The production data pipeline for Agent Traces.

Capture model responses, tool calls, execution spans, and application events in one S3-backed lakehouse. Turn high-volume, evolving JSON into query-ready and eval-ready data—without operating a fragmented Trace pipeline.

[Start for free](https://app.databend.com/register) · [Contact sales](https://www.databend.com/contact-us/)

## PROVEN IN PRODUCTION

Validated by a leading foundation model company.

- **TB-scale per hour**: Peak production Trace ingestion
- **Trillion scale**: Cumulative Trace data volume
- **100K+ spans**: In a single Trace
- **500 MB per Trace**: Largest single Trace observed

Reported production case figures. Workload-specific results, not service limits or a performance guarantee.

[Read the customer story](https://www.databend.com/blog/category-customer/agent-trace-eval-pipeline/)

## 01 / THE CHALLENGE

### Massive write volumes

Online Trace data can reach TBs per hour and grow to trillions of records, pushing traditional databases beyond sustained ingestion limits.

### Deeply nested Traces

A single long-running task can invoke thousands of tools, process millions of context tokens, and generate complex, multi-level spans.

### Constant schema drift

Every model or tool upgrade can introduce new JSON fields, forcing frequent pipeline changes when ingestion depends on a fixed schema.

### Fragmented data pipelines

Teams often stitch together Kafka, Flink, Airflow, a data warehouse, and object storage—creating longer pipelines and higher operational costs.

## Full reference architecture

Databend Cloud unifies raw Trace storage, incremental SQL transformation, scheduling, and analytics in one managed pipeline. Preserve the complete execution context in S3, process only new events, and continuously produce query-ready Trace models for Evals, debugging, replay, attribution, and training.

Reference flow: Agent / app events → Kafka → S3 Stage → load Task → events (VARIANT) → Stream → Task → traces → data applications. Kafka is an example ingestion layer, not a required component.

AI Agents and web or application events flow through Kafka to an S3 Stage. A load Task uses COPY INTO and JSON cleanup to store events as VARIANT with computed columns, clustering, and an inverted index. Stream captures new rows; Task uses MERGE INTO and refreshes aggregates to create trace models for debugging, Evals, replay, attribution, and training or reinforcement learning. Independent elastic warehouses isolate ingestion, transformation, analysis, and reclustering. Moving dashes indicate data direction, not measured throughput.

### 1. Ingest

Kafka / S3 Stage / COPY INTO

Capture AI Agent and web or application events through an existing ingestion layer such as Kafka. Land NDJSON batch files in an S3 Stage, then use a load Task with COPY INTO and JSON cleanup to load raw events.

OUTPUT: Durable raw events, ready for processing

### 2. Retain

VARIANT / Computed columns

Preserve complete, heterogeneous JSON in the events table with native VARIANT. Extract frequently queried fields with computed columns, and use clustering and inverted indexes to support retrieval without discarding the original payload.

OUTPUT: Complete execution context in S3-backed storage

### 3. Transform

Stream / Task / MERGE INTO

A Stream captures newly arrived rows. A Task runs incremental SQL to extract fields, normalize and enrich spans, MERGE INTO trace models, and refresh aggregates—without repeatedly transforming the full history.

OUTPUT: Query-ready and eval-ready traces

### 4. Analyze & improve

SQL / Independent warehouses

Query trace models for debugging, Evals, replay, attribution, and training or reinforcement learning data preparation. Isolate ingestion, transformation, analytics, and reclustering on independent elastic warehouses.

OUTPUT: Reusable datasets for the next model iteration

[Full reference architecture](https://www.databend.com/solutions/agent-trace/#architecture)

## ONE MANAGED PIPELINE

### Scale with Agent workloads

Sustain TB-scale Trace ingestion per hour with parallel loading and horizontal worker scaling. Keep writes stable as agent traffic and Trace volume grow.

### Stay flexible as Agent data evolves

Preserve evolving, deeply nested JSON, then extract and search the fields that matter—without rebuilding the pipeline for every schema change.

### Run one managed pipeline

Use Stream and Task for in-database processing and scheduling, without separate Flink or Airflow pipelines. Independent warehouses isolate workloads, while the managed service shortens the path to production.

### Control your Trace data

Store complete raw traces in a customer-controlled lakehouse, with control over retention periods, field extraction, and analytical models for each agent and application.

## Capabilities

| Capability | What it delivers |
| --- | --- |
| Native VARIANT | Preserves deeply nested and evolving JSON without defining every field upfront. |
| Stream + Task | Processes newly arrived events with incremental SQL pipelines. |
| Cluster Key + Recluster | Reduces scanning when retrieving long-running Traces by time and trace_id. |
| Elastic Warehouses | Isolates ingestion, transformation, querying, and maintenance workloads. |
| S3-Backed Storage | Supports customer-controlled retention and historical reprocessing. |
| PrivateLink + Masking | Protects sensitive prompts, model outputs, and application data. |

[Explore Databend Cloud](https://www.databend.com/databend-cloud/)

## Use cases

### Debug complete Agent runs

Reconstruct model calls, tool invocations, decisions, and parallel branches by trace_id.

### Build continuous Evals

Transform production Trace data into datasets for regression testing and model comparison.

### Replay critical executions

Use retained execution context to investigate failures and validate changes to models, prompts, tools, and agent harnesses.

### Attribute outcomes

Connect quality, cost, and latency to the decisions and components that produced them.

### Create training and RL data

Convert successful execution paths into structured assets for post-training and reinforcement learning.

[Read the technical deep dive](https://www.databend.com/blog/category-engineering/agent-trace-ingestion/)

## FAQ

### What is an Agent Trace data pipeline?

An Agent Trace data pipeline captures the complete execution history of an AI agent and turns it into data that teams can query and reuse. This includes model inputs and outputs, tool calls, retrieval steps, state changes, token usage, intermediate results, errors, and final outcomes. Unlike a basic logging pipeline, it must support long-running and branching executions, evolving JSON structures, incremental transformation, and downstream workflows such as Evals, replay, attribution, and training. Databend Cloud provides the storage, processing, scheduling, and analytical layers required to operate this pipeline on one S3-backed data foundation.

### How is Databend Cloud different from Langfuse?

Langfuse is an LLM engineering and observability platform. Its self-hosted deployment includes PostgreSQL, ClickHouse, Redis or Valkey, blob storage, and application containers; Langfuse Cloud manages this infrastructure, with retention and usage governed by its current plans. Databend Cloud focuses on the underlying data pipeline for large-scale Agent Traces: native VARIANT storage in a customer-controlled lakehouse, SQL-defined JSON extraction and analytical models, incremental processing, and independent compute workloads. It provides a data foundation for debugging and evaluation workflows, rather than a like-for-like replacement for every Langfuse application feature. Consult each product’s current documentation for deployment and plan details.

### Can Databend Cloud handle deeply nested and evolving JSON?

Yes. Databend Cloud stores raw JSON payloads in the native VARIANT data type, including nested objects and arrays. Teams do not need to define every possible field before ingestion. They can preserve the original payload, extract frequently queried fields with SQL, and update their transformation logic as agents, models, tools, and Trace formats evolve. This separates reliable data ingestion from downstream modeling, preventing every upstream JSON change from becoming an immediate pipeline migration.

### How does Databend Cloud process new Trace data incrementally?

A Stream tracks changes that have not yet been consumed from the raw Trace table. A Task runs SQL on a schedule or when new rows are available. Together, Stream and Task can extract JSON fields, normalize event names, enrich spans, update detail tables, and generate aggregations using only newly arrived data. Teams avoid repeatedly scanning and transforming the complete historical dataset. The original raw records remain available for auditing, replay, and reprocessing when analytical requirements change.

### Does the solution require separate ETL and orchestration systems?

Not for the core Trace transformation pipeline. Databend Cloud can sit behind an existing ingestion layer such as Kafka or object storage, then handle raw data retention, incremental SQL transformation, task scheduling, analytical modeling, and querying in one managed service. Teams can continue using external systems where their architecture requires them, but they do not need to deploy separate engines for every stage between raw Trace ingestion and analytical data production.

## Build your production Agent Trace pipeline.

One S3-backed data foundation on AWS. From execution events to debugging, evaluation, replay, attribution, and training.

[Start for free](https://app.databend.com/register) · [Contact sales](https://www.databend.com/contact-us/)