Massive write volumes
Online Trace data can reach TBs per hour and grow to trillions of records, pushing traditional databases beyond sustained ingestion limits.
AGENT TRACE SOLUTION
The production
data pipeline for
Agent Traces.
Capture model responses, tool calls, execution spans, and application events in one S3-backed lakehouse. Turn high-volume, evolving JSON into query-ready and eval-ready data—without operating a fragmented Trace pipeline.
Explore the architecture Trace explorerIllustrative data
TRACE IDtr_8f2a…
CompletedEXECUTION SPANS0 — 2.84 s
Agent executionVARIANT
{
"trace_id": "tr_8f2a",
"span": "agent.run",
"model": "reasoning-model",
"status": "completed",
"duration_ms": 2840
} S3-backed lakehouse Native JSON. Real SQL. Your data. Your retention.
PROVEN IN PRODUCTION
Read the customer story Validated by a leading foundation model company.
- Peak production Trace ingestion
- TB-scaleper hour
- Cumulative Trace data volume
- Trillionscale
- In a single Trace
- 100K+spans
- Largest single Trace observed
- 500 MBper Trace
Reported production case figures. Workload-specific results, not service limits or a performance guarantee.
01 / THE CHALLENGEAgent Traces are more
Agent Traces are more
than application logs.
Long-running executions. Branching tool calls. Ever-changing JSON. Your data pipeline needs to keep up with your agents.
Deeply nested Traces
A single long-running task can invoke thousands of tools, process millions of context tokens, and generate complex, multi-level spans.
Constant schema drift
Every model or tool upgrade can introduce new JSON fields, forcing frequent pipeline changes when ingestion depends on a fixed schema.
Fragmented data pipelines
Teams often stitch together Kafka, Flink, Airflow, a data warehouse, and object storage—creating longer pipelines and higher operational costs.
02 / ONE DATA FOUNDATIONFrom Agent execution
From Agent execution
to model improvement.
Databend Cloud unifies raw Trace storage, incremental SQL transformation, scheduling, and analytics in one managed pipeline. Preserve the complete execution context in S3, process only new events, and continuously produce query-ready Trace models for Evals, debugging, replay, attribution, and training.
Databend CloudONE MANAGED PIPELINE
Kafka / S3 Stage / COPY INTO
Ingest
Capture AI Agent and web or application events through an existing ingestion layer such as Kafka. Land NDJSON batch files in an S3 Stage, then use a load Task with COPY INTO and JSON cleanup to load raw events.
OUTPUTDurable raw events, ready for processing
Independent elastic warehousesIngest · Transform · Analyze · ReclusterAutomatic scaling / Auto-suspend
Full reference architecture
Scroll horizontally to explore the full diagram.
Ingest
Capture AI Agent and web or application events through an existing ingestion layer such as Kafka. Land NDJSON batch files in an S3 Stage, then use a load Task with COPY INTO and JSON cleanup to load raw events.
Retain
Preserve complete, heterogeneous JSON in the events table with native VARIANT. Extract frequently queried fields with computed columns, and use clustering and inverted indexes to support retrieval without discarding the original payload.
Transform
A Stream captures newly arrived rows. A Task runs incremental SQL to extract fields, normalize and enrich spans, MERGE INTO trace models, and refresh aggregates—without repeatedly transforming the full history.
Analyze & improve
Query trace models for debugging, Evals, replay, attribution, and training or reinforcement learning data preparation. Isolate ingestion, transformation, analytics, and reclustering on independent elastic warehouses.
Scale with Agent workloads
Sustain TB-scale Trace ingestion per hour with parallel loading and horizontal worker scaling. Keep writes stable as agent traffic and Trace volume grow.
Stay flexible as Agent data evolves
Preserve evolving, deeply nested JSON, then extract and search the fields that matter—without rebuilding the pipeline for every schema change.
Run one managed pipeline
Use Stream and Task for in-database processing and scheduling, without separate Flink or Airflow pipelines. Independent warehouses isolate workloads, while the managed service shortens the path to production.
Control your Trace data
Store complete raw traces in a customer-controlled lakehouse, with control over retention periods, field extraction, and analytical models for each agent and application.
03 / CORE CAPABILITIESBuilt for production.
Built for production.
Not just the demo.
The storage, processing, and governance primitives your Trace pipeline needs. Together in Databend Cloud.
Explore Databend Cloud| Capability | What it delivers |
|---|---|
| Native VARIANT | Preserves deeply nested and evolving JSON without defining every field upfront. |
| Stream + Task | Processes newly arrived events with incremental SQL pipelines. |
| Cluster Key + Recluster | Reduces scanning when retrieving long-running Traces by time and trace_id. |
| Elastic Warehouses | Isolates ingestion, transformation, querying, and maintenance workloads. |
| S3-Backed Storage | Supports customer-controlled retention and historical reprocessing. |
| PrivateLink + Masking | Protects sensitive prompts, model outputs, and application data. |
04 / PUT YOUR TRACES TO WORKEvery execution.
Every execution.
A chance to improve.
Keep the full history. Create the datasets that help engineering and model teams build better agents.
01
Debug complete Agent runs
Reconstruct model calls, tool invocations, decisions, and parallel branches by trace_id.
Find the why02
Build continuous Evals
Transform production Trace data into datasets for regression testing and model comparison.
Measure every change03
Replay critical executions
Use retained execution context to investigate failures and validate changes to models, prompts, tools, and agent harnesses.
Learn from failures04
Attribute outcomes
Connect quality, cost, and latency to the decisions and components that produced them.
Connect cause and effect05
Create training and RL data
Convert successful execution paths into structured assets for post-training and reinforcement learning.
Improve the next iteration Ready to go deeper? Explore the engineering behind the pipeline.Read the technical deep dive
05 / QUESTIONS, ANSWEREDA little more
A little more
detail.
Everything you need to understand the data foundation behind your agents.
Get the Markdown versionWhat is an Agent Trace data pipeline?
An Agent Trace data pipeline captures the complete execution history of an AI agent and turns it into data that teams can query and reuse. This includes model inputs and outputs, tool calls, retrieval steps, state changes, token usage, intermediate results, errors, and final outcomes. Unlike a basic logging pipeline, it must support long-running and branching executions, evolving JSON structures, incremental transformation, and downstream workflows such as Evals, replay, attribution, and training. Databend Cloud provides the storage, processing, scheduling, and analytical layers required to operate this pipeline on one S3-backed data foundation.
How is Databend Cloud different from Langfuse?
Langfuse is an LLM engineering and observability platform. Its self-hosted deployment includes PostgreSQL, ClickHouse, Redis or Valkey, blob storage, and application containers; Langfuse Cloud manages this infrastructure, with retention and usage governed by its current plans. Databend Cloud focuses on the underlying data pipeline for large-scale Agent Traces: native VARIANT storage in a customer-controlled lakehouse, SQL-defined JSON extraction and analytical models, incremental processing, and independent compute workloads. It provides a data foundation for debugging and evaluation workflows, rather than a like-for-like replacement for every Langfuse application feature. Consult each product’s current documentation for deployment and plan details.
Can Databend Cloud handle deeply nested and evolving JSON?
Yes. Databend Cloud stores raw JSON payloads in the native VARIANT data type, including nested objects and arrays. Teams do not need to define every possible field before ingestion. They can preserve the original payload, extract frequently queried fields with SQL, and update their transformation logic as agents, models, tools, and Trace formats evolve. This separates reliable data ingestion from downstream modeling, preventing every upstream JSON change from becoming an immediate pipeline migration.
How does Databend Cloud process new Trace data incrementally?
A Stream tracks changes that have not yet been consumed from the raw Trace table. A Task runs SQL on a schedule or when new rows are available. Together, Stream and Task can extract JSON fields, normalize event names, enrich spans, update detail tables, and generate aggregations using only newly arrived data. Teams avoid repeatedly scanning and transforming the complete historical dataset. The original raw records remain available for auditing, replay, and reprocessing when analytical requirements change.
Does the solution require separate ETL and orchestration systems?
Not for the core Trace transformation pipeline. Databend Cloud can sit behind an existing ingestion layer such as Kafka or object storage, then handle raw data retention, incremental SQL transformation, task scheduling, analytical modeling, and querying in one managed service. Teams can continue using external systems where their architecture requires them, but they do not need to deploy separate engines for every stage between raw Trace ingestion and analytical data production.
Build your production
Agent Trace pipeline.
One S3-backed data foundation on AWS.
From execution events to debugging, evaluation, replay, attribution, and training.
Fully managed Native SQL Customer-controlled data
