# Agent Trace Solution # The Production Data Pipeline for Agent Traces Capture model responses, tool calls, execution spans, and application events in one S3-backed lakehouse. Turn high-volume, evolving JSON into query-ready and eval-ready data—without operating a fragmented Trace pipeline. [Start for free](https://app.databend.com/register) · [Contact sales](https://www.databend.com/contact-us/) # Proven in Production VALIDATED BY A LEADING FOUNDATION MODEL COMPANY ## TB-Scale per Hour Peak production Trace ingestion ## Trillion-Scale Cumulative Trace data volume ## 100K+ Spans in a single Trace ## 500 MB Maximum size of a single Trace [Read the customer story](https://www.databend.com/blog/category-customer/agent-trace-eval-pipeline/) # The Challenge ## Massive Write Volumes Online Trace data can reach TBs per hour and grow to trillions of records, pushing traditional databases beyond sustained ingestion limits. ## Deeply Nested Traces A single long-running task can invoke thousands of tools, process millions of context tokens, and generate complex, multi-level spans. ## Constant Schema Drift Every model or tool upgrade can introduce new JSON fields, forcing frequent pipeline changes when ingestion depends on a fixed schema. ## Fragmented Data Pipelines Teams often stitch together Kafka, Flink, Airflow, a data warehouse, and object storage—creating longer pipelines and higher operational costs. # The Solution & Architecture ## One Data Foundation from Agent Execution to Model Improvement Databend Cloud unifies raw Trace storage, incremental SQL transformation, scheduling, and analytics in one managed pipeline. Preserve the complete execution context in S3, process only new events, and continuously produce query-ready Trace models for Evals, debugging, replay, attribution, and training. ## Architecture: From Raw Agent Events to Eval-Ready Data [Architecture: From Raw Agent Events to Eval-Ready Data](https://www.databend.com/solutions/agent-trace/#architecture) # Benefits ## Scale with Agent Workloads Sustain TB-scale Trace ingestion per hour with parallel loading and horizontal worker scaling. Keep writes stable as agent traffic and Trace volume grow. ## Stay Flexible as Agent Data Evolves Preserve evolving, deeply nested JSON, then extract and search the fields that matter—without rebuilding the pipeline for every schema change. ## Run One Managed Pipeline Use Stream and Task for in-database processing and scheduling, without separate Flink or Airflow pipelines. Independent warehouses isolate workloads, while the managed service shortens the path to production. ## Control Your Trace Data Store complete raw traces in customer-controlled lakehouse, with full control over retention periods, field extraction, and analytical models for each agent and application. # Core Capabilities ## Built for Production Agent Trace Data | Capability | What It Delivers | | --- | --- | | Native VARIANT | Preserves deeply nested and evolving JSON without defining every field upfront. | | Stream + Task | Processes newly arrived events with incremental SQL pipelines. | | Cluster Key + Recluster | Reduces scanning when retrieving long-running Traces by time and trace_id. | | Elastic Warehouses | Isolates ingestion, transformation, querying, and maintenance workloads. | | S3-Backed Storage | Supports customer-controlled retention and historical reprocessing. | | PrivateLink + Masking | Protects sensitive prompts, model outputs, and application data. | [Explore Databend Cloud](https://www.databend.com/databend-cloud/) # Use Cases ## Debug Complete Agent Runs Reconstruct model calls, tool invocations, decisions, and parallel branches by trace_id. ## Build Continuous Evals Transform production Trace data into datasets for regression testing and model comparison. ## Replay Critical Executions Reproduce failures and validate changes to models, prompts, tools, and agent harnesses. ## Attribute Outcomes Connect quality, cost, and latency to the decisions and components that produced them. ## Create Training and RL Data Convert successful execution paths into structured assets for post-training and reinforcement learning. # FAQ ## What is an Agent Trace data pipeline? An Agent Trace data pipeline captures the complete execution history of an AI agent and turns it into data that teams can query and reuse. This includes model inputs and outputs, tool calls, retrieval steps, state changes, token usage, intermediate results, errors, and final outcomes. Unlike a basic logging pipeline, it must support long-running and branching executions, evolving JSON structures, incremental transformation, and downstream workflows such as Evals, replay, attribution, and training. Databend Cloud provides the storage, processing, scheduling, and analytical layers required to operate this pipeline on one S3-backed data foundation. ## How is Databend Cloud different from Langfuse? Self-hosted Langfuse requires teams to operate PostgreSQL, ClickHouse, Redis or Valkey, blob storage, and application containers. At high volume and concurrency, scaling and maintaining these components can increase operational complexity and infrastructure costs. Langfuse Cloud removes this burden, but data access and retention vary by plan, with additional usage billed in units. Databend Cloud provides one managed pipeline for large-scale Agent Trace data. It stores heterogeneous payloads as native VARIANT data in a customer-controlled lakehouse, where teams control retention and define JSON extraction and analytical models in SQL. It is built for TB-scale workloads with less infrastructure to operate. ## Can Databend Cloud handle deeply nested and evolving JSON? Yes. Databend Cloud stores raw JSON payloads in the native VARIANT data type, including nested objects and arrays. Teams do not need to define every possible field before ingestion. They can preserve the original payload, extract frequently queried fields with SQL, and update their transformation logic as agents, models, tools, and Trace formats evolve. This separates reliable data ingestion from downstream modeling, preventing every upstream JSON change from becoming an immediate pipeline migration. ## How does Databend Cloud process new Trace data incrementally? A Stream tracks changes that have not yet been consumed from the raw Trace table. A Task runs SQL on a schedule or when new rows are available. Together, Stream and Task can extract JSON fields, normalize event names, enrich spans, update detail tables, and generate aggregations using only newly arrived data. Teams avoid repeatedly scanning and transforming the complete historical dataset. The original raw records remain available for auditing, replay, and reprocessing when analytical requirements change. ## Does the solution require separate ETL and orchestration systems? Not for the core Trace transformation pipeline. Databend Cloud can sit behind an existing ingestion layer such as Kafka or object storage, then handle raw data retention, incremental SQL transformation, task scheduling, analytical modeling, and querying in one managed service. Teams can continue using external systems where their architecture requires them, but they do not need to deploy separate engines for every stage between raw Trace ingestion and analytical data production. # Build Your Production Agent Trace Pipeline Build a production-ready Agent Trace pipeline on AWS—from raw execution events to debugging, evaluation, replay, attribution, and training. [Start for free](https://app.databend.com/register) · [Contact sales](https://www.databend.com/contact-us/)