Data Engineering in the AI Era - Taming Fragmented Agent Trace Pipelines with a Unified Lakehouse

This article is adapted from a talk given by Databend founder River Ng at the industry tech salon on "Methodology for Agent Deployment: From Technical Construction to Enterprise Implementation." It draws on production experience across an AI application company, an industrial manufacturer, a public-sector big data group, a financial institution, and a foundation model company.

This post assumes familiarity with event streaming, cloud object storage, SQL analytics, and modern data warehouse or lakehouse architectures.

AI Development Is Getting Faster. Data Engineering Is Not.

A typical AI data pipeline starts with model APIs and application events, passes through tracing systems, Kafka, log stores, transformation jobs, and schedulers, and eventually produces an evaluation dataset or a serving table for BI. Each component may be reasonable in isolation. At production scale, however, the full chain becomes difficult to operate.

The problem becomes more pronounced in multi-cloud environments. Model and application data may be distributed across AWS, GCP, Azure and other platforms. The same transformation logic is then reimplemented in multiple databases and scripts, while data is repeatedly copied between systems.

This creates three related problems:

• Operational complexity: More connectors, schedulers, credentials, and failure points must be maintained.

• Inconsistent semantics: The same metric may be calculated differently across clouds or pipelines.

• Weak reproducibility: Teams cannot reliably reconstruct which data and logic produced an evaluation result or business metric.

The central challenge is therefore not simply storing more data. It is building a lakehouse that can absorb continuous change without allowing the pipeline to fragment around every new model, tool, or application.

Agent Traces Must Become Evaluation Data

Prompt and harness changes cannot be evaluated by intuition. Engineering teams need to determine whether a release improved task completion, reduced unnecessary tool calls, lowered token consumption, or introduced regressions in specific workloads.

That requires more than a trace viewer. Raw traces must be transformed into datasets that can support:

• model, prompt, and harness comparisons;

• task replay and regression analysis;

• tool-call and path attribution;

• cost and latency analysis;

• supervised fine-tuning and reinforcement learning.

Evot, an open-source project developed by Databend, demonstrates the application layer of this workflow. It records and visualizes how agents such as Claude Code, Pi, and DeepSeek Agent invoke models and tools, including execution steps, latency, and path differences. The harder engineering problem sits below that interface: continuously ingesting the underlying traces and converting them into reliable evaluation datasets.

Why Agent Trace Data Is Difficult to Process

Production scale changes the workload

Small AI demonstrations may operate on a handful of documents or a few million records. Production agents generate a different workload entirely. A single complex task may run for hours, accumulate millions of context tokens, and produce a GB-scale trace. A foundation model company discussed in this article continuously lands close to 100 TB of data per day from hundreds or thousands of servers.

The system must ingest this data without allowing JSON parsing, aggregation, or ad hoc analysis to interrupt the write path. Retention also matters: when tens of billions of events arrive every day and must remain available for six months or longer, storage design becomes as important as query performance.

JSON schemas evolve continuously

Agent traces are deeply nested and only partially predictable. A span may contain model inputs and outputs, tool parameters, tool responses, event arrays, intermediate state, token usage, and error details. Different models—or different versions of the same model—may add fields, omit fields, or change data types.

Not every payload described as JSON is valid JSON. Production pipelines encounter mixed types inside arrays, malformed escaping, extra braces, and fields that violate an expected format. Invalid records must be isolated without blocking the rest of the stream.

Rigidly modeling every field before ingestion makes the pipeline brittle. Keeping everything as opaque JSON avoids schema changes at write time, but pushes repeated parsing costs onto every downstream query. A practical architecture must preserve the complete source payload while extracting stable, frequently queried fields into an analytical model.

Multi-Cloud Sprawl Turns Data Into an Integration Problem

Fast-moving AI companies tend to adopt specialized services quickly. Over time, data becomes divided across cloud providers, databases, tracing systems, and transformation tools. Teams may find different results for the same query depending on which environment they inspect—or may not know which cloud contains the authoritative copy.

The operational burden extends beyond data movement. Engineers must coordinate job timing, manage cross-cloud traffic, scale individual components, investigate partial failures, and keep duplicated transformation logic aligned.

One foundation model company had three engineers dedicated to maintaining this type of pipeline. Expected growth from an upcoming model release made the existing architecture increasingly difficult to sustain.

A Unified Data Pipeline for Trace, Evaluation, and Analytics

The redesigned architecture consolidated data ingestion and transformation around a shared lakehouse:

Trace data from multiple clouds first enters a unified Kafka layer and is then written continuously to Databend Cloud. Once inside the platform, the pipeline performs validation, parsing, field extraction, semantic normalization, trace aggregation, quality checks, and serving-table refreshes.

This architecture is built around four principles.

1. Preserve the raw event before imposing structure

Raw payloads are stored in the

VARIANT

Malformed events are marked for review rather than allowed to block valid data. Once validated, frequently used fields such as model, prompt, tool, token count, cost, and status are extracted into durable analytical tables. Spans are then grouped by

trace_id

The result is a deliberate separation between two needs:

• raw data preserves fidelity and accommodates schema evolution;

• modeled data supports efficient, repeatable analysis.

2. Perform ELT in the shared data platform

ELT is valuable for more than reducing data movement. When transformations remain inside one platform, table-level and column-level lineage can be captured with the data itself. Engineers can determine which source tables produced a serving table and which upstream fields contributed to a derived metric.

This is difficult when logic is spread across standalone scripts, schedulers, stream processors, and multiple databases. Consolidating transformation does not remove the need for governance, but it makes dependencies visible and auditable.

3. Process only new data

Databend Streams track changes to source tables. Tasks run scheduled or event-driven transformations over those changes, while UDFs can implement custom processing or distribution logic.

The pattern can be summarized as:

This avoids repeatedly scanning the entire history whenever new traces arrive. Some Databend users apply the same

Stream + Task + UDF

4. Separate storage from compute—and isolate compute workloads

Object storage provides durable capacity without requiring a fixed storage cluster sized for future growth. Stateless compute warehouses can then be assigned by workload:

• an ingestion warehouse handles continuous writes;

• a processing warehouse transforms and aggregates data;

• a serving warehouse supports user-facing queries and BI.

These warehouses share the same stored data but scale independently. A serving warehouse, for example, can start at 08:00 and suspend after 20:00 when no requests remain, while ingestion continues uninterrupted. Multi-cluster warehouses can add compute during concurrency spikes and release it when demand falls.

The design prevents a large evaluation query or background transformation from consuming the resources required by the real-time ingestion path.

Case Highlight: A Foundation Model Company Processing Nearly 100 TB per Day

The foundation model company described earlier needed to consolidate trace data distributed across several clouds and systems. Its production workload included:

• complex agent tasks that ran for hours;

• GB-scale traces with millions of context tokens;

• tens of billions of events per day;

• nearly 100 TB of data landing in the lakehouse each day;

• a retention period of at least six months for a significant share of the data.

The original pipeline required three engineers to coordinate components, maintain jobs, and investigate failures. After ingestion and transformation were consolidated in Databend Cloud, routine operations required approximately 0.5 FTE—about one-sixth of the previous staffing level. An intern could handle much of the daily operation, while alerts surfaced exceptions for targeted investigation.

The new architecture also separated always-on ingestion from scheduled serving. Data continued to arrive while user-facing compute could start and stop according to demand. Tasks automated incremental processing that had previously depended on multiple interfaces and manually coordinated jobs.

The most important result was not component reduction by itself. Production traces became a reusable data asset: new model or harness versions could be run against historical workloads, evaluated against previous results, and examined by task, path, cost, or failure mode before release.

Is Kafka Always Necessary?

Kafka remains appropriate when a pipeline requires low-latency fan-out, ordered streams, replayable offsets, mature consumer management, or isolation between producers and consumers. It should not be removed simply to reduce the architecture diagram.

Object storage notifications can provide a simpler alternative for workloads that already write immutable files and can tolerate object-level delivery semantics. Applications write directly to object storage, notifications identify new objects, and downstream processing consumes only the additions.

This pattern can reduce cross-cloud transfer and operational costs in some environments, particularly when object storage is already the system of record. The trade-off is that object notifications do not automatically provide all of Kafka's delivery, ordering, backpressure, and replay behavior. The choice should follow workload requirements rather than a blanket “no Kafka” rule.

The Same Architecture Pattern Beyond Agent Traces

The underlying problem—high-volume, continuously arriving, imperfect data—also appears outside foundation model development.

The manufacturing case is especially instructive. Machine telemetry and agent traces appear unrelated at the application level, yet they impose similar data requirements: sustained ingestion, irregular records, raw-data fidelity, incremental cleanup, and workload-specific serving.

A Practical Framework: Ingest, Adapt, Isolate, and Serve

The production lessons in these cases can be condensed into four requirements:

1. Ingest reliably. Preserve high-volume raw data before downstream assumptions discard useful context.

2. Adapt continuously. Use flexible JSON storage and incremental transformation to accommodate evolving schemas and business logic.

3. Isolate workloads. Prevent ingestion, transformation, evaluation, and serving queries from competing for the same fixed compute resources.

4. Serve governed data. Produce trace, evaluation, BI, and training datasets with consistent definitions, lineage, access control, and auditability.

Building a Unified Lakehouse for AI and Agent Workloads

Data engineering in the AI era is not only about storing model and agent output. A useful lakehouse must absorb sustained growth, adapt to schema and application changes, execute complex transformations reliably, and make the resulting data available to models, agents, engineers, and business applications.

For Agent Trace and evaluation workloads, the path from raw event to trusted dataset is part of the model-development system itself. When ingestion, incremental processing, lineage, workload isolation, and serving are designed as one coherent architecture, teams spend less time coordinating infrastructure and more time measuring whether their AI systems are actually improving.

That direction also changes the division of responsibility between the platform and its users. Structured, semi-structured, and unstructured data should be accessible through a consistent analytical interface; users should be able to express transformations in SQL without manually managing every partition or index. Data organization, execution planning, and resource scaling should increasingly be handled by the system rather than repeated across application teams.