# Databend Blog — Engineering > Engineering deep-dives: storage, execution engine, query optimization, Rust, and performance. 44 posts. - [Databend Blog](https://www.databend.com/blog-llms.txt): Blog overview: all categories and latest posts. Article links point directly to original Markdown (including frontmatter; some posts contain MDX syntax). Remove the final .md from a URL to view the web page. ## Posts - [Debugging Payment Failures With OpenTelemetry and SQL in Databend Cloud](https://www.databend.com/blog/category-engineering/debugging-payment-failures-opentelemetry-sql.md): 2026-11-04 — Follow a simulated checkout request from error logs to traces, then use SQL to measure latency and calculate the value of failed orders. (tags: databend) - [Same PASS, Different Paths: Evaluating Agent Traces at the Span Level with Jev](https://www.databend.com/blog/category-engineering/same-pass-different-paths-span-level-eval.md): 2026-09-24 — Two coding agents can run the same test suite and produce byte-for-byte identical patches, yet take very different routes. (tags: databend) - [What PASS Misses: Lessons from Six Coding Agent Traces](https://www.databend.com/blog/category-engineering/what-pass-misses-six-coding-agent-traces.md): 2026-09-24 — I asked gpt-6-astra and claude-fable-5.1 to fix the same serde_json parsing bug through Evot, Pi, and DeepSeek Harness. All six runs passed the full test suite, and their core patches were byte-for-byte identical. (tags: databend) - [Jev Went Viral, So We Integrated It into Lakehouse](https://www.databend.com/blog/category-engineering/jev-in-lakehouse.md): 2026-09-22 — Can classification and filtering based on semantic meaning stay inside SQL? This article walks through a Python UDF integration between Databend and Jev, then examines where remote model calls fit in terms of latency, cost, and data governance. (tags: databend) - [Native data lineage in Databend: Tracing metric sources and assessing change impact](https://www.databend.com/blog/category-engineering/databend-native-data-lineage.md): 2026-09-17 — Available since Databend v1.2.935, Databend's native Data Lineage feature uses query plans to capture object-level and column-level dependencies directly inside the database. SQL queries and the Databend Cloud lineage graph provide two ways to inspect these relationships for change impact analysis, troubleshooting, data asset governance, and sensitive data tracking. (tags: databend) - [Databend Incremental Materialized Views - Change Tracking, Incremental Refresh, and Consistent Reads](https://www.databend.com/blog/category-engineering/incremental-materialized-views.md): 2026-09-02 — Materialized views are more than persisted query results. This article explains how Databend combines Change Tracking, checkpoints, aggregate states, and read fix to reduce refresh cost without sacrificing query correctness. It also outlines where incremental materialized views work well today. (tags: databend) - [Data Engineering in the AI Era - Taming Fragmented Agent Trace Pipelines with a Unified Lakehouse](https://www.databend.com/blog/category-engineering/unified-lakehouse-agent-traces.md): 2026-08-28 — AI accelerates application development, but it often leaves data teams with longer pipelines, more components, and inconsistent definitions across clouds. A unified, object-storage-based data platform can simplify this architecture by preserving raw data, processing changes incrementally, isolating workloads, and producing governed datasets for evaluation, analytics, and training. (tags: databend) - [Databend Cluster Key Best Practices: Choosing Columns, Key Order, and Granularity](https://www.databend.com/blog/category-engineering/cluster-key-best-practices.md): 2026-08-25 — Do not copy every frequently filtered column into `CLUSTER BY`. Start with the query families responsible for the most cumulative scanning, identify the selective predicates they share, match time granularity to the typical query window, and test alternative key orders under controlled conditions. Validate both the physical layout with `clustering_information` and the actual scan reduction with `EXPLAIN ANALYZE`, then compare the long-term query savings with the cost of maintaining that layout. (tags: databend) - [From Traditional Partitions to Micro-Partitions—How Snowflake and Databend Reduce Data Scanning](https://www.databend.com/blog/category-engineering/cluster-key-pruning.md): 2026-08-20 — A cluster key is neither a B-tree index nor a set of independent access paths for individual columns. It improves physical data locality so that similar key values tend to land in the same or adjacent micro-partitions or blocks. This narrows each physical unit's Min/Max range, reduces overlap, and allows the engine to prune more data before reading it. (tags: databend) - [From Kafka to Databend Cloud: Engineering a Trillion-Scale Agent Trace Ingestion Pipeline](https://www.databend.com/blog/category-engineering/agent-trace-ingestion.md): 2026-08-12 — This post is written for data engineers building Agent Trace pipelines, log analytics systems, Kafka-to-warehouse ingestion, or high-throughput semi-structured data platforms. It explains how Agent Trace data flows from Kafka into Databend Cloud, and how the open-source project [bend-ingest-kafka](https://github.com/databendcloud/bend-ingest-kafka) is designed for this workload. (tags: databend) - [From Global Sorts to Sketches](https://www.databend.com/blog/category-engineering/optimizer-sketches.md): 2026-07-28 — This article explains how Databend combines KLL, Top-N, and Count-Min Sketch (CMS) to build optimizer statistics with lower `ANALYZE` overhead. Together, they balance histogram efficiency with accurate estimates for skewed and high-frequency values. (tags: databend) - [Six Practical QUALIFY Patterns](https://www.databend.com/blog/category-engineering/qualify-patterns.md): 2026-07-23 — `QUALIFY` does more than remove a layer of nesting. It keeps window-function logic and the filter applied to its result in the same semantic stage, making analytical SQL easier to read, review, and maintain. This article covers six practical patterns, including event deduplication, Top N queries, CTE boundaries, and named windows. (tags: databend) - [Proving an AI-Written AST Visitor Migration Did Not Break SQL Semantics](https://www.databend.com/blog/category-engineering/ast-visitor-migration-semantic-verification.md): 2026-07-14 — An AST Visitor API migration is easy for AI to generate but hard to trust. This article explains how we validated that a new traversal preserved SQL semantics using source-derived AST metadata, runtime traces, independent comparison, diagnostics, real SQL inputs, and coverage feedback. (tags: databend) - [Orchestrating Reliable Data Ingestion from S3 with Airflow](https://www.databend.com/blog/category-engineering/s3-with-airflow.md): 2026-07-08 — This post walks through a minimal Airflow pipeline that uploads local CSV or NDJSON files to S3, then loads them into a cloud data warehouse with `COPY INTO`. The pattern keeps orchestration in Airflow, raw files in object storage, and ingestion plus analytics in the warehouse layer. It also gives you a practical starting point for retries, replay, credential handling, and production hardening (tags: databend) - [Small Bitmap, Big Gains: Optimizing Large-Scale Bitmap Aggregation in Databend](https://www.databend.com/blog/category-engineering/bitmap.md): 2026-07-06 — TL;DR: Databend introduces HybridBitmap to keep small sets lightweight and switch to RoaringTreemap only when needed. This reduces unnecessary deserialization and allocation, improving bitmap aggregation performance in small-bitmap-heavy workloads. (tags: databend) - [Building a 2KB Histogram with Sub -0.2% Percentile Error](https://www.databend.com/blog/category-engineering/openraft-histogram.md): 2026-07-01 — A lightweight histogram for tracking latency distributions with 2 KB of memory and sub -0.2% error. It uses float-like encoding for O(1) bucket indexing and trapezoid interpolation for accurate percentile estimation — with no floating-point math on the recording path. (tags: databend) - [Trace Is Evals: Data Engineering for Agent Trace Analysis and Attribution](https://www.databend.com/blog/category-engineering/trace-is-evals.md): 2026-06-22 — AI agents are moving from demos into long-running production workflows. That shift changes what we need from the data layer. In the chatbot era, the main question was whether the model could answer a single prompt. In the RAG era, the question became whether the model could retrieve and use the right context. In the agent era, the harder question is operational: (tags: databend) - [Building an Event-Driven CDC Pipeline: MySQL to Databend via AWS DMS and S3](https://www.databend.com/blog/category-product/mysql-to-databend-via-dms-s3.md): 2026-04-28 — Build a near-real-time CDC pipeline from MySQL to Databend Cloud using AWS DMS and S3. This guide explains how to use an append-only raw layer and Databend Streams to achieve 1-2 minute latency without self-hosted infrastructure. (tags: databend) - [Snowtree: Databend's Best Practices for AI-Native Development](https://www.databend.com/blog/category-engineering/snowtree.md): 2026-01-26 — Snowtree codifies Databend Team's AI-native development workflow with isolated worktrees, line-by-line review, and native CLI integration. (tags: Snowtree, AI, Worktree, Workflow, GitHub) - [Is Parquet becoming the bottleneck? Why new storage formats are emerging in 2025 (Lance, Vortex, and more)](https://www.databend.com/blog/category-engineering/2025-09-15-storage-format.md): 2025-09-15 — Parquet gave data lakes a common language: columnar layout, good compression, and fast scans. That still works well for classic analytics. But workloads have changed. We now mix wide scans with point lookups, handle embeddings and images, and run on S3-first stacks. On NVMe you want lots of tiny random reads. On S3 you want fewer, larger range requests. A format tuned for one world can feel chatty or slow in the other. (tags: Parquet, Lance Format, Apache Vortex, Columnar Storage, Storage Format, S3-Native Analytics, Object Storage, Data Lakehouse, Database Internals, Query Engine, Performance Engineering, SIMD, Random Access, AI Data Storage, Databend) - [Why We Built Our Own SQL Parser From Scratch: A Rust Implementation Story](https://www.databend.com/blog/category-engineering/2025-09-10-query-parser.md): 2025-09-10 — The story of why Databend chose to build its own SQL parser, the challenges faced, and how Rust's type system guided the solution. (tags: Rust, Parser, SQL, Databend, Engineering) - [Rust for Big Data: How We Built a Cloud-Native MPP Query Executor on S3 from Scratch](https://www.databend.com/blog/engineering/rust-for-big-data-how-we-built-a-cloud-native-mpp-query-executor-on-s3-from-scratch.md): 2025-09-05 — Deep dive into building a cloud-native MPP query executor on Amazon S3 with Rust — covering async execution, an actor-based scheduler, lock-free backpressure, and cost-aware I/O for high CPU utilization. (tags: Rust, S3, Databend, Engineering) - [Implementing a Lock-Free Channel in Rust for Databend’s Query Pipeline](https://www.databend.com/blog/engineering/implementing-lock-free-channel-rust-databend-query-pipeline.md): 2025-08-20 — A practical look at Databend's lock-free channel that connects query operators. This post describes a simple design that stores three state flags in the unused bits of a pointer to move large data blocks with minimal allocation and without traditional locks. (tags: Rust, Concurrency, Databend, Engineering) - [Slashing S3 Costs: How Databend Reduced List API Expenses by 70%](https://www.databend.com/blog/category-engineering/spill-list.md): 2025-01-17 — Databend, an open-source data warehouse, depends entirely on object storage services like Amazon S3 for data persistence. These services charge for API calls, and list operations can be particularly expensive. As our user base expanded, we observed a significant increase in our S3 costs, especially related to data spilling operations. This article outlines our efforts to optimize Databend's interaction with object storage, leading to a 70% reduction in S3 list API costs and notable performance enhancements. (tags: s3 infra, Engineering) - [Towards Efficient Distributed Group Aggregation](https://www.databend.com/blog/category-engineering/2024-04-12-towards-efficient-distributed-group-aggregation.md): 2024-04-12 — Databend recently improved its data processing capabilities by implementing a new aggregation hash table inspired by DuckDB. This update enhances group aggregation performance significantly in both standalone and distributed environments. Benchmarks using ClickBench/hits demonstrate enhancements ranging from 7% to 56% across various configurations, including local machines and cloud setups. This optimized hash table also supports spill functionality, accommodating larger datasets effectively. These enhancements ensure more efficient data processing, particularly in distributed scenarios. (tags: databend, hashtable, group aggregation) - [Diving into the Future: Serverless Data Warehouse Platform Architecture](https://www.databend.com/blog/category-engineering/2023-04-13-diving-into-the-future-serverless-data-warehouse-platform-architecture.md): 2023-04-13 — This article will introduce the design and trade-offs of the Databend Cloud architecture, as well as the work mechanism of its various components. We hope that this article will help you better understand the advantages of the cloud, making your cloud-based data warehouse more cost-effective, efficient, secure, and user-friendly. (tags: Databend, Databend Cloud, Serverless) - [Connecting to Databend with DBeaver](https://www.databend.com/blog/category-engineering/dbeaver.md): 2023-03-30 — Ways to connect to Databend via DBeaver (tags: beginner) - [Designing and Implementing the Query Result Cache](https://www.databend.com/blog/category-engineering/cache.md): 2023-03-09 — The efficiency of returning results for multiple identical queries has considerably boosted with the introduction of query result cache in Databend 1.0. This post explains how it works. (tags: cache) - [Databend 1.0 Release](https://www.databend.com/blog/category-engineering/databend-release-v1.0.md): 2023-03-05 — Check out what's fresh in Databend 1.0 (tags: databend, V1.0) - [Connecting to Databend and Running Queries with bendsql](https://www.databend.com/blog/category-engineering/bendsql.md): 2023-03-02 — Introducing the CLI tool bendsql (tags: beginner) - [Deploying a single instance of Databend on MinIO](https://www.databend.com/blog/category-engineering/deploying-databend-using-on-minio.md): 2023-02-24 — Using Databend to facilitate data analysis with MinIO (tags: beginner) - [Optimizing Databend Binary Builds with Profile-guided Optimization](https://www.databend.com/blog/category-engineering/profile-guided-optimization.md): 2023-02-13 — rustc supports doing profile-guided optimization (PGO). Let's see how we can make it work for Databend. (tags: databend, Profile-guided Optimization) - [Why You Should Try Sccache](https://www.databend.com/blog/category-engineering/sccache.md): 2023-02-02 — sccache (tags: databend, sccache) - [Rewrite sqllogictest framework in rust](https://www.databend.com/blog/category-engineering/2022-12-29-sqllogictest-framework.md): 2022-12-29 — sqllogictest, rust (tags: databend, sqllogictest, rust) - [CPU and Memory Profiling for Rust](https://www.databend.com/blog/category-engineering/profiling-rust.md): 2022-12-19 — CPU and Memory Profiling (tags: databend, CPU, Memory, Profiling) - [Databend Indexing Explained](https://www.databend.com/blog/category-engineering/databend-indexing.md): 2022-12-13 — Indexing (tags: databend, indexing) - [Speed up hive queries with databend](https://www.databend.com/blog/category-engineering/hive-on-databend.md): 2022-11-28 — hive on databend (tags: databend, hive) - [Replace Bloom Filter with Xor Filter to Speed up Queries](https://www.databend.com/blog/category-engineering/xor-filter.md): 2022-11-23 — Xor Filter (tags: databend, xor, bloom) - [Time Travel - Going Back to Move Forward](https://www.databend.com/blog/category-engineering/time-travel2.md): 2022-10-10 — time travel (tags: databend, time travel) - [Designing and Querying JSON in Databend](https://www.databend.com/blog/category-engineering/json-datatypes.md): 2022-09-14 — JSON (tags: databend, JSON) - [Find Peter Parker in Databend](https://www.databend.com/blog/category-engineering/time-travel.md): 2022-09-08 — time travel (tags: databend, time travel) - [Benchmarking Databend using TPC-H](https://www.databend.com/blog/category-engineering/2022-08-08-benchmark-tpc-h.md): 2022-08-08 — Benchmarking Databend using TPC-H (tags: databend, benchmark, TPC-H) - [The New Databend SQL Planner](https://www.databend.com/blog/category-engineering/new-planner.md): 2022-07-19 — The New Databend SQL Planner (tags: databend, planner) - [From Git to Fuse Engine](https://www.databend.com/blog/category-engineering/databend-engine.md): 2022-07-05 — About Databend engine (tags: databend, git, cloud warehouse)