Databend Product Updates - From Spatial Index Join to Eval Data Pipelines

Over June and July 2026, Databend shipped 182 changes across query optimization, spatial analytics, open lakehouse integration, data pipelines, stability, and security.

The optimizer can now model skewed data more accurately with KLL Histograms, Count-Min Sketches, and Top-N statistics. Spatial indexes have moved beyond scan pruning into join execution. Paimon joins the catalog ecosystem, data export supports more formats, and Stream and Task improvements make incremental pipelines easier to operate.

This post covers the most important changes and shows how several of them fit together in a practical model evaluation pipeline.

Two Months, 182 Updates

From

v1.2.916-nightly

v1.2.930-nightly

Bug fixes made up the largest category, but this was more than a stability cycle. Optimizer statistics, Spatial Index Join, and Paimon Catalog all introduced meaningful new capabilities.

Better Statistics for Better Query Plans

An optimizer chooses join order, scan strategy, and operator combinations from its estimate of the underlying data. When that estimate misses skew or frequent values, an otherwise valid query can receive an unnecessarily expensive plan.

Databend strengthened its statistics stack in four areas:

• KLL Histogram: Builds

  ANALYZE
  

  histograms from a KLL Sketch, balancing statistical accuracy with memory usage.

• Count-Min Sketch: Estimates value frequency with low space overhead, helping the optimizer detect heavy hitters and skew.

• Top-N selectivity: Improves equality-filter estimates and refreshes Top-N statistics after data is appended.

• Pruning I/O in

  EXPLAIN
  

  : Makes the I/O cost of pruning visible in the query plan, so developers can see how much work an index removes.

For users, the benefit is straightforward: more accurate plans without rewriting SQL. After running

ANALYZE

ANALYZE TABLE events;

EXPLAIN
SELECT *
FROM events
WHERE user_id = 42;

Spatial Indexes Move Into Join Execution

Databend already used spatial indexes to prune scans. This release cycle extends them into join processing.

Consider a query that maps points of interest to administrative regions. Evaluating the geometry predicate against every possible row pair becomes expensive as both tables grow. An R-Tree-based Spatial Index Join first narrows the candidate set, then applies the exact spatial predicate.

The update includes:

• Local Spatial Index Join using an R-Tree to avoid a full pairwise comparison.

• Distributed Broadcast Spatial Join for joining a large dataset with a smaller spatial table across a cluster.

• Lower geometry-processing overhead by streaming bounding-box extraction and reading the SRID directly from the EWKB header with

  geozero
  

  .

Databend also removed the previous spatial runtime filters from join planning and added regression coverage for spatial join correctness.

SELECT
    r.region_name,
    p.name
FROM regions AS r
JOIN pois AS p
    ON ST_CONTAINS(r.boundary, p.location);

Paimon Joins the Open Lakehouse Ecosystem

One of the largest lakehouse updates is the new Paimon Catalog, contributed by the community.

Databend can now read Paimon tables directly and supports distributed writes. Teams already storing data in Paimon can connect Databend to their analytics workflow without first copying data into another format or storage layer.

Iceberg compatibility also improved. Predicate pushdown now preserves supported conjuncts more reliably, and CI coverage includes reading Iceberg Variant Metadata to reduce the risk of regressions.

The direction is consistent: Databend should work with data where it already lives, rather than make teams migrate it before they can query it.

More Ways to Export and Trace Data

The latest data movement work expands both output formats and file-level observability.

Avro, ORC, and Arrow

COPY INTO @stage

COPY INTO @stage/out
FROM events
FILE_FORMAT = (TYPE = ORC);

Better Schema Inference and File Metadata

NDJSON schema inference has been improved. Stage queries also expose additional metadata columns:

• metadata$file_path
  

• metadata$file_basename
  

• metadata$file_content_key
  

• metadata$file_last_modified
  

For CSV, Text, NDJSON, and Avro,

metadata$filename

SELECT
    metadata$file_path,
    metadata$file_last_modified,
    $1
FROM @stage/in (FILE_FORMAT => 'ndjson');

SQL and Pipeline Improvements

Several smaller changes remove friction from common development and operations workflows:

• ILIKE
  

  adds case-insensitive pattern matching without wrapping expressions in

  LOWER()
  

  .

• PostgreSQL aggregate syntax compatibility reduces query rewrites during migration.

• Stream Backlog adds an API and table function for estimating pending changes.

• Materialized CTEs provide explicit execution semantics when an intermediate result should be reused.

• Virtual columns in the community edition improve access to frequently queried fields in semi-structured data.

• Faster Time Travel lookup adds a

  NO_CHECK
  

  snapshot path and uses UUID v7 for timestamp navigation.

SELECT *
FROM logs
WHERE msg ILIKE '%timeout%';

SELECT *
FROM stream_backlog('my_stream');

Smoother Distributed Writes and Storage Maintenance

Storage and execution work focused on parallelism, clustering, and reducing resource spikes.

Distributed Multi Insert

For statements such as

INSERT ALL

INSERT FIRST

This lets large multi-target writes use the cluster more effectively.

Clustering and Background Work

Other changes include:

• Cluster-depth percentile metrics.

• Skipping unnecessary Partial Sort work for ordered tasks.

• Removal of the legacy Hilbert Clustering implementation.

• A new Parquet writer and better string-statistics sizing for shared prefixes.

• Incremental block deletion in Vacuum2 to reduce peak resource usage.

• Limits on concurrent I/O operations to prevent storage request spikes.

Tighter Security and Tenant Boundaries

Security updates focused on outbound access and multi-tenant isolation:

• Endpoint Egress Policy: An allowlist-based

  endpoint_url_policy
  

  limits which external endpoints a query can access.

• Stage path traversal protection: A dedicated policy protects Stage paths, with control moved from Settings to Config.

• Credential redaction: Connection credentials are further hidden in logs and error messages.

• Tenant isolation fixes: Session tenant overrides, HTTP session state, and an empty role-cache edge case during reload races were addressed.

The goal is to make external data access explicit and controlled while keeping tenant state isolated under production concurrency.

Databend Meta Adds Lua Transactions

Databend Meta now supports Lua Transactions, allowing operators to orchestrate transactional metadata operations with Lua scripts. A monotonic

metactl.now_ms()

The Meta layer also added Protobuf storage variants, removed Proto encoding failure paths, continued splitting key error builders, decoupled create options for indexes and databases, separated dictionary updates by ID, and added bulk-load and random-read benchmarks to

metabench

What the 80 Bug Fixes Addressed

The 80 fixes cluster around four production concerns.

Query Crashes and Stalls

• Prevented stack overflow and

  SIGSEGV
  

  in deeply nested expressions.

• Fixed a potential self-deadlock during runtime destruction.

• Fixed a Recluster loop caused by inconsistent block-size estimates.

Result Correctness

• Corrected

  UNNEST
  

  behavior for nullable Array and Variant values.

• Fixed domain calculation for Date and Timestamp arithmetic overflow.

• Resolved SELECT alias shadowing in

  GROUPING SETS
  

  .

• Ensured variables are set to

  NULL
  

  when a subquery returns no rows.

Join and Predicate Planning

• Prevented equi-hash joins from incorrectly falling back to nested-loop joins.

• Improved safe handling of single-row inequality joins.

• Preserved null-safe join keys through null-filter rules.

• Fixed lost union coercion during filter pushdown.

Upgrade Compatibility

• Restored reads for legacy Bincode v4 segments.

• Restored rollback compatibility for cluster statistics pages.

• Made HLL row-count statistics deterministic in commit metadata.

These fixes are not always visible in a feature demo, but they determine whether long-running queries, pipelines, and upgrades remain trustworthy.

Building a Model Eval Observability Pipeline

The following example connects Stage metadata,

VARIANT

Eval JSON → S3 Stage → Raw Table → Stream → Clean Task → Clean Table

Evaluation observations arrive in S3 as NDJSON. A scheduled Task loads new files into a raw table. A Stream captures appended rows, and a second Task turns nested JSON into structured data for analysis.

1. Load Raw Events and Preserve Lineage

The raw table stores the complete payload in

VARIANT

CREATE TABLE eval_raw (
    payload   VARIANT,
    file_path STRING,
    loaded_at TIMESTAMP
);

COPY INTO eval_raw
FROM (
    SELECT
        $1,
        metadata$file_path,
        NOW()
    FROM @eval_stage (FILE_FORMAT => 'ndjson')
);

This keeps the original event available for schema evolution and makes every record traceable to its source file.

2. Capture Incremental Changes

CREATE STREAM eval_stream ON TABLE eval_raw;

The Stream lets downstream processing consume only new rows instead of rescanning the raw table.

3. Flatten the Payload

Databend uses path expressions to access

VARIANT

::

FLATTEN

INSERT INTO eval_clean
SELECT
    payload:model_name::STRING   AS model_name,
    payload:eval_id::STRING      AS eval_id,
    payload:timestamp::TIMESTAMP AS eval_time,
    sample:input::STRING         AS input,
    sample:score::DOUBLE         AS score,
    sample:passed::BOOLEAN       AS passed
FROM eval_stream,
LATERAL FLATTEN(input => payload:samples) AS sample
WHERE payload:model_name IS NOT NULL;

4. Monitor Backlog

SELECT *
FROM stream_backlog('eval_stream');

If the backlog continues to grow, the cleaning Task may need to run more frequently or use additional compute. Stage metadata provides lineage, the Stream tracks incremental changes, and Stream Backlog shows whether processing is keeping up.

Task reliability also improved during this release cycle. Fixes aligned Task History with Databend Cloud, released stuck failed runs, preserved Task options during

ALTER SET

For an unattended pipeline, these execution guarantees matter as much as new SQL syntax.

Databend Cloud Agent Enters Private Preview

Databend Cloud Agent is an AI assistant embedded in the Databend Cloud console. It is currently in private preview and will become available more broadly over time.

Users can describe a task in natural language, and the Agent can act through 21 capability domains and roughly 110 tools. Supported workflows include querying data, managing warehouses, investigating billing, configuring data synchronization, and creating visualizations.

The goal is not only to explain what a user should do, but to complete the operation inside the console.

Client and Driver Updates

The surrounding client ecosystem also moved forward.

bendsql

• v0.34.1, released July 8: Added Python 3.14 support and key-pair authentication; fixed incorrect splitting of

  BEGIN ... END
  

  blocks at internal semicolons.

• v0.34.2, released July 16: Fixed the Node.js Windows build.

databend-jdbc

Versions

v0.4.7

v0.4.8

databend-client

These updates rarely lead a product demo, but they directly affect application reliability and developer experience.

What This Means for Agent Trace Workloads

The June–July updates reinforce Databend across optimizer accuracy, spatial analytics, lakehouse interoperability, pipeline operations, security, and production reliability. Together, they also strengthen a workload that combines many of these requirements: Agent Trace analytics.

Agent runs continuously produce evolving Prompt, Tool Call, Span, latency, token, error, and Eval data. Databend can retain the original trace in

VARIANT

Rather than splitting ingestion, transformation, Eval analysis, retrieval, and long-term retention across isolated systems, teams can manage them in one S3-native, elastic warehouse. Structured SQL analytics, JSON and full-text search, vector retrieval, Agent Trace, and Eval workloads can operate on the same data foundation. These capabilities have been validated at scale in production by leading AI companies in China.