--- title: 'Databend product updates for August and September 2026: materialized views, data sharing, and lineage' date: 2026-10-14 slug: category-product/aug-sep-bimonthly-report cover_url: 'https://static.databend.cn/img/202610/banner-en.png' description: 'An expensive analytical query often leaves more work after the result arrives. Data teams still need to decide whether to persist the result, how to refresh it as source data changes, and how to share it without building another export and synchronization pipeline. A schema change raises a different question: which downstream tables, columns, and reports will it affect?' tags: [databend] category: Product authors: - DatabendLabs --- Databend's August and September 2026 updates address that workflow. Materialized views and Dynamic Tables add options for persisting reusable results. Data Sharing provides read-only access across tenants in the same Databend Meta cluster, including independent Stream consumption. Data lineage adds object-level and column-level dependency queries. The release also includes changes to partitioning, TopN execution, indexing, resource management, and query correctness. This post is intended for data engineers and platform teams familiar with analytical SQL, views, change tracking, and incremental data processing. {/* truncate */} ## Release snapshot Across GitHub Releases, the Databend repository published 20 nightly releases, from `v1.2.931-nightly` through `v1.2.950-nightly`, and two patch releases: `v1.2.925-patch-6` and `v1.2.925-patch-13`. After deduplicating pull request entries by number, the nightly release notes contain 272 shipped changes. ![](https://static.databend.cn/img/202610/release-overview-en.png) These figures describe changes included in published releases. They are not a count of pull requests merged during the same calendar period. The `feat` category includes internal tooling, while `fix` includes test and CI work, so the category totals should not be read as counts of user-facing features or production incidents. ## Materialized views: reusing results with less repeated work A materialized view persists query results or intermediate aggregate states. A regular view stores only the query definition; a materialized view also has to account for source changes, refresh behavior, read consistency, and query rewrite. ### Change tracking and hybrid reads Databend materialized views use `CHANGE_TRACKING` on the source table to identify changes. `REFRESH MATERIALIZED VIEW` updates the persisted result. - Appends support incremental refresh. - Non-aggregate materialized views use internal source row identifiers to process updates and deletes. - Aggregate materialized views store aggregate states and can merge existing states with states derived from appended data. Updates or deletes in the source may still require a full recomputation. - Hybrid Reads can combine persisted results with source-table increments that have not yet been refreshed in append-only workloads, reducing the need for a full source scan. ### Query rewrite and maintenance The optimizer now has materialized-view rewrite rules, an `enable_materialized_view_rewrite` setting, cost-based candidate selection, and scan routing. A candidate view must meet requirements such as output-column compatibility. The optimizer compares its cost with the original plan before choosing an execution path. Materialized views also support cluster keys, reclustering, compaction, indexes, and visibility through `system.materialized_views`. During compaction, aggregate views can merge aggregate states again while preserving their semantics. The following example creates a category-level aggregate over an append-oriented sales table: ```SQL CREATE TABLE sales ( category STRING, amount INT ) CHANGE_TRACKING = TRUE; CREATE MATERIALIZED VIEW mv_sales ( category, total_amount ) CLUSTER BY (category) AS SELECT category, SUM(amount) FROM sales GROUP BY category; REFRESH MATERIALIZED VIEW mv_sales; SELECT * FROM mv_sales; ``` Materialized views are available in Databend Enterprise Edition and require the corresponding license. A materialized view currently uses one persisted Fuse table with `CHANGE_TRACKING` enabled as its source. Arbitrary multi-table query definitions are not supported. ### Controlling refresh work in batches September added a batch-size hint for refresh operations: ```SQL REFRESH MATERIALIZED VIEW mv_sales LIMIT 100000; ``` `LIMIT` is an approximate row-count hint that consumes changes at complete source-snapshot boundaries. It is not a strict row limit. If one source commit exceeds the requested size, Databend may consume the whole commit. The limit may also be ignored when a change range contains `UPDATE` or `DELETE` operations, or when suitable count information is unavailable. The feature is primarily intended for batched maintenance of append-oriented workloads. ### When a Dynamic Table is a better fit This release adds Dynamic Tables with full refresh. A Dynamic Table uses a separate Fuse-based table engine and persists results derived from multiple tables. `REFRESH DYNAMIC TABLE` explicitly recomputes the result, which makes it suitable for queries such as multi-table joins. A query against a Dynamic Table reads the result of its most recent refresh, so the data may lag behind its sources. The current implementation does not include automatic scheduling, incremental refresh, or the freshness compensation available through materialized-view Hybrid Reads. Use a materialized view for reusable single-table results and optimizer rewrites. A Dynamic Table fits multi-table derived results when an explicit full-refresh interval is acceptable. ## Data Sharing丨live access across tenants Data Sharing lets isolated tenants work from the provider's source data while keeping separate data-processing environments. In this context, a tenant is a Databend data and permission boundary. It may represent an independently managed business unit or a customer of a platform service. Without Data Sharing, cross-tenant consumption often requires exports, copies, and synchronization jobs. A provider can instead add selected Fuse tables to a Share and grant access to a consumer tenant. The consumer creates a local read-only database binding and queries the shared objects without first copying the complete source table. ### From provider authorization to consumer access A Share exposes one database and selected Fuse tables within it. The provider controls the shared objects and tenant grants. The consumer accesses the data through a local database name. A Provider Connection resolves storage access without copying provider credentials into the consumer's database binding. Replace ``, ``, and `` with names from your environment: ```SQL -- Provider: create a Share and grant access to the database, table, and consumer CREATE SHARE sales_share CONNECTION = ; GRANT USAGE ON DATABASE sales TO SHARE sales_share; GRANT SELECT ON TABLE sales.orders TO SHARE sales_share; ALTER SHARE sales_share ADD ACCOUNTS = ; -- Consumer: create a local read-only database binding CREATE DATABASE shared_sales FROM SHARE .sales_share; SELECT * FROM shared_sales.orders LIMIT 10; ``` The sharing relationship uses stable Share, database, and table IDs. Renaming an object preserves access to that object. Dropping a table and creating another table with the same name does not transfer the old grant. The provider can stop future access by revoking the table grant or removing the consumer account. ### Independent Stream consumption September also added consumer-local Streams on shared tables. Once the provider enables `CHANGE_TRACKING`, each consumer can maintain its own Stream position and feed changes from the shared source into a local incremental pipeline. Consumers can process the same shared business data on different schedules without sharing progress. A Stream records consumption state; it does not run the downstream synchronization or transformation job. ### Scope and limitations - Sharing works across tenants in the same Databend Meta cluster. It does not connect arbitrary independent deployments across clusters. - A Share currently exposes selected Fuse tables. Consumers have read-only access and cannot modify the source table or reshare the database. - Provider tables with row-access or column-masking policies cannot currently be shared directly across tenants. - Streams require change tracking on the provider table. Dropping and recreating the consumer's shared database invalidates its existing Streams. - Data Sharing is an Enterprise Edition feature and requires the `DataSharing` license. ## Data lineage: tracing sources and downstream impact Databend extracts object and column data flows from query plans, then aggregates successful query history to build lineage relationships. You can use the result to trace a dataset to its sources or inspect downstream dependencies before changing a table or column. Two table functions provide access to the lineage graph: - `GET_LINEAGE` traverses upstream sources or downstream consumers at object or column level, with a distance from one to five hops. - `GET_LINEAGE_NEIGHBORS` returns directly connected objects and is useful when expanding a lineage graph one level at a time. ```SQL -- Find upstream objects for a result table SELECT * FROM get_lineage('default.analytics.sales_summary', 'TABLE', 'UPSTREAM', 3); -- Find directly connected objects SELECT * FROM get_lineage_neighbors('default.analytics.sales_summary', 'TABLE'); ``` Relationships use a consistent `source -> target` direction. A relationship can include the latest relevant `query_id`, SQL text, duration, row count, and other query details, which links a dependency to the operation that produced it. The release also adds lineage refresh for existing views, extraction from explicitly materialized CTEs, materialized-view lineage, and relationship capture from bound query plans. Lineage collection and aggregation are asynchronous and best effort. Databend does not backfill history from before lineage was enabled, and failures in logging or history processing can leave gaps. The result should not be treated as a complete audit record. Column lineage tracks columns that contribute to output values; it does not include every column used only for filtering, joins, or sorting. Check the history-table configuration and the SystemHistory license before enabling the feature. ## Scan and storage efficiency ### Fuse partition layout Fuse tables now support `PARTITION BY`. The table layout records the partition expression, exact partition values, and physical boundaries. Append writes, compaction, reclustering, and mutation paths maintain those boundaries so the scanner has more precise information for pruning. You can add a partition definition to an existing unpartitioned table with `ALTER TABLE ... PARTITION BY (...)`. This operation does not rewrite historical data. Older blocks without partition statistics remain readable, and queries scan them conservatively. Databend does not currently support arbitrary replacement of an existing partition key. Hash-distributed partition writes distribute partitioned writes by hash. Related fixes separate partition and clustering layouts and improve write merging. ### TopN scan pruning A fused TopN operator and runtime scan filters use the current sort boundary to filter data during scanning. The release also rechecks TopN boundaries while reading blocks, supports TopN pruning for typed virtual columns, and pushes down aggregate rank-limit boundaries. These changes target ordered `LIMIT` queries and ranking workloads by reducing reads and intermediate processing. The gain depends on data layout, predicates, and the final execution plan. Optimizer statistics now propagate column statistics through `UNION ALL` and include statistics for Sort and Window operators, typed histogram bounds, and selectivity estimates for Boolean expressions. ### Full-text search and multidimensional clustering - Inverted indexes add a Japanese tokenizer. - Index data is built with data blocks. Index-segment merging can reuse tokenization work and reduce object-storage requests. - The ngram bloom index has a more efficient build path. - Hilbert clustering adds metadata, statistics, and reclustering support through the current implementation. The legacy implementation removed in the previous period is not retained. - Reclustering can run concurrent tasks and has lower coordination and diagnostic overhead for sort and merge work. ## Reliability, governance, and operations ### Cancellation and resource cleanup Cancellation paths now interrupt synchronous spill processors and hash-join streams more reliably. Fixes cover task shutdown after a Flight exchange closes, hash-exchange blocking when a receiver exits early, and spill operations that remained blocked after downstream work completed. These cases occur when a client disconnects, a query is cancelled, or a consumer stops after reading only part of a result. ### Memory use and execution correctness Memory-related changes tighten the boundary at final aggregation, allow Window output to spill, remove unused columns before Window buffering, limit Parquet row-group bytes, allocate decode buffers by selected-row count, and stream Vacuum GC with root-node memory control. Correctness fixes cover nullable correlated and lateral join keys, consistent distributed time functions, session time zones for Window `RANGE INTERVAL`, date interval arithmetic, three-valued logic folding, and hashing and equality rules for floating-point zero and NaN. ### Derived data and history lifecycle Databend fixes correctness after change-tracking counters are reset and adds checks around materialized-view refresh and rewrite. Vacuum no longer removes active temporary tables or staging tables used by an in-progress CTAS operation. Time Travel now returns an explicit error when a requested snapshot is missing. Testing now includes a type matrix, metamorphic testing, and a table-kind by write-path matrix, increasing coverage across data types, table engines, and write paths. ### Row-level TTL metadata The release adds DDL and metadata support for defining, changing, and removing row-level TTL expressions. Background scheduling and expired-row deletion are not included at this stage. Defining a TTL expression does not automatically delete expired data. ### Meta communication, notifications, and tasks Databend Meta adds a Raft protocol secret, compatibility-mode support, Raft peer TLS, and configurable gRPC handshake authentication. Multiple credentials can overlap during rotation. Meta administration tools now use `--user` and `--password-file`, and command-line clients add gRPC TLS support. These options require certificate configuration, an authentication policy, and a planned upgrade order. Installing the new version does not automatically encrypt all Meta communication. `WEBHOOK_BODY_TEMPLATE` can format JSON payloads for services such as Lark, Slack, and DingTalk. Databend Query defines and validates the template; Cloud Control performs substitution and delivery. Confirm that the server-side service supports the feature before using it. Task definitions can now parse constant `EXECUTE IMMEDIATE` scripts at definition time, which helps with pipeline configuration and diagnostics. ### Client ecosystem `databend-jdbc` released `v0.4.9` and `v0.4.10` in August. Version `v0.4.9` preserves binary values and empty content in prepared values. Version `v0.4.10` updates Maven Central publishing credentials, manual-release tag constraints, and shaded-package references in tests. `bendsql` had no new GitHub Release during the period covered by this post. ## Where these updates fit | Scenario | Features to evaluate | Expected value and boundary | | ------------------------------------------------ | ------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Repeated queries and shared intermediate results | Materialized views for single-table results; Dynamic Tables for multi-table derived results | Reduce repeated scans and computation. Dynamic Tables use explicit full refresh and may lag behind their sources. | | Cross-tenant data collaboration | Data Sharing with consumer-local Streams | Reduce exports, duplicate copies, and synchronization jobs within one Meta cluster. Consumers still need downstream processing tasks. | | Incident tracing and change-impact analysis | Object-level and column-level lineage | Trace an unexpected metric to its sources and inspect affected downstream objects before a schema change. Lineage is best effort and is not a complete audit record. | | Query efficiency and runtime stability | Fuse partitioning, TopN pruning, indexes, resource cleanup, and memory controls | Reduce scanning and improve cleanup under large queries, cancellation, and early client disconnects. Results depend on data layout and execution plans. | Before adopting a feature, check the target Databend version, license, and documented limitations. Validate performance changes against production-shaped workloads. ## Upgrade notes This release period removes several capabilities. Check existing deployments for dependencies before upgrading: - Aggregating Index: Databend is moving this use case to materialized views. Reassess definitions, licensing, and refresh strategy. Existing indexes are not migrated automatically. - PROXY table engine: materialized views replace the previous direction for this capability. - Hive Catalog: the catalog is removed as use of Hive tables and the surrounding ecosystem has declined. - Lance unload support: historical metadata remains decodable for compatibility, but access to data in Lance format returns an unsupported error. Move affected workflows to a supported format. - Read-only `ENGINE = DELTA` table engine: new table syntax is no longer supported, and existing tables using this engine fail after the upgrade. Review the Release Notes for the target version, plan the required migrations, and run compatibility tests before upgrading. Visit the [Databend GitHub repository](https://github.com/databendlabs/databend) and [Databend documentation](https://docs.databend.com) for Release Notes, feature details, and usage guidance.