This Week in Databend #129

Databend is a modern cloud data warehouse, serving your massive-scale analytics needs at low cost and complexity. Open source alternative to Snowflake. Also available in the cloud: https://app.databend.com .

What's New

Stay informed about the latest features of Databend.

Supporting Standard Streams

Standard Streams are capable of capturing all types of data changes, including updates and deletions. Compared to append-only streams, standard streams are more comprehensive and suitable for more complex data change scenarios.

The syntax for CREATE STREAM also supports options to control the type of stream.

CREATE STREAM [IF NOT EXISTS] [<database_name>.]<stream_name>
  ON TABLE [<database_name>.]<table_name>
    [AT (STREAM => <stream_name>)]
    [append_only = true|false]
    [COMMENT = '<comment>']

If append_only=true, it creates an append-only stream, which is used only for appending data and does not track updates or deletions.
If append_only=false, it creates a standard stream. It is capable of capturing all types of data changes, including updates and deletions, providing comprehensive data modification tracking.

If you would like to learn more, please contact the Databend team or refer to the resources listed below:

PR #14272 | feat: standard stream

Code Corner

Discover some fascinating code snippets or projects that showcase our work or learning journey.

Implementing a Raft Consensus Algorithm with just 300 Lines

Databend's metadata service uses the Raft consensus algorithm and has open-sourced and maintains openraft, an asynchronous Raft implementation based on tokio.

one_file_raft is a concise demonstrative implementation that realizes the Raft consensus algorithm in a single Rust file with roughly 300 lines of effective code.

Created by the main maintainer of openraft, drmingdrmer, it serves to demonstrate the core principles of the distributed consensus protocol without the complexity of application-specific business logic, edge case management, and error handling. This implementation focuses on the basic features of Raft, such as leader election, log replication, and log commitment, while omitting advanced features like log compaction and log cleaning.

GitHub - drmingdrmer/one_file_raft

Highlights

We have also made these improvements to Databend that we hope you will find helpful:

Added support for Query-level cache statistics.
Implemented cost-based optimization for distributed queries.
Added date-related function MONTHS_BETWEEN.
Added Task DAG related functions TASK_DEPENDENTS and TASK_DEPENDENTS_ENABLE.
Added v1/system API to display system information.

What's Up Next

We're always open to cutting-edge technologies and innovative ideas. You're more than welcome to join the community and bring them to Databend.

Tenant Data Migration Tool

databend-metactl --export supports exporting all data from the metadata service at once. However, there is still a lack of tools for migrating exact data belonging to a specific tenant.

To further meet the metadata migration needs in multi-tenant scenarios, Databend plans to design and implement a dedicated tool. This tool will export and decode data from the metadata service and filter out unnecessary data.

Issue #14405 | Tracking issue: dump data belonging to a tenant from meta-service

Please let us know if you're interested in contributing to this feature, or pick up a good first issue at https://link.databend.rs/i-m-feeling-lucky to get started.