This Week in Databend #121

Databend is a modern cloud data warehouse, serving your massive-scale analytics needs at low cost and complexity. Open source alternative to Snowflake. Also available in the cloud: https://app.databend.com .

What's New

Stay informed about the latest features of Databend.

Supporting Append-only Stream

A Databend stream currently supports Append-only mode, allowing it to capture data insertions. This encompasses potential data deletions or updates for the recently added rows. However, the stream consistently reflects the final changes to the table since its creation. For example, if a row is added and subsequently updated with new values, the stream will accurately describe the changes based on the updated values.

CREATE STREAM [IF NOT EXISTS] [<database>.]<stream> 
  ON TABLE [<database>.]<table> 
    [<stream_point>] [COMMENT = '<string_literal>']

Here is a simple example:

databend> create table t(a int);
Query OK, 0 rows affected (0.07 sec)

databend> insert into t values(1);
Query OK, 1 row affected (0.06 sec)

databend> alter table t set options(change_tracking=true);
Query OK, 0 rows affected (0.06 sec)

databend> create stream s on table t;
Query OK, 0 rows affected (0.04 sec)

databend> insert into t values(2);
Query OK, 1 row affected (0.07 sec)

databend> select * from s;
+------+
| a    |
+------+
|    2 |
+------+
1 row in set (0.07 sec)
Read 2 rows, 59.00 B in 0.032 sec., 61.82 rows/sec., 1.78 KiB/sec.

If you want to learn more, please feel free to contact the Databend team, or check out the resources listed below.

Code Corner

Discover some fascinating code snippets or projects that showcase our work or learning journey.

Switching from parquet2 to parquet-rs

Due to issues with bug fixes and long-term maintenance for parquet2, Databend has switched the default Parquet Reader to arrow-rs/parquet.

This change, effective from v1.2.223-nightly, helps improve the stability of Parquet file reading and writing. Currently, Databend's Fuse engine still uses parquet2.

PR #13754 | feat: change default value of use_parquet2 to 0

Highlights

We have also made these improvements to Databend that we hope you will find helpful:

Introduced a new time function: current_timestamp.
Added support for new array functions: array_to_string and array_flatten.
Introduced a new table function called stream_status and enhanced the functionality of the existing fuse_encoding table function.
Added support for User-Defined Function (UDF) authorization management.

What's Up Next

We're always open to cutting-edge technologies and innovative ideas. You're more than welcome to join the community and bring them to Databend.

Supporting Iceberg and Delta Tables with Table Engine

Iceberg tables currently work with Iceberg directories only. By separating tables from directories, we could experiment with creating tables compatible with Iceberg or Delta engines using the specified syntax:

create table <ident> from <location> engine=iceberg|delta [connection_name="my_connection", <other args>]

Issue #13787 | support iceberg and delta with table engine

Please let us know if you're interested in contributing to this feature, or pick up a good first issue at https://link.databend.rs/i-m-feeling-lucky to get started.