This Week in Databend #124

Databend is a modern cloud data warehouse, serving your massive-scale analytics needs at low cost and complexity. Open source alternative to Snowflake. Also available in the cloud: https://app.databend.com .

What's New

Stay informed about the latest features of Databend.

Compatibility Added for Delta and Iceberg Table Engines

Databend now supports Delta and Iceberg table engines, enabling seamless integration of tables managed across diverse data lakes. This example uses the Delta table engine and a CONNECTION to create a Delta table stored on S3:

create connection my_s3_conn storage_type = 's3' access_key_id ='minioadmin' secret_access_key ='minioadmin' ENDPOINT_URL='http://127.0.0.1:9900';

create table test_delta engine = delta location = 's3://testbucket/admin/data/delta/delta-table/' connection_name = 'my_s3_conn'

Currently, Databend is limited to basic read operations when interacting with tables created using the Delta table engine.

If you would like to learn more, please contact the Databend team or refer to the resources listed below:

Code Corner

Discover some fascinating code snippets or projects that showcase our work or learning journey.

ClickHouse HTTP Handler Disabled by Default

As the ClickHouse HTTP Handler is now disabled by default in Databend, a new setting, enable_clickhouse_handler, has been introduced to control its activation, with a default value of 0.

If you need to use the ClickHouse HTTP Handler, execute the following command to enable it:

set global enable_clickhouse_handler=1;

PR #13965 | feat: add setting to disable clickhouse handler

Replacing Bincode with Borsh

Due to potential binary inconsistencies with Bincode across different versions and usage scenarios, Databend decided to replace Bincode with Borsh for serializing aggregate function states.

Borsh has a format that is almost identical to the format used by Databend for aggregate states. It provides consistency guarantees, follows safe coding practices, and has clear format specifications and good performance.

PR #13997 | refactor: using borsh instead of bincode for serde agg function state

Highlights

We have also made these improvements to Databend that we hope you will find helpful:

Added support for spilling Sort when reaching memory limits.
Added support for internal stream columns.
Added Support for SHOW LOCKS.
Read the documentation Docs | system.streams and Docs | system.stream_status to learn how to use system tables for obtaining stream-related information.

What's Up Next

We're always open to cutting-edge technologies and innovative ideas. You're more than welcome to join the community and bring them to Databend.

Introducing Password Policy

Databend is introducing password policy in response to user security needs. This will allow the implementation of access denials following incorrect password attempts and the enforcement of password complexity requirements.

CREATE PASSWORD POLICY [ IF NOT EXISTS ] <name>
  [ PASSWORD_MIN_LENGTH = <integer> ]
  [ PASSWORD_MAX_LENGTH = <integer> ]
  [ PASSWORD_MIN_UPPER_CASE_CHARS = <integer> ]
  [ PASSWORD_MIN_LOWER_CASE_CHARS = <integer> ]
  [ PASSWORD_MIN_NUMERIC_CHARS = <integer> ]
  [ PASSWORD_MIN_SPECIAL_CHARS = <integer> ]
  [ PASSWORD_MIN_AGE_DAYS = <integer> ]
  [ PASSWORD_MAX_AGE_DAYS = <integer> ]
  [ PASSWORD_MAX_RETRIES = <integer> ]
  [ PASSWORD_LOCKOUT_TIME_MINS = <integer> ]
  [ PASSWORD_HISTORY = <integer> ]
  [ COMMENT = '<string_literal>' ]

Issue #13994 | feat: PASSWORD POLICY

Please let us know if you're interested in contributing to this feature, or pick up a good first issue at https://link.databend.rs/i-m-feeling-lucky to get started.