--- title: 'This Week in Databend #82' date: 2023-02-22 slug: category-weekly/2023-02-22-databend-weekly tags: [weekly] category: Weekly authors: - PsiACE --- [Databend](https://github.com/databendlabs/databend) is a modern cloud data warehouse, serving your massive-scale analytics needs at low cost and complexity. Open source alternative to Snowflake. Also available in the cloud: [https://app.databend.com](https://app.databend.com) . {/* truncate */} ## What's New Check out what we've done this week to make Databend even better for you. ### Features & Improvements :sparkles: **AST** - select from stage support uri with connection options ([#10066](https://github.com/databendlabs/databend/pull/10066)) **Catalog** - Iceberg/create-catalog ([#9017](https://github.com/databendlabs/databend/pull/9017)) **Expression** - type decimal support agg func min/max ([#10085](https://github.com/databendlabs/databend/pull/10085)) - add sum/avg for decimal types ([#10059](https://github.com/databendlabs/databend/pull/10059)) **Pipeline** - enrich core pipelines processors ([#10098](https://github.com/databendlabs/databend/pull/10098)) **Query** - create stage, select stage, copy, infer_schema support named file format ([#10084](https://github.com/databendlabs/databend/pull/10084)) - query result cache ([#10042](https://github.com/databendlabs/databend/pull/10042)) **Storage** - table data cache ([#9772](https://github.com/databendlabs/databend/pull/9772)) - use `drop_table_by_id` api in `drop all` ([#10054](https://github.com/databendlabs/databend/pull/10054)) - native storage format support nested data types ([#9798](https://github.com/databendlabs/databend/pull/9798)) ### Code Refactoring :tada: **Meta** - add compatible layer for upgrade ([#10082](https://github.com/databendlabs/databend/pull/10082)) - More elegant error handling ([#10112](https://github.com/databendlabs/databend/pull/10112), [#10114](https://github.com/databendlabs/databend/pull/10114), etc.) **Cluster** - support exchange sorting ([#10149](https://github.com/databendlabs/databend/pull/10149)) **Executor** - add check processor graph completed ([#10166](https://github.com/databendlabs/databend/pull/10166)) **Planner** - apply constant folder at physical plan builder ([#9889](https://github.com/databendlabs/databend/pull/9889)) **Query** - use accumulating to impl single state aggregator ([#10125](https://github.com/databendlabs/databend/pull/10125)) **Storage** - adopt OpenDAL's batch delete support ([#10150](https://github.com/databendlabs/databend/pull/10150)) - adopt OpenDAL query based metadata cache ([#10162](https://github.com/databendlabs/databend/pull/10162)) ### Build/Testing/CI Infra Changes :electric_plug: - release deb repository ([#10080](https://github.com/databendlabs/databend/pull/10080)) - release with systemd units ([#10145](https://github.com/databendlabs/databend/pull/10145)) ### Bug Fixes :wrench: **Expression** - no longer return Variant as common super type ([#9961](https://github.com/databendlabs/databend/pull/9961)) - allow auto cast from string and variant ([#10111](https://github.com/databendlabs/databend/pull/10111)) **Cluster** - fix limit query hang in cluster mode ([#10006](https://github.com/databendlabs/databend/pull/10006)) **Storage** - wrong column statistics when contain tuple type ([#10068](https://github.com/databendlabs/databend/pull/10068)) - compact not work as expected with add column ([#10070](https://github.com/databendlabs/databend/pull/10070)) - fix add column min/max stat bug ([#10137](https://github.com/databendlabs/databend/pull/10137)) ## What's On In Databend Stay connected with the latest news about Databend. ### Query Result Cache In the past week, Databend now supports caching of query results! ``` ┌─────────┐ 1 ┌─────────┐ 1 │ ├───►│ ├───►Dummy───►Downstream Upstream────►│Duplicate│ 2 │ │ 3 │ ├───►│ ├───►Dummy───►Downstream └─────────┘ │ │ │ Shuffle │ ┌─────────┐ 3 │ │ 2 ┌─────────┐ │ ├───►│ ├───►│ Write │ Upstream────►│Duplicate│ 4 │ │ 4 │ Result │ │ ├───►│ ├───►│ Cache │ └─────────┘ └─────────┘ └─────────┘ ``` **Learn More** - [PR | feat(query): query result cache](https://github.com/databendlabs/databend/pull/10042) - [Docs | RFC: Query Result Cache](https://docs.databend.com/developer/community/rfcs/query-result-cache) - [Tracking Issue | RFC: query result cache](https://github.com/databendlabs/databend/issues/10011) ### Table Data Cache Databend now supports table data cache: - disk cache: raw column(compressed) data of the data block. - in-memory cache(experimental): deserialized column objects of a data block. For cache-friendly workloads, the performance gains are significant. **Learn More** - [PR | feat: table data cache](https://github.com/databendlabs/databend/pull/9772) - [Docs | Query Server Configuration - Cache](https://docs.databend.com/guides/deploy/references/node-config/query-config#cache-section) ### Deb Source & Systemd Support Databend now offers the official Deb package source and supports the use of `systemd` to manage the service. For DEB822 Source Format: ```bash sudo curl -L -o /etc/apt/sources.list.d/databendlabs.sources https://repo.databend.com/deb/databendlabs.sources sudo apt update sudo apt install databend sudo systemctl start databend-meta sudo systemctl start databend-query ``` **Learn More** - [PR | chore(ci): release with systemd units](https://github.com/databendlabs/databend/pull/10145) - [Docs | Installing Databend - Package Manager](https://docs.databend.com/guides/deploy/deploy/download) ## What's Up Next We're always open to cutting-edge technologies and innovative ideas. You're more than welcome to join the community and bring them to Databend. ### Service Activation Progress Report When starting a Query/Meta node, it is necessary to perform checks and output them explicitly to help the user diagnose faults and confirm status. **Example:** ```bash storage check succeed meta check failed: timeout, no response. endpoints: xxxxxxxx . status check failed: address already in use. ``` [Issue 10193: Feature: output the necessary progress when starting a query/meta node](https://github.com/databendlabs/databend/issues/10193) Please let us know if you're interested in contributing to this issue, or pick up a good first issue at [https://link.databend.com/i-m-feeling-lucky](https://link.databend.com/i-m-feeling-lucky) to get started. ## Changelog You can check the changelog of Databend Nightly for details about our latest developments. - [v0.9.47-nightly](https://github.com/databendlabs/databend/releases/tag/v0.9.47-nightly) - [v0.9.46-nightly](https://github.com/databendlabs/databend/releases/tag/v0.9.46-nightly) - [v0.9.45-nightly](https://github.com/databendlabs/databend/releases/tag/v0.9.45-nightly) - [v0.9.44-nightly](https://github.com/databendlabs/databend/releases/tag/v0.9.44-nightly) - [v0.9.43-nightly](https://github.com/databendlabs/databend/releases/tag/v0.9.43-nightly) - [v0.9.42-nightly](https://github.com/databendlabs/databend/releases/tag/v0.9.42-nightly) - [v0.9.41-nightly](https://github.com/databendlabs/databend/releases/tag/v0.9.41-nightly) - [v0.9.40-nightly](https://github.com/databendlabs/databend/releases/tag/v0.9.40-nightly) ## Contributors Thanks a lot to the contributors for their excellent work this week. | [andylokandy](https://github.com/andylokandy) | [ariesdevil](https://github.com/ariesdevil) | [b41sh](https://github.com/b41sh) | [Big-Wuu](https://github.com/Big-Wuu) | [BohuTANG](https://github.com/BohuTANG) | [cameronbraid](https://github.com/cameronbraid) | | :-------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------: | | [andylokandy](https://github.com/andylokandy) | [ariesdevil](https://github.com/ariesdevil) | [b41sh](https://github.com/b41sh) | [Big-Wuu](https://github.com/Big-Wuu) | [BohuTANG](https://github.com/BohuTANG) | [cameronbraid](https://github.com/cameronbraid) | | [Chasen-Zhang](https://github.com/Chasen-Zhang) | [ClSlaid](https://github.com/ClSlaid) | [dantengsky](https://github.com/dantengsky) | [drmingdrmer](https://github.com/drmingdrmer) | [everpcpc](https://github.com/everpcpc) | [johnhaxx7](https://github.com/johnhaxx7) | | :----------------------------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------: | | [Chasen-Zhang](https://github.com/Chasen-Zhang) | [ClSlaid](https://github.com/ClSlaid) | [dantengsky](https://github.com/dantengsky) | [drmingdrmer](https://github.com/drmingdrmer) | [everpcpc](https://github.com/everpcpc) | [johnhaxx7](https://github.com/johnhaxx7) | | [lichuang](https://github.com/lichuang) | [mergify[bot]](https://github.com/apps/mergify) | [PsiACE](https://github.com/PsiACE) | [RinChanNOWWW](https://github.com/RinChanNOWWW) | [soyeric128](https://github.com/soyeric128) | [sundy-li](https://github.com/sundy-li) | | :-------------------------------------------------------------------------------------------------------------------------------: | :--------------------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------------: | :-------------------------------------------------------------------------------------------------------------------------------: | | [lichuang](https://github.com/lichuang) | [mergify[bot]](https://github.com/apps/mergify) | [PsiACE](https://github.com/PsiACE) | [RinChanNOWWW](https://github.com/RinChanNOWWW) | [soyeric128](https://github.com/soyeric128) | [sundy-li](https://github.com/sundy-li) | | [suyanhanx](https://github.com/suyanhanx) | [TCeason](https://github.com/TCeason) | [Xuanwo](https://github.com/Xuanwo) | [xudong963](https://github.com/xudong963) | [youngsofun](https://github.com/youngsofun) | [zhang2014](https://github.com/zhang2014) | | :----------------------------------------------------------------------------------------------------------------------------------: | :------------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------: | :----------------------------------------------------------------------------------------------------------------------------------: | :-----------------------------------------------------------------------------------------------------------------------------------: | :---------------------------------------------------------------------------------------------------------------------------------: | | [suyanhanx](https://github.com/suyanhanx) | [TCeason](https://github.com/TCeason) | [Xuanwo](https://github.com/Xuanwo) | [xudong963](https://github.com/xudong963) | [youngsofun](https://github.com/youngsofun) | [zhang2014](https://github.com/zhang2014) | | [zhyass](https://github.com/zhyass) | | :----------------------------------------------------------------------------------------------------------------------------: | | [zhyass](https://github.com/zhyass) |