What's Changed

Exciting New Features ✨

• feat: Add hdfs support in iceberg and fill iceberg statistics by @Xuanwo in #17352
• feat: support specifying columns to include in the "Attach table" statement by @dantengsky in #17442
• feat(metrics): add progress metrics via collector by @flaneur2020 in #17359
• feat(binder): support update from by @Dousir9 in #17464
• feat(query): add table stats api to admin interface by @everpcpc in #17471
• feat: Use max/min to estimate ndv by @Xuanwo in #17468
• feat: add table option copy_dedup_full_path by @youngsofun in #17473
• feat(query): add markov model function and feistel function for obfuscator by @forsaken628 in #17437
• feat(sqlsmith): fuzz test support more dbs as seed by @b41sh in #17489
• feat: SHOW CREATE TABLE add quote for ident by @KKould in #17505
• feat: object warehouse support rbac by @TCeason in #17262
• feat: (semi-)sync table/field comments with the table being attached by @dantengsky in #17478
• feat(raft-log): enhance raft-log reliability and add compatibility tests by @drmingdrmer in #17510
• feat: add log.file.max_size and rolling test by @BohuTANG in #17508
• feat: csv/tsv/ndjson support querying file meta data. by @youngsofun in #17512
• feat(query): support explain(DECORRELATED,verbose) by @forsaken628 in #17518
• feat(query): add show_roles table function by @TCeason in #17517
• feat: support query metadata from parquet. by @youngsofun in #17527
• feat: skip reading empty files when load/query location. by @youngsofun in #17522
• feat(base): enhanced query-level memory management by @zhang2014 in #17358
• feat(query): Supports folding of show grants results for roles and users by @TCeason in #17543
• feat(query): support query level spill setting by @zhang2014 in #17542
• feat: support WITHIN GROUP clause for AggregateFunction by @KKould in #17519
• feat(query): add catalog apis by @everpcpc in #17493
• feat: support load Avro Files. by @youngsofun in #17548
• feat(query): add api for users & roles by @everpcpc in #17553
• feat(query): support nondeterministic update by @forsaken628 in #17555
• feat: string_agg is compatible with multiple types as arguments instead of just String by @KKould in #17570
• feat: add table function fuse_vacuum2() by @SkyFan2002 in #16049
• feat: auto vacuum by @dantengsky in #17579
• feat(query): database dml for iceberg catalog by @sundy-li in #17578
• feat: UNPIVOT supports AS by @forsaken628 in #17595
• feat(query): bloom index support prune expr with simple casts by @forsaken628 in #17533
• feat(query): add catalog stats api by @everpcpc in #17602
• feat(query): support create/drop table under iceberg catalog by @TCeason in #17619
• feat(query): add iceberg table functions by @sundy-li in #17626
• feat: heartbeat to avoid query result timeout. by @youngsofun in #17624
• feat: Enable test and doctest for all crates by @Xuanwo in #17633
• feat: Add bendsave which can backup and restore databend data by @Xuanwo in #17503
• feat: impl function strip_null_value for Variant by @KKould in #17640
• feat(query): support left plan's from clause contains subquery by @forsaken628 in #17621
• feat: support specifying compression when unloading to parquet. by @youngsofun in #17664
• feat(meta): implement distributed semaphore based on meta-service by @drmingdrmer in #17651
• feat: add support for storing logs in table by @dqhl76 in #17598
• feat(config): warn unknown field instead of panic on start up by @flaneur2020 in #17504
• feat: support nextval as field default value. by @youngsofun in #17670
• feat: impl function: regexp_extract & regexp_extract_all by @KKould in #17658
• feat(query): support mul fro interval type with int types by @sundy-li in #17684
• feat: impl udf server add custom header by @KKould in #17691
• feat(query): support query iceberg table by version by @TCeason in #17693
• feat: hybrid cache by @dantengsky in #17704
• feat: introduce column oriented segment into by @SkyFan2002 in #17653
• feat: databend-metactl add command watch and upsert by @drmingdrmer in #17739
• feat: Support Displaying Runtime Filters in EXPLAIN Output by @SkyFan2002 in #17726
• feat(query): page index and range index support eliminate cast by @forsaken628 in #17740
• feat(optimizer): optimizer trace log by @BohuTANG in #17746
• feat(optimizer): add optimizer skip list by @BohuTANG in #17750
• feat: function extract, date_part,date_diff support more date part by @TCeason in #17759
• feat: table function that dumps snapshot information by @dantengsky in #17763
• feat(optimizer): SExpr visitor by @BohuTANG in #17768
• feat(meta-client): add cache crate for databend-meta service by @drmingdrmer in #17766
• feat(query): table_statistics(<database_name>[, table_name]) table functions by @BohuTANG in #17781
• feat(query): support cluster level concurrent limit by @zhang2014 in #17778
• feat(query): support date_between(date_part, startdate, enddate) by @TCeason in #17782
• feat: impl Create/Drop Index for Ngram Index by @KKould in #17789
• feat: block stream write by @zhyass in #17744
• feat: Bump opendal to 0.53.1 by @Xuanwo in #17817
• feat: hybrid column data cache by @dantengsky in #17771

Thoughtful Bug Fix 🔧

• fix(query): fix WindowPartitionTopNExchange painc with an empty block by @forsaken628 in #17453
• fix(parser): allow timestamp as a function name by @forsaken628 in #17455
• fix(query): fix order by derived column with limit return wrong values by @b41sh in #17457
• fix(query): fold constant subquery to build filter plan instead of join plan by @b41sh in #17448
• fix(query): fix left semi optimize to inner join by @sundy-li in #17458
• fix(query): fix bind internal column by @b41sh in #17463
• fix(query): spill config should be mask by @forsaken628 in #17467
• fix(query): fix variant get string function auto cast null to SQL NULL by @b41sh in #17466
• fix: InterpreterMetrics could subtract with overflow by @forsaken628 in #17461
• fix(query): fix stack overflow if excessive use of union all by @zhang2014 in #17475
• fix: check table mutability in copy into by @dantengsky in #17480
• fix: impl hive disable_table_info_refresh for hive to avoid error when show table status from by @BohuTANG in #17484
• fix(binder): AggregateRewriter should skip the rewrite of Pivot.aggregate by @forsaken628 in #17483
• fix(query): unsupport datetime format item should not return panic error by @TCeason in #17490
• fix(binder): different UDAF call must have unique display_name by @forsaken628 in #17500
• fix(query): tpcds q69 fail by @forsaken628 in #17516
• fix(query): fix display ast with quoted comments by @sundy-li in #17529
• fix: ensure result correctness when enable pruning cache by @dqhl76 in #17524
• fix(query): user/role name not support \b and \f by @TCeason in #17530
• fix(query): ident role_name not support ' " \b \f in parse by @TCeason in #17534
• fix: explain update sql panic by @zhyass in #17474
• fix(query): show roles rewrite kind is err by @TCeason in #17547
• fix: agg input does not match union output when subquery by @KKould in #17552
• fix: with recursive bind table panic by @KKould in #17554
• fix: parquet meta cache key include md5 or mtime or query_id. by @youngsofun in #17556
• fix(query): fix incorrect domain folder by @sundy-li in #17549
• fix: The update in MERGE INTO was mistakenly optimized as a delete by @SkyFan2002 in #17571
• fix(query): show catalog support iceberg engine by @TCeason in #17569
• fix: session leak when execute explain pipeline in cluster mode by @dqhl76 in #17572
• fix(storage): hilbert cluster information by @zhyass in #17577
• fix: tpcds Q69 fail by @forsaken628 in #17588
• fix(parser): allow reorder the task options in CREATE TASK/ALTER TASK by @flaneur2020 in #17537
• fix: parquet meta cache key should not use mtime. by @youngsofun in #17589
• fix(query): /v1/roles return all roles for account admin by @everpcpc in #17596
• fix(query): fix fold agg mismatch datatypes by @sundy-li in #17593
• fix: the output columns of cte related columns lack the columns required by cte itself by @KKould in #17576
• fix: tpcds spill test fail by @forsaken628 in #17597
• fix(query): fix comment display by @sundy-li in #17615
• fix(query): fix cast nested types cause panic by @b41sh in #17622
• fix(query): fix string view memory size calculate error by @b41sh in #17629
• fix: trim object preix error by @SkyFan2002 in #17636
• fix(query): to_date parse string with format should use UTC timezone by @TCeason in #17641
• fix(query): fix incorrect left mark join schema by @sundy-li in #17642
• fix: disk cache failed to build on macOS by @dantengsky in #17643
• fix(ddl): fix modify column data type null to not null failed by @b41sh in #17650
• fix(query): to_date(9999-12-31, format_string) should success by @TCeason in #17652
• fix(ci): flaky test by @zhyass in #17663
• fix: Enforce storage http client to use native tls by @Xuanwo in #17686
• fix(query): fix variant failed to successfully apply virtual column in bind join by @KKould in #17673
• fix: persistent log table may loss records when shutdown by @dqhl76 in #17695
• fix: flaky unit test test_persistent_log_write by @dqhl76 in #17701
• fix: DNS resolve error in some cases by @Xuanwo in #17699
• fix(query): broadcast subquery for markjoin if condition has null values by @sundy-li in #17706
• fix(query): process_like directly unwrap is not safe by @TCeason in #17722
• fix: flaky hybrid cache test by @dantengsky in #17735
• fix(query): ClientSessionManager::refresh_state subtracts with overflow if clock drift is present by @forsaken628 in #17742
• fix: align EXPLAIN output with actual partition pruning statistics by @dqhl76 in #17491
• fix: correct state type for Decimal256 AVG aggregation by @dantengsky in #17774
• fix(query): don't check file format in proto-conv by @sundy-li in #17793
• fix: Remove set nonblocking = false for mysql tcp stream by @Xuanwo in #17798
• fix(metrics): remove high cardinality labels in metrics by @flaneur2020 in #17820

Code Refactor 🎉

• refactor: s3 region auto detect error add timeout/endpoint by @BohuTANG in #17482
• refactor: re-generate hint file after alter table operations by @dantengsky in #17511
• refactor: remove timeout from snapshot hint refresh process by @dantengsky in #17532
• refactor(meta): simplify map key handling and improve naming consistency by @drmingdrmer in #17545
• refactor: add VirtualDataScheam to TableMeta by @KKould in #17591
• refactor: extract map-api and watcher to separate crates by @drmingdrmer in #17594
• refactor(meta): adopt external crate for shared state machine logic by @drmingdrmer in #17609
• refactor: move MatchSeq to external crate map-api by @drmingdrmer in #17610
• refactor(meta): upgrade OpenRaft to v0.10.0-alpha.9 by @drmingdrmer in #17612
• refactor(query): refactor json functions by @b41sh in #16840
• refactor(query): refactor test style by @sundy-li in #17630
• refactor: Improved Hilbert Clustering with Range Partition by @zhyass in #17424
• refactor: Unified query for hilbert and linear clustering information by @zhyass in #17618
• refactor: optimize file reading for disk-based cache on Unix by @dantengsky in #17638
• refactor(query): Optimizations for Variant type column filter by @b41sh in #17646
• refactor(query): remove background service by @zhang2014 in #17659
• refactor: use BlocksSerializer to replace StringBlock to simplify the serialization by @KKould in #17667
• refactor(optimizer): ir by @BohuTANG in #17685
• refactor(planner): refine the file organization by @BohuTANG in #17687
• refactor(optimizer): optimizer/operator by @BohuTANG in #17689
• refactor(optimizer): optimizers by @BohuTANG in #17692
• refactor(optimizer): optimizers by @BohuTANG in #17697
• refactor(query): rewrite function call expr to cast expr by @forsaken628 in #17669
• refactor(optimizer): optimize trait and pipeline by @BohuTANG in #17712
• refactor: enhance copy from avro. by @youngsofun in #17709
• refactor(optimizer): cascade by @BohuTANG in #17714
• refactor(query): Introduces dedicated structs for expression types by @forsaken628 in #17719
• refactor: extract crate databendlabs/display-more to display various types by @drmingdrmer in #17721
• refactor(optimizer): rules by @BohuTANG in #17727
• refactor(optimizer): move rule/utils by @BohuTANG in #17731
• refactor: use Weak ref to break cyclic references by @drmingdrmer in #17743
• refactor(optimizer): refine hyper_dp/dphyp.rs by @BohuTANG in #17753
• refactor(meta-client): Improve MetaHandshakeError by @drmingdrmer in #17755
• refactor: Bump arrow-udf-runtime so that we can use new pyo3 by @Xuanwo in #17758
• refactor: add helper methods to trait LicenseManager by @dantengsky in #17761
• refactor: reduce MetaClientError by @drmingdrmer in #17764
• refactor(optimizer): s_expr.rs by @BohuTANG in #17767
• refactor: DecimalSumState overflow handling in SUM aggregation by @dantengsky in #17773
• refactor(optimizer): refine the tpcds tests by @BohuTANG in #17777
• refactor(query): Combining TransformSortMergeLimit, TransformSortMerge, and TransformSortSpill by @forsaken628 in #17762
• refactor: add query_profile and query_details persistent log tables by @dqhl76 in #17702
• refactor: Bump opendal to 0.53 for better metrics by @Xuanwo in #17788
• refactor(fuse): runtime filter into two files: expr_bloom_filter.rs and expr_runtime_pruner.rs by @BohuTANG in #17804
• refactor: move segment generation into TableMutationAggregator for MERGE INTO by @zhyass in #17800
• refactor: upgrade min-meta-service-version to 1.2.677 by @drmingdrmer in #17783
• refactor(query): auto generate virtual columns for variant column by @b41sh in #17707
• refactor(query): remove unsafe implementation of Send and Sync for AcquireQueueGuard by @drmingdrmer in #17818

Build/Testing/CI Infra Changes 🔌

• ci: add workflow to publish deb repo by @everpcpc in #17485
• ci: fix download deb packages by @everpcpc in #17486
• ci: fix gpg key for deb distribution by @everpcpc in #17487
• ci: add view sqlsmith test by @TCeason in #17472
• ci: package meta & query deb separately by @everpcpc in #17488
• ci: fix pack binaries by @everpcpc in #17544
• ci: upgrade tj-actions/changed-files to v46 by @everpcpc in #17613
• ci: fix flaky ttc test by @forsaken628 in #17665
• ci: release target with hdfs by @everpcpc in #17671
• ci: fix local benchmark artifact by @everpcpc in #17674
• ci: add hadoop envs for query image by @everpcpc in #17715
• ci: tmp disable local benchmark by @everpcpc in #17717
• ci: fix local benchmark by @everpcpc in #17718
• ci: fix archive benchmark result by @everpcpc in #17754
• ci: fix test with ttc conainter by @everpcpc in #17769
• ci: tmp disable test 20_0025_stackoverflow by @everpcpc in #17770
• ci: fix start test container by @everpcpc in #17772
• ci: fix build feature for benchmark by @everpcpc in #17796
• ci: use larger runner for release by @everpcpc in #17799
• ci: fix ttc test by @everpcpc in #17807
• ci: fix check system info by @everpcpc in #17822

Others 📒

• chore(metrics): tune the histogram bucket for seconds/milliseconds by @flaneur2020 in #17197
• chore(ci): fix benchmark comment failure by @zhang2014 in #17470
• chore(query): add monotonicity function property by @sundy-li in #17469
• chore(query): add placeholder parser by @sundy-li in #17481
• chore(ci): fix benchmark comment failure by @zhang2014 in #17492
• chore: add more log info(like temp dir) for the vacuum temp files hook by @BohuTANG in #17494
• chore(query): continue the vacuum query hook even aborted by @sundy-li in #17497
• chore: Split metactl into lib meta/control and bin metactl by @Xuanwo in #17496
• chore: modify license url by @TCeason in #17513
• chore: refine some fuse_time_travel_size related logs by @dantengsky in #17515
• chore(query): allow arrow dictionary array into databend by @sundy-li in #17521
• chore(query): revert 17516 by @zhang2014 in #17539
• chore: rename EvalExpireTime to Expirable for better semantics by @drmingdrmer in #17540
• chore(query): siphash support generic types by @sundy-li in #17535
• chore(query): remove old sort spill by @forsaken628 in #17546
• chore: polish error message in CommitSink by @dantengsky in #17558
• chore: reorganize meta service watcher module by @drmingdrmer in #17557
• chore: add infer filter unit test by @BohuTANG in #17574
• chore(optimizer): deduplicate_join_condition unit tests by @BohuTANG in #17585
• chore: optimize sparse data read by @zhyass in #17581
• chore(optimizer): normalize_disjunctive_filter unit test by @BohuTANG in #17592
• chore(storage): add block file size by @zhyass in #17580
• chore(query): add udf script configs by @sundy-li in #17601
• chore(query): optimize show tables from iceberg catalog by @TCeason in #17603
• chore(meta): add meta-meta compatibility chart svg by @drmingdrmer in #17606
• chore(optimizer): add tpcds unit test for optimizer by @BohuTANG in #17605
• chore: update cast.rs by @eltociear in #17590
• chore(optimizer): snow_plan -> good_plan by @BohuTANG in #17611
• chore(cluster): remove useless debug log by @zhang2014 in #17614
• chore: optimize show tables with multi catalogs by @TCeason in #17632
• chore: optimize building bloom index for null by @zhyass in #17625
• chore: Add bendsave in release by @Xuanwo in #17644
• chore: Make sure bendsave is built while releasing by @Xuanwo in #17649
• chore: make error message when CTAS committing failed more user-friendly by @SkyFan2002 in #17645
• chore: calc rows per block for recluster by @zhyass in #17639
• chore: missing column index for output columns in explain verbose by @forsaken628 in #17657
• chore(query): check schema mismatch in projection render result by @sundy-li in #17654
• chore(query): add glob function by @sundy-li in #17666
• chore(optimizer): add rule/push_down_filter_join unit test by @BohuTANG in #17676
• chore: Always print detailed error from opendal by @Xuanwo in #17681
• chore: use default encoding of parquet columns instead of plain. by @youngsofun in #17688
• chore(query): Making Column Debug more readable by @forsaken628 in #17690
• chore(query): add cache id for distribute scheduler by @zhang2014 in #17708
• chore: optimize select name from system.tables by @TCeason in #17700
• chore(query): allow group by constant literal by @sundy-li in #17711
• chore: minor refine on RoleMgr by @drmingdrmer in #17683
• chore(query): Optimize cast expr error display by @forsaken628 in #17710
• chore: show file path when copy meet invalid parquet file. by @youngsofun in #17716
• chore: impl Display for WatchResponse and related pb types by @drmingdrmer in #17725
• chore: add local build databend package by @wubx in #17737
• chore(query): delete no useful function epoch(Int64Type)->IntervalType by @TCeason in #17730
• chore: use Usage privilege optimize USEDB Plan by @TCeason in #17679
• chore(logger): upgrade logforth to support global max_files limit for log_dir by @BohuTANG in #17637
• chore: increase the concurrency of fuse_time_travel_size by @SkyFan2002 in #17745
• chore: Using an ordered BTreeSet instead of a HashSet as a ColumnSet implementation by @forsaken628 in #17747
• chore: Bump rust toolchain by @Xuanwo in #17751
• chore: Bump Rust toolchain to nightly-2025-04-04 by @Xuanwo in #17752
• chore: show file path when copy decompress fail. by @youngsofun in #17732
• chore: check ee license for disk cache by @dantengsky in #17749
• chore: upgrade outdated dependencies by @forsaken628 in #17776
• chore(optimizer): spill out statistics yaml in tpcds_test by @BohuTANG in #17784
• chore(query): add iceberg table metadata cache by @sundy-li in #17780
• chore(ci): remove hdfs from docker image by @everpcpc in #17786
• chore: bump jsonb 0.5.0 by @b41sh in #17698
• chore: refine error handling in persistent log table by @dqhl76 in #17792
• chore(ci): change the sccache log level to debug by @forsaken628 in #17790
• chore: Extracting databend_common_version to help improve compilation cache hit rate by @forsaken628 in #17801
• chore: make databend_common_expression compilation cacheable by @forsaken628 in #17806
• chore(ci): add ci test_stateless_cluster_linux timeout minute to 18 by @TCeason in #17813

New Contributors

• @KKould made their first contribution in #17505
• @eltociear made their first contribution in #17590

Full Changelog: https://github.com/databendlabs/databend/releases/tag/v1.2.725