Blog

This Week in Databend #112

PsiACESep 24, 2023

Databend is a modern cloud data warehouse, serving your massive-scale analytics needs at low cost and complexity. Open source alternative to Snowflake. Also available in the cloud: https://app.databend.com .

What's On In Databend

Stay connected with the latest news about Databend.

Understanding User-Defined Function

User-Defined Functions (UDFs) enable you to define their own custom operations to process data within Databend, enabling better data handling, task execution, and the creation of more efficient data workflows.

User-Defined Functions (UDFs) are typically written using lambda expressions or implemented via a UDF server with programming languages such as Python and are executed as part of Databend's query processing pipeline. Advantages of using UDFs include:

  • Customized Data Transformations.
  • Performance Optimization.
  • Code Reusability.

If you are interested in learning more, please check out the resources listed below.

Code Corner

Discover some fascinating code snippets or projects that showcase our work or learning journey.

Refactoring Databend Metrics Component with Prometheus

metrics-rs is a general abstraction layer across different metrics solutions like statsd, prometheus, new-relic, etc. However, metrics-rs is not well-suited for measuring metrics about histogram.

Nowadays prometheus is the de facto standard in the metrics area. Using a raw Prometheus client without pushing metrics to other systems like StatsD offers several benefits:

  • Improved Performance: By adhering to best practices for metric handling, memory allocation for metrics can be optimized to achieve O(1) complexity, eliminating the need for local buffered queues.
  • Minimizing Abstraction Layers: Streamlining metric handling helps reduce unnecessary abstraction layers, simplifying the code path for improved clarity and comprehension.
  • Improved Coding Standards: Addressing the existing variance in metric practices across different modules by adopting Prometheus community standards can help establish a more consistent and cohesive approach to metrics within our codebase.

Now, Databend's observability metrics have been fully migrated to Prometheus implementation. This brings you a more comprehensive and reliable observability experience while keeping the original metrics almost unchanged.

If you are interested in learning more, please check out the resources listed below:

Highlights

We have also made these improvements to Databend that we hope you will find helpful:

  • Added support for
    COMPACT
    distributed execution.
  • Added the
    json_path_exists
    function.
  • Added the
    recluster_block_size
    setting to control the block size during re-clustering.
  • Added support for conversion from
    DECIMAL
    type to
    INT
    type.
  • Added support for an inverted filter to optimize filter execution, leading to a performance boost of up to 4 times in certain scenarios.
  • SQLSmith testing now supports generating table functions, window functions, subqueries, and the
    WITH
    clause.

What's Up Next

We're always open to cutting-edge technologies and innovative ideas. You're more than welcome to join the community and bring them to Databend.

Implementing the
GREATEST
Function

The

GREATEST
function returns the maximum value in a list of expressions.

Syntax:

GREATEST( <expr1> [ , <expr2> ... ] )

Example:

SELECT id, name, category, price, rating,
CASE
WHEN rating = 1 THEN 0.02
WHEN rating = 2 THEN 0.04
WHEN rating = 3 THEN 0.06
WHEN rating = 4 THEN 0.08
ELSE 0.1
END AS increase_percentage_based_on_rating,
rank() OVER (PARTITION BY category ORDER BY rating) AS rating_rank,
CASE
WHEN rating_rank = 1 THEN 0.2
WHEN rating_rank = 2 THEN 0.1
ELSE 0
END AS increase_percentage_based_on_rank,
GREATEST(increase_percentage_based_on_rating,
increase_percentage_based_on_rank) AS final_increase_percentage,
CAST(price * (1 + final_increase_percentage) AS DECIMAL(10, 2))
AS adjusted_price
FROM products

Issue #12944 | feat: GREATEST function

Please let us know if you're interested in contributing to this feature, or pick up a good first issue at https://link.databend.com/i-m-feeling-lucky to get started.

Changelog

You can check the changelog of Databend Nightly for details about our latest developments.

Full Changelog: https://github.com/datafuselabs/databend/compare/v1.2.116-nightly...v1.2.128-nightly

Share this post

Subscribe to our newsletter

Stay informed on feature releases, product roadmap, support, and cloud offerings!