---
title: 'Native data lineage in Databend: Tracing metric sources and assessing change impact'
date: 2026-09-17
slug: category-engineering/databend-native-data-lineage
cover_url: 'https://static.databend.cn/img/202609/databend-data-lineage-in-action-1080x460-v5.png'
description: "Available since Databend v1.2.935, Databend's native Data Lineage feature uses query plans to capture object-level and column-level dependencies directly inside the database. SQL queries and the Databend Cloud lineage graph provide two ways to inspect these relationships for change impact analysis, troubleshooting, data asset governance, and sensitive data tracking."
tags: [databend]
category: Engineering
authors:
  - youngsofun
---

**TL;DR:** Available since Databend v1.2.935, Databend's native Data Lineage feature uses query plans to capture object-level and column-level dependencies directly inside the database. SQL queries and the Databend Cloud lineage graph provide two ways to inspect these relationships for change impact analysis, troubleshooting, data asset governance, and sensitive data tracking.

## Following data dependencies

A schema change can affect downstream views, processing jobs, and reports. An unexpected metric may require tracing source data and transformation logic. Sensitive fields also need to be tracked as they move through a pipeline.

{/* truncate */}

In a small environment, engineers can often answer these questions through experience, SQL searches, and manual investigation. As tables, views, stages, and pipelines form a larger dependency network, an inventory of objects alone is insufficient. Engineers also need to understand how data moves between them.

Data lineage describes that path through data creation, transformation, and consumption. Upstream lineage identifies an object's sources: a customer sales summary, for example, may depend on order details and a customer dimension table. Downstream lineage identifies where an object's data is used, such as the aggregate tables, views, and processing jobs that depend on an orders table.

Before changing a column type, dropping a column, modifying a schema, or restructuring a data model, downstream lineage helps identify the potential impact. It reduces manual dependency checks and the risk of reports or jobs failing after a change. When report values are incorrect, column-level lineage provides a path through source data, transformations, and downstream usage, narrowing the investigation compared with searching SQL and job logs individually.

---

## What Databend records

### Object and column dependencies

The lineage model contains tables, views, stages, and external catalog objects as nodes. Edges describe data writes and view-definition dependencies. Table-level lineage captures relationships between objects; column-level lineage maps source columns to the target columns they produce.

Column mappings cover aggregates, computed expressions, aliases, and nested queries. If several source columns contribute to a target column, each becomes an upstream dependency. These relationships can be followed through intermediate tables or views, as the customer segmentation example below illustrates.

Databend records two kinds of relationship:

- Data Movement captures data written from one object to another through `CREATE TABLE ... AS SELECT`, `INSERT ... SELECT`, multi-table `INSERT`, `REPLACE`, `MERGE`, and `COPY`.

- View Lineage captures dependencies between a view definition and its source objects.

### Streams, Stages, and external Catalogs

Reads through a Stream resolve to its underlying table in the lineage model. Stages participate in object-level lineage for data loaded into tables or unloaded from tables into a Stage. External Catalog objects can appear as endpoints connecting Databend objects with external data. Traversal stops at the external Catalog boundary.

---

## A customer segmentation example

### Enable lineage on a self-hosted deployment

Data Lineage is an enterprise feature. A self-hosted deployment requires an enterprise license. Add the following configuration to `databend-query.toml` on every Query node:

```Plaintext
[lineage]
on = true
```

### Create the processing chain

The example uses three tables. `fact_orders` stores order details. `agg_customer_sales` aggregates each customer's total spending, order count, and most recent order time. `customer_segments` assigns customer segments based on total spending.

```SQL
CREATE OR REPLACE DATABASE lineage_demo;

CREATE OR REPLACE TABLE lineage_demo.fact_orders (
    order_id BIGINT,
    customer_id BIGINT,
    amount DECIMAL(12, 2),
    order_time TIMESTAMP
);

CREATE OR REPLACE TABLE lineage_demo.agg_customer_sales AS
SELECT
    customer_id,
    sum(amount) AS total_amount,
    count(*) AS order_count,
    max(order_time) AS last_order_time
FROM lineage_demo.fact_orders
GROUP BY customer_id;

CREATE OR REPLACE TABLE lineage_demo.customer_segments AS
SELECT
    customer_id,
    total_amount,
    order_count,
    if(total_amount >= 1000, 'high_value', 'standard') AS segment,
    now() AS updated_at
FROM lineage_demo.agg_customer_sales;
```

After a query succeeds, Databend asynchronously writes its lineage events to the built-in history table. Under normal operating conditions, new relationships are typically queryable and visible within seconds, without waiting for an hourly or daily offline scan.

### Trace upstream objects

`GET_LINEAGE` accepts an object name, object type, traversal direction, and an optional hop count. Supported types are `TABLE`, `VIEW`, `STAGE`, and `COLUMN`. Use `UPSTREAM` to trace sources or `DOWNSTREAM` to find consumers. The hop count ranges from 1 to 5 and defaults to 5.

The following query starts at the customer segmentation table and requests two hops of upstream lineage:

```SQL
SELECT
    distance,
    source_object_database,
    source_object_name,
    target_object_database,
    target_object_name,
    process
FROM GET_LINEAGE(
    'lineage_demo.customer_segments',
    'TABLE',
    'UPSTREAM',
    2
)
ORDER BY distance;
```

`distance = 1` identifies a direct relationship. `distance = 2` identifies a second-hop relationship through an intermediate object. The `process` field is a JSON string containing query and execution context; fields within it may be empty depending on the relationship type and execution context.

### Assess a column's downstream impact

For column-level analysis, supply the fully qualified column name and set the object type to `COLUMN`:

```SQL
SELECT
    distance,
    source_object_name,
    source_column_name,
    target_object_name,
    target_column_name
FROM GET_LINEAGE(
    'lineage_demo.fact_orders.amount',
    'COLUMN',
    'DOWNSTREAM',
    5
)
ORDER BY distance, target_object_name, target_column_name;
```

This query identifies derived columns that may be affected by a change to `amount`.

---

## How native collection resolves dependencies

Databend resolves these dependencies from actual query plans according to SQL semantics, including joins across multiple tables. Relationships retain query and execution context so engineers can inspect their origins, verify transformation logic, and investigate data problems.

For Data Movement involving native Databend tables in the `default` Catalog, endpoints are identified by object IDs and column IDs. A lineage query resolves their current names from metadata. Renaming a table or column therefore preserves the existing relationship to that object without requiring name-based rematching or reconstruction of the processing chain. Stable identities reduce incorrect connections during ongoing lineage maintenance and keep relationships consistent as data models evolve.

---

## Backfill lineage for existing views

Views created after lineage is enabled have their dependencies recorded at creation time. For views that already exist, preview the changes with a dry run before refreshing:

```SQL
REFRESH LINEAGE FOR ALL VIEWS DRY RUN;
REFRESH LINEAGE FOR ALL VIEWS;
```

`REFRESH LINEAGE` reconciles lineage for all views in the `default` Catalog and requires the global `SUPER` privilege. The dry run reports relationships that would be added, updated, or deleted, without writing data.

---

## Inspect dependencies and query context in Databend Cloud

In Databend Cloud, open Database Explorer, select a table or view, and choose the Lineage tab. The feature is disabled by default; submit a support ticket to have Databend enable it.

Graph nodes represent tables or views, and edges show upstream and downstream relationships. Expand a node to inspect column dependencies. Clicking an edge between table nodes opens the query and execution context recorded when that relationship was created.

![](https://static.databend.cn/img/202609/d8b686ca-a066-4305-a0a1-53cea27b4e4a.png)

Column-level lineage also helps trace sensitive fields through transformations to their target columns. Combined with access controls and `Masking Policy`, these relationships support assessments of data exposure for security audits and compliance governance.

In environments with many tables and views, similar names can refer to data with different meanings. Lineage shows actual data flows, helping engineers identify important datasets, duplicate processing chains, and objects with no recorded downstream dependencies.

---

## Using dependencies in everyday investigations

Databend combines table-level dependencies, column mappings, and execution context in its lineage query results. The same relationships support tracing metric sources, assessing schema changes, and reviewing how data is used across a processing chain.

For configuration and usage details, see the [Databend Data Lineage documentation](https://docs.databend.com/guides/data-management/data-lineage).