The old AST Visitor API was becoming hard to work with, so we wanted to move to a new API.

At first glance, this does not look like a difficult implementation problem. ASTs contain many node types, but much of the migration work follows repeated patterns. It is exactly the kind of structured, repetitive task where AI can generate a lot of useful code.

The real concern is different: after the code is generated, how do we know it did not quietly change SQL semantics?

In a database engine, the AST Visitor is not an isolated utility. It is infrastructure. It can be used by the parser, binder, optimizer, formatter, access control logic, expression rewrites, and other downstream components. A migration bug may not show up as a compiler error. It may show up later as:

• a node that is no longer visited;

• a field that is visited twice;

• a traversal order that used to be stable but is now different;

• an enter/exit relationship that no longer matches the old behavior;

• an early-return or skip path that behaves differently.

The code can look reasonable while the behavioral impact is deferred to a later stage of query processing.

This is why code review alone is a weak safety net. There are too many nodes to expand mentally. Generated code tends to look repetitive, which makes review fatigue worse. The right goal is to turn correctness into executable evidence rather than asking reviewers to manually verify every traversal.

The validation loop eventually looked like this:

AST source ─→ generated observation code ─────────────┐
                                                       │
SQL corpus ─→ parser ─→ AST ─→ old visitor ────────────┼→ trace ─→ comparator ─→ diff report
                              new visitor ────────────┘                         │
                                                                                 │
coverage analysis ←──────────────────────────────────────────────────────────────┘
    │
    └→ add missing SQL cases upstream

This system was not designed all at once. It grew from a smaller validation core, then expanded with real SQL inputs and coverage feedback.

Start From Source-Derived AST Metadata

The first requirement is a reliable view of the AST structure.

The wrong approach is to manually build another complex model of the AST inside the validation tool. If the goal is to check whether the visitor missed something, duplicating node classifications, recursive relationships, and special traversal rules only moves the audit problem into another codebase. Now the validator itself needs the same level of review as the migration.

The better approach is to extract structure as directly as possible from the source code.

Types, enum variants, structs, and fields are already defined in source. The scanner should mechanically expand them and produce metadata for later code generation. It should keep logic minimal:

• enumerate structs, enums, variants, and fields;

• preserve links back to source locations;

• represent one declaration as one structural record;

• list exclusions explicitly;

• avoid inferring traversal semantics;

• avoid copying special visitor ordering rules.

The trust model is simple: reviewers only need to audit a small set of generic extraction rules, not another hand-written AST model as large as the AST itself.

This metadata answers one question: what AST structure exists and should be considered? It does not answer how a visitor actually traverses that structure. That has to come from runtime observation.

Separate Observation From Equivalence

After collecting AST metadata, the next step is to observe actual visitor behavior.

The most important design decision is to separate observation from comparison.

Generated observation code should only produce facts. It records:

• which hook was entered;

• which node was seen;

• how many times an event occurred;

• where the event happened in the AST path;

• in what order the event occurred;

• whether control behavior such as skip or early return happened.

It should not decide whether an event is correct. It should not sort, deduplicate, normalize, or filter events during recording.

Equivalence should be handled by a separate comparator. The comparator reads the old and new visitor traces and checks them against explicit traversal contracts.

Observation layer                         Comparison layer

same AST ─→ old visitor ─→ old trace ─┐
                                      ├→ comparator ─→ diff report
         └→ new visitor ─→ new trace ─┘

This separation matters because it keeps generated code mechanical and keeps correctness rules concentrated in a smaller, auditable component.

The principle is:

Generate facts. Hand-write rules. Observe first, compare independently.

AI can help generate migration code and some observation code. But the rules that define correctness should stay explicit, centralized, and reviewable.

Define What Trace Equivalence Means

Once the observation layer can produce stable traces, the next question becomes concrete: what kind of trace difference means the visitor semantics changed?

The simplest idea is to compare the set of visited items. That catches missing nodes or hooks, but it is not enough.

A set loses count information. Visiting a node once and visiting it twice look identical in a set. To detect missing and duplicated visits, the comparator needs at least multiset semantics, or better, event records that preserve counts.

Order is another dimension. Some visitors only care whether a node was observed at all. Others rely on traversal order, even if that dependency is implicit. Examples include:

• seeing definitions before references;

• applying normalization in visit order;

• stopping at the first matching item;

• using nested enter and exit events to maintain state.

This means the comparator cannot use a single global rule for all hooks. It needs to check different properties depending on the semantics of each hook:

• whether the visited items are the same;

• whether event counts are the same;

• whether order-sensitive event sequences are the same;

• whether enter, exit, skip, recursion, and early-return behavior match.

At this point, “equivalent” becomes precise: the old and new implementations do not need to have identical code. They need to have no unexplained difference in the observable traversal behavior that the system depends on.

The old API is used as a historical baseline, but it should not be treated as inherently correct. If the new implementation intentionally fixes an old behavior, or if the new API intentionally changes a contract, that difference should be recorded explicitly. It should not be silently filtered out just to make the comparison pass.

Make Failures Directly Actionable

A comparator that only says “old trace and new trace differ” is not enough.

Without diagnostic detail, developers still need to inspect a large amount of visitor code to find the actual problem. The check is automated, but the repair loop remains expensive.

The trace protocol and comparator therefore need to carry diagnostic context:

• which input or test triggered the failure;

• which event is missing or duplicated;

• which hook changed order;

• the AST node path associated with the event;

• the first position where old and new traces diverged;

• a small amount of context before and after the divergence.

This requirement affects the observation layer. To report an AST path, the recorder must capture path information. To find the first order difference, the trace cannot be pre-sorted. To detect duplicate visits, the recorder cannot deduplicate events too early.

Failure diagnostics are not just presentation details at the end of the pipeline. They are part of the trace protocol.

At this point, the validation core is useful:

AST source ─→ generated observation code ─────────────┐
                                                       │
given AST ───────────────→ old visitor ────────────────┼→ trace ─→ independent comparator ─→ diff report
                          new visitor ────────────────┘

Given an AST, the system can run both visitors, compare observable behavior, and produce a diff that points directly to the migration issue.

The remaining problem is input coverage: which ASTs should we feed into this core?

Use Real SQL Corpora as Validation Input

A working validation core does not mean the migration is sufficiently tested. It only compares ASTs that are actually passed into it.

Manually constructing ASTs does not scale well:

• AST construction code is heavy;

• hand-written examples tend to cover only what the author remembered;

• edge cases in real SQL are easy to miss.

So we reused the parser’s existing SQL corpus:

1. parse existing SQL into ASTs;

2. run the old visitor on the same AST;

3. run the new visitor;

4. collect both traces;

5. compare the traces with the independent comparator.

Parser golden cases are a natural input source. A single SQL statement can cover multiple nodes and nested relationships. These cases also represent input shapes the project already accepts as valid. Compared with maintaining a large set of hand-written AST fixtures, they are much closer to what the visitor will see in practice.

The conclusion has to be stated carefully. This proves that old and new traversal behavior is equivalent on the ASTs reached by the current SQL corpus. It does not prove equivalence over the entire AST state space.

That distinction matters.

Use Coverage to Expose Blind Spots

Runtime equivalence checks always depend on one assumption: the inputs must actually reach the AST branches we care about.

When we first reused parser SQL cases, we knew the tests ran. We did not know how much of the hand-written walk logic they exercised. If an enum variant, optional field, or special path never appears in the input corpus, the old and new visitors can match perfectly while that part of the migration remains unvalidated.

Coverage becomes a feedback signal:

1. run equivalence comparison on the current SQL corpus;

2. measure coverage of hand-written walk logic and related branches;

3. identify AST paths that were never reached;

4. infer what kind of SQL would reach them;

5. add those SQL cases upstream when possible;

6. rerun both equivalence checks and coverage analysis.

Coverage does not prove correctness. It answers a different question: how much of the space we want to validate did our inputs actually touch?

The two signals complement each other:

• trace comparison tells us whether behavior matches on reached inputs;

• coverage tells us which behavior has not been reached yet.

When a gap can be expressed with real SQL, it should be added to the parser corpus. That way the same case helps parser coverage, AST construction coverage, and visitor validation. For states that cannot be reliably constructed through SQL, targeted local tests can fill the gap.

Treat It as a Feedback System, Not a Checklist

The final system includes source scanning, runtime traces, independent comparison, failure localization, real SQL inputs, and coverage feedback. These pieces do different jobs.

The first four form the validation core. Given an AST, they can compare old and new behavior and generate actionable diffs.

Real SQL inputs expand the set of ASTs the core sees. Coverage feedback shows which parts of the input space remain invisible.

Each component answers a different question:

• source scanning: which AST structures exist and should be considered?

• observation: what did each visitor actually do?

• comparison: what counts as equivalent behavior?

• SQL corpus: what real AST shapes are being tested?

• diff reporting: where should developers look first?

• coverage: what has not been exercised?

• new SQL cases: how do we expand the validated space?

The loop keeps improving the migration:

migration change ─→ trace diff ─→ fix migration or record intentional change

AST definition change ─→ generated output change ─→ expose unhandled structure

coverage gap ─→ add SQL corpus case ─→ expand equivalence comparison

poor diagnostic output ─→ extend trace protocol ─→ improve future diff reports

So this is not “five steps to finish a visitor migration.” A better description is: keep adding different kinds of evidence until the migration is constrained by a validation loop that can run continuously.

What This Validation Can Prove

The system is designed to support these conclusions:

• known AST types, variants, and fields are included in the structural model;

• on ASTs reached by the current corpus, old and new visitors visit the same items;

• event counts match, so missing and duplicated visits are visible;

• order-sensitive hooks do not have unexplained order changes;

• skip
  

  , recursion, and early-return behavior do not have unexplained differences;

• uncovered structures remain visible instead of being hidden behind a passing test run;

• intentional changes and known old-API issues are recorded explicitly.

It still is not a mathematical proof over every possible AST. A finite SQL corpus only constrains reached inputs. Coverage can expose paths that did not run, but it does not prove that code paths that did run are correct. The generator and comparator also need to be reviewed.

The accurate name for this approach is executable equivalence validation with explicit coverage boundaries.

It does not claim to prove everything once and forever. It turns a migration that would otherwise rely heavily on human confidence into an engineering process that can find gaps, add evidence, and converge over time.

AI Generates Code. The System Owns Correctness Boundaries.

This validation loop was built for a visitor migration, but it also clarifies the role of AI in infrastructure work.

AI is good at generating large amounts of repetitive, structured code. It is also useful for quickly applying local fixes after a diff is found. What it does not do reliably on its own is guarantee global completeness in a large, sparse AST with implicit contracts.

A better division of responsibility is:

• AI generates and migrates implementation code;

• the observation system records runtime behavior faithfully;

• the comparator executes explicit equivalence rules;

• humans define traversal contracts, interpret differences, and decide which behavior changes are intentional.

This is not a lower correctness standard for AI-generated code. Human-written and AI-generated infrastructure changes should go through the same independent validation. AI simply makes it more obvious that correctness cannot depend on line-by-line review alone.

Why This Matters for Databend

Databend is a cloud-native data warehouse for modern analytical workloads. SQL parsing, AST representation, binding, optimization, expression handling, and plan generation are all core infrastructure behind complex query execution.

That kind of infrastructure has to satisfy two requirements at the same time:

• move quickly enough to support new syntax, functions, query patterns, and AI-assisted engineering workflows;

• preserve correctness, so migrations, refactors, and API evolution do not silently change SQL semantics.

The AST Visitor migration is a small but representative example of the engineering discipline required here. The goal is not just to generate code faster. The goal is to place complex change inside a loop that is observable, comparable, diagnosable, and continuously expandable.

This also matches how Databend needs to evolve as an open, Rust-native, cloud-native warehouse. Product capabilities matter, but the database engine also needs internal mechanisms that make those capabilities maintainable and verifiable over time.

For teams adopting AI in engineering workflows, this is the more practical question: did AI merely produce more code, or did the system become capable of validating that code before it enters critical infrastructure?

Conclusion

Replacing an old Visitor API with a new one is not difficult because of the amount of code. It is difficult because the migration needs credible evidence that traversal semantics did not change accidentally.

The validation system grew around one central question: for a given AST, do the old and new visitors behave equivalently in the ways that matter? Source scanning, runtime observation, independent comparison, and failure diagnostics formed the validation core. Real SQL corpora expanded the input space. Coverage feedback kept the unvalidated space visible.

AI can accelerate the production of migration code, but it should not be trusted to prove its own output correct. What makes this kind of infrastructure migration manageable is independent observation, centralized rules, real inputs, actionable diffs, and explicit coverage boundaries.