Testing#

Tests are critical to ensure that DataFusion is working properly and is not accidentally broken during refactorings. All new features should have test coverage and the entire test suite is run as part of CI.

Testing Quick Start#

While developing a feature or bug fix, best practice is to run the smallest set of tests that gives confidence for your change, then expand as needed.

Initially, run the tests in the crates you changed. For example, if you made changes to files in datafusion-optimizer/src, run the corresponding crate tests:

cargo test -p datafusion-optimizer

Then, run the sqllogictest suite, which provides a strong speed–coverage tradeoff for development: it runs quickly while offering broad regression coverage across most SQL behavior in DataFusion.

cargo test --profile=ci --test sqllogictests

Finally, before submitting a PR, run the tests for the core datafusion and datafusion-cli crates:

cargo test -p datafusion
cargo test -p datafusion-cli

Some integration tests require optional external services such as Docker-backed containers and may skip when unavailable.

Testing Overview#

DataFusion has several levels of tests in its Test Pyramid and tries to follow the Rust standard Testing Organization described in The Book.

Run tests using cargo:

cargo test

You can also use other runners such as cargo-nextest.

cargo nextest run

Unit tests#

Tests for code in an individual module are defined in the same source file with a test module, following Rust convention.

For example, to run tests in the datafusion crate:

cargo test -p datafusion

The test_util module provides useful macros to write unit tests effectively, such as assert_batches_sorted_eq and assert_batches_eq for RecordBatches and assert_contains / assert_not_contains which are used extensively in the codebase.

sqllogictests Tests#

DataFusion’s SQL implementation is tested using sqllogictest. You can run these tests with commands like:

# Run all tests
cargo test --profile=ci --test sqllogictests
# Run a specific test file
cargo test --profile=ci --test sqllogictests -- aggregate.slt
# Run a specific test file and update expected outputs
cargo test --profile=ci --test sqllogictests -- aggregate.slt --complete
# Run and update expected outputs for all test files
cargo test --profile=ci --test sqllogictests -- --complete

sqllogictests may be less convenient for new contributors who are familiar with writing .rs tests as they require learning another tool. However, sqllogictest based tests are much easier to develop and maintain as they 1) do not require a slow recompile/link cycle and 2) can be automatically updated.

Like similar systems such as DuckDB, DataFusion has chosen to trade off a slightly higher barrier to contribution for longer term maintainability.

DataFusion runs sqlite’s test suite in the merge queue before merging PRs into main. For local instructions, see Running Tests: sqlite.

Snapshot testing (cargo insta)#

Insta is used for snapshot testing. Snapshots are generated and compared on each test run. If the output changes, tests will fail.

To review the changes, you can use Insta CLI:

cargo install cargo-insta
cargo insta review

Extended Tests#

DataFusion has extended tests (defined in extended.yml) that take significantly longer to run than the standard suite. They provide additional correctness coverage and must pass in the merge queue before a PR merges into main.

To conserve CI resources, these tests do not run on ordinary PR updates. They also run on pushes to release branches (branch-*). You can run them manually.

For local SQLite test instructions, see the instructions in the documentation.

Rust Integration Tests#

There are several public interface tests for the DataFusion library in the tests directory.

You can run these tests individually using cargo as normal command such as

cargo test -p datafusion --test parquet_integration

SQL “Fuzz” testing#

DataFusion uses the SQLancer for “fuzz” testing: it generates random SQL queries and execute them against DataFusion to find bugs.

The code is in the datafusion-sqllancer repository, and we welcome further contributions. Kudos to @2010YOUY01 for the initial implementation.

Documentation Examples#

We use Rust doctest to verify examples from the documentation are correct and up-to-date. These tests are run as part of our CI and you can run them locally with the following command:

cargo test --doc

API Documentation Examples#

As with other Rust projects, examples in doc comments in .rs files are automatically checked to ensure they work and evolve along with the code.

User Guide Documentation#

Rust example code from the user guide (anything marked with ```rust) is also tested in the same way using the doc_comment crate. See the end of core/src/lib.rs for more details.

Documentation Link Checks#

./dev/rust_lint.sh runs the internal markdown link check. If lychee is missing, the script installs the version pinned in ci/scripts/utils/tool_versions.sh. It uses an existing installation as is, even if the version differs from the pin.

To run the check on its own:

source ci/scripts/utils/tool_versions.sh
cargo install lychee --locked --version "${LYCHEE_VERSION}"
bash ci/scripts/markdown_link_check.sh

Notes:

• The script is run with bash and is compatible with the default Bash on macOS (no mapfile dependency).
• The CI configuration currently checks internal markdown links only. External http(s) and mailto links are excluded to avoid flaky failures.
• The check only reports broken links. ./dev/rust_lint.sh --write does not change them.

When a link is broken, lychee prints the file and URL/path that failed. For example:

[docs/source/user-guide/cli/overview.md]:
  [ERROR] file:///.../docs/source/user-guide/cli/missing-page.md | Cannot find file: File not found. Check if file exists and path is correct

Rust doc comments are validated by rustdoc in CI and can be checked locally with:

bash ci/scripts/rust_docs.sh

ASF Status Check Validation#

ci/scripts/check_asf_yaml_status_checks.py checks that every required status check in .asf.yaml matches a job in .github/workflows, and that rust.yml skips only its listed jobs on pushes to main. ./dev/rust_lint.sh runs it and needs python3 with PyYAML. The uv workspace provides both:

uv run ./dev/rust_lint.sh

To run the check on its own:

uv run python3 ci/scripts/check_asf_yaml_status_checks.py

Security Audit#

ci/scripts/security_audit.sh runs cargo audit on the root Cargo.lock with the advisory exceptions that CI uses. ./dev/rust_lint.sh runs it and installs cargo-audit if it is missing. To run the audit on its own:

./ci/scripts/security_audit.sh

The audit fetches the RustSec advisory database. A new advisory or a different cargo-audit version can change the result without any change to the repository.

Large File Check#

ci/scripts/check_large_files.sh fails if any file committed between a base ref and a head ref is larger than 1.5 MB, the same check the “Large files PR check” workflow runs on pull requests. ./dev/rust_lint.sh runs it against the merge base of HEAD and main on the remote that points at apache/datafusion, or on origin when there is no such remote. To run the check on its own, or against a different range:

./ci/scripts/check_large_files.sh
./ci/scripts/check_large_files.sh --base upstream/main --head my-branch

Only committed files are checked. Commit a change before running it.

Dependency Checks#

CI runs two dependency checks, and ./dev/rust_lint.sh runs both:

• ci/scripts/check_circular_dependencies.sh builds and runs dev/depcheck, which fails on dependency cycles between DataFusion crates.
• ci/scripts/check_unused_dependencies.sh runs cargo machete --with-metadata from the repository root. The lint suite installs cargo-machete with the version in ci/scripts/utils/tool_versions.sh if it is missing.

To run either check on its own:

./ci/scripts/check_circular_dependencies.sh
./ci/scripts/check_unused_dependencies.sh

Examples README Check#

datafusion-examples/README.md is generated from the documentation comments in datafusion-examples/examples/<group>/main.rs. ci/scripts/check_examples_docs.sh regenerates it and fails if the committed file differs. ./dev/rust_lint.sh runs it and needs cargo and npx. To run the check on its own, or to update the README:

./ci/scripts/check_examples_docs.sh
./ci/scripts/check_examples_docs.sh --write

Config and Function Docs Check#

configs.md, aggregate_functions.md, scalar_functions.md, and window_functions.md are generated by dev/update_config_docs.sh and dev/update_function_docs.sh. To check that they are up to date, run the ci/scripts/check_generated_docs.sh script, which also runs as part of ./dev/rust_lint.sh. Run it with --write to update the pages.

Benchmarks#

Criterion Benchmarks#

Criterion is a statistics-driven micro-benchmarking framework used by DataFusion for evaluating the performance of specific code-paths. In particular, the criterion benchmarks help to both guide optimisation efforts, and prevent performance regressions within DataFusion.

Criterion integrates with Cargo’s built-in benchmark support and a given benchmark can be run with

cargo bench --bench BENCHMARK_NAME

A full list of benchmarks can be found here.

cargo-criterion may also be used for more advanced reporting.

Parquet SQL Benchmarks#

The parquet SQL benchmarks can be run with

 cargo bench --bench parquet_query_sql

These randomly generate a parquet file, and then benchmark queries sourced from parquet_query_sql.sql against it. This can therefore be a quick way to add coverage of particular query and/or data paths.

If the environment variable PARQUET_FILE is set, the benchmark will run queries against this file instead of a randomly generated one. This can be useful for performing multiple runs, potentially with different code, against the same source data, or for testing against a custom dataset.

The benchmark will automatically remove any generated parquet file on exit, however, if interrupted (e.g. by CTRL+C) it will not. This can be useful for analysing the particular file after the fact, or preserving it to use with PARQUET_FILE in subsequent runs.

Comparing Baselines#

By default, Criterion.rs will compare the measurements against the previous run (if any). Sometimes it’s useful to keep a set of measurements around for several runs. For example, you might want to make multiple changes to the code while comparing against the master branch. For this situation, Criterion.rs supports custom baselines.

 git checkout main
 cargo bench --bench sql_planner -- --save-baseline main
 git checkout YOUR_BRANCH
 cargo bench --bench sql_planner --  --baseline main

Note: For MacOS it may be required to run cargo bench with sudo

sudo cargo bench ...

More information on Baselines

Upstream Benchmark Suites#

Instructions and tooling for running upstream benchmark suites against DataFusion can be found in benchmarks.

These are valuable for comparative evaluation against alternative Arrow implementations and query engines.