Upgrade Guides#

DataFusion 56.0.0#

Note: DataFusion 56.0.0 has not been released yet. The information provided in this section pertains to features and changes that have already been merged to the main branch and are awaiting release in this version.

Upgrade arrow/parquet to 60.0.0 and object_store to 0.14.2#

DataFusion 56.0.0 uses arrow and parquet 60.0.0, and object_store 0.14.2. This may require updates to your Cargo.toml if you have direct dependencies on these crates.

See the Arrow 60.0.0 release notes and the object_store 0.14.2 upgrade guide for details on breaking changes in those versions.

Field and schema metadata now use arrow’s Metadata type#

Arrow 60 introduced a dedicated Metadata type for field and schema metadata in place of HashMap<String, String>. Following that change, several DataFusion APIs now use Metadata as well, such as DFSchema::metadata and ExprSchema::metadata return &Metadata. CastExpr::target_metadata and TryCastExpr::target_metadata return Option<&Metadata>.

Migration guide:

// Before
let meta: &HashMap<String, String> = schema.metadata();
let field = Field::new("a", DataType::Int64, true)
    .with_metadata([("k".to_string(), "v".to_string())].into());

// After
let meta: &Metadata = schema.metadata();
let field = Field::new("a", DataType::Int64, true)
    .with_metadata(Metadata::new().with("k", "v"));

Metadata supports .get(), .iter(), .is_empty() and .extend() like HashMap, and converts to and from HashMap<String, String> and BTreeMap<String, String> via From, so most call sites need only a type change. Use FieldMetadata::to_hashmap where a HashMap is still required.

Missing Parquet null counts are treated as unknown#

DataFusion now preserves an omitted Parquet null_count statistic as unknown. Previously, it assumed zero nulls, which could discard matching NULL rows in IS NULL filters and ORDER BY ... NULLS FIRST LIMIT queries, or produce an incorrectly high COUNT(column) from metadata.

Queries over affected files may read or sort more data because these optimizations require a known null count:

• Pruning row groups for IS NULL filters.
• Computing COUNT(column) using only file metadata.
• Pruning row groups with a dynamic NULLS FIRST TopK filter.
• Eliminating a sort over nullable columns in sorted, non-overlapping files. Such queries may now retain a full SortExec, even when the data has no NULLs.

parquet-rs versions before 53.1.0 omitted zero null counts. This includes files produced with the older parquet-rs dependency used by DataFusion releases before 42.1.0. Arrow devlive-community/knowforge#6490 changed the writer to record known zero counts. Files with explicit counts retain their existing behavior.

To identify column chunks with bounds but no null count, run this query in the DataFusion CLI:

SELECT row_group_id, path_in_schema
FROM parquet_metadata('data.parquet')
WHERE (stats_min IS NOT NULL OR stats_max IS NOT NULL)
  AND stats_null_count IS NULL;

Rewriting affected files with a current DataFusion version, for example using COPY ... TO with Parquet statistics enabled, records the missing counts and restores optimizations that depend on them. Preserve the required data ordering and ordering metadata when rewriting sorted files.

ForeignSession::create_physical_plan is unsupported#

ForeignSession::create_physical_plan no longer forwards to the library that owns the session. It now returns a NotImplemented error because forwarding can re-enter an installed foreign planner, and the execution-plan handle returned by the old callback cannot restore local Rust type identities for downcasting. The original FFI_SessionRef callback slot remains in place for ABI compatibility with DataFusion 55 consumers, but calling that callback returns the same error.

The session-owning library should instead export its original planner as a datafusion_ffi::query_planner::FFI_QueryPlanner before installing a foreign planner. The foreign planner can retain and invoke that handle to receive a serialized physical plan reconstructed with local type identities. See the datafusion_ffi::query_planner module documentation for the complete delegation pattern. ForeignSession::query_planner, optimize, and physical_optimizers continue to forward to the owning session across the FFI boundary.

GroupColumn now requires values_preserving#

Custom implementations of the public GroupColumn trait must implement values_preserving. This method returns selected rows without changing the stored values or their group indices. It preserves the requested order and supports repeated indices.

Migration guide:

Implement values_preserving, call selection.validate_num_groups(self.len())? before reading, and return rows in selection.iter() order without changing the builder state.

datafusion-proto: common option and constraint conversions are fallible#

Protobuf conversions for CsvOptions, JsonOptions, ParquetCdcOptions, Constraint, and Constraints now reject integer values that do not fit usize. Their infallible From implementations have been replaced with TryFrom. The existing TryFrom conversions for ParquetOptions and TableParquetOptions now also validate all usize-backed fields. A constraint without a constraint_mode returns an error instead of panicking.

Migration guide:

// Before
let csv = CsvOptions::from(&proto_csv);
let cdc = ParquetCdcOptions::from(proto_cdc);
let constraints: Constraints = proto_constraints.into();

// After
let csv = CsvOptions::try_from(&proto_csv)?;
let cdc = ParquetCdcOptions::try_from(proto_cdc)?;
let constraints = Constraints::try_from(proto_constraints)?;

See issue devlive-community/knowforge#24170 for details.

ExecutionOptions has a new enable_nlj_coordinated_fallback field#

ExecutionOptions gained a public enable_nlj_coordinated_fallback: bool field (default true). It controls whether the memory-limited NestedLoopJoinExec fallback shares per-chunk build-side state across probe partitions, which is what lets LEFT, LEFT SEMI, LEFT ANTI, LEFT MARK and FULL joins spill instead of failing with ResourcesExhausted when the right side has several partitions.

Who is affected:

• Users constructing ExecutionOptions (or ConfigOptions) with an exhaustive struct literal. Reading or SET-ing configuration is unaffected.
• Distributed engines that run each output partition as an independent task. The coordination assumes all probe partitions run in one process; with one coordinator per task the shared probe-thread counter never reaches zero and the fallback would stall. Such engines should set the flag to false, which keeps the previous fail-fast behaviour for the affected join types.

Migration guide:

Set the new field, or fill it from Default:

// Before
ExecutionOptions {
    batch_size: 8192,
    // ... every other field ...
}

// After: set it explicitly
ExecutionOptions {
    batch_size: 8192,
    enable_nlj_coordinated_fallback: true,
    // ... every other field ...
}

// After: or let the remaining fields come from Default, which also keeps
// future field additions from breaking the literal
ExecutionOptions {
    batch_size: 8192,
    ..Default::default()
}

Distributed engines opting out:

let mut config = SessionConfig::new();
config.options_mut().execution.enable_nlj_coordinated_fallback = false;

datafusion.optimizer.use_statistics_registry is deprecated and ignored#

The datafusion.optimizer.use_statistics_registry config flag is deprecated and now ignored (setting it emits a deprecation warning). The pluggable StatisticsRegistry is always consulted during the single statistics walk: providers registered on the session take effect directly. With no providers registered the registry is a no-op, so default behavior is unchanged from the previous default (use_statistics_registry = false).

Who is affected:

• Anyone who set datafusion.optimizer.use_statistics_registry. The setting is still accepted (with a deprecation warning) but has no effect, and will be removed in a future release.

Migration guide:

• Drop any use_statistics_registry setting.
• To use statistics providers, register them on the session with SessionStateBuilder::with_statistics_registry(...). To keep the built-in NDV-aware providers the flag previously enabled, register StatisticsRegistry::default_with_builtin_providers().
• To opt out, register nothing (the default).

Generated protobuf JsonWriterOptions gained a compression_level field#

The generated public protobuf JsonWriterOptions struct now carries an optional compression_level so JSON sink plans preserve explicitly configured compression levels across serialization.

Who is affected:

• Users constructing generated JsonWriterOptions values with an exhaustive struct literal.

Migration guide:

Set compression_level explicitly, or fill it from Default:

// Before
JsonWriterOptions { compression }

// After
JsonWriterOptions {
    compression,
    compression_level: None,
}
// or
JsonWriterOptions {
    compression,
    ..Default::default()
}

The protobuf wire format remains backward compatible.

See PR devlive-community/knowforge#24945 for details.

Generated protobuf Parquet structs gained state fields#

The generated public protobuf ParquetScanExecNode struct now carries an optional metadata_size_hint, and ParquetSink now carries optional sorting_columns. These fields preserve the corresponding Parquet source and sink settings across physical-plan serialization.

Who is affected:

• Users constructing either generated struct with an exhaustive struct literal.

Migration guide:

Add metadata_size_hint: None to ParquetScanExecNode literals and sorting_columns: None to ParquetSink literals to retain the previous behavior, or use ..Default::default() for unspecified fields.

The protobuf wire format remains backward compatible.

See PR devlive-community/knowforge#25057 for details.

Generated protobuf CsvWriterOptions changed#

The generated CsvWriterOptions now includes compression_level, timestamp_tz_format, and terminator. Its four existing format fields changed from String to Option<String> to distinguish unset from explicitly empty formats. This affects exhaustive struct literals and direct format-field access.

Wrap existing formats in Some or use None, and initialize the new fields (explicitly or through Default):

// Before (unchanged fields omitted)
CsvWriterOptions {
    date_format: "%Y-%m-%d".to_string(),
    datetime_format: String::new(),
    timestamp_format: String::new(),
    time_format: String::new(),
    // ...
}

// After
CsvWriterOptions {
    date_format: Some("%Y-%m-%d".to_string()),
    datetime_format: None,
    timestamp_format: None,
    time_format: None,
    compression_level: None,
    timestamp_tz_format: None,
    terminator: Vec::new(),
    // ...
}

The protobuf wire format remains backward compatible.

See PR devlive-community/knowforge#25058 for details.

DefaultStatisticsProvider is deprecated#

datafusion_physical_plan::operator_statistics::DefaultStatisticsProvider is deprecated. It is redundant: the statistics walk (StatisticsContext) falls back to each operator’s statistics_from_inputs when the provider chain delegates or is empty, so a terminal “default” provider is no longer needed. It is no longer part of StatisticsRegistry::default_with_builtin_providers().

Migration guide:

• Remove DefaultStatisticsProvider from any custom provider chain; register no terminal provider instead (the walk falls back on its own).

StatisticsRegistry::compute and compute_base are deprecated#

Use the walk instead:

StatisticsContext::new_with_registry(registry)
    .compute_extended(plan, &StatisticsArgs::new())?; // or .compute(...) for core Statistics

API change for floor and ceil UDF#

The output type of the floor and ceil UDFs has been changed from the exact input type to a rescaled type with the same bit width. For example, for input Decimal32(7,2) floor now returns Decimal32(6,0), where the new precision is p - s + 1, matching Spark’s behaviour. See [devlive-community/knowforge#24703] for more details.

Who is affected:

• Users storing query result with these UDFs in a fixed schema
• Users relying on arrow_typeof for these UDFs

Migration guide:

Change the expected type or wrap the expression in CAST. It’s recommended to avoid relying on decimal’s exact precision and scale.

map_extract / element_at return an empty list for absent keys#

map_extract (and its alias element_at) previously returned a single-element list containing NULL when the key was not present in the map. It now returns an empty list, matching the documented behavior and DuckDB. Two related cases changed at the same time, also matching DuckDB:

• A NULL map input now yields NULL instead of [NULL].
• A NULL lookup key now yields [] instead of [NULL].

A key that is present with a NULL value still returns [NULL], so absent keys and NULL values are now distinguishable.

Migration guide:

-- Before
SELECT map_extract(MAP {'a': 1}, 'missing');  -- [NULL]

-- After
SELECT map_extract(MAP {'a': 1}, 'missing');  -- []

Expressions that assumed the result always has exactly one element, for example by checking its length, unnesting it, or comparing it to [NULL], should treat an empty list as the absent-key case instead.

See issue devlive-community/knowforge#24981 and issue devlive-community/knowforge#24983 for details.

DataFrame::from_columns accepts IntoIterator#

DataFrame::from_columns now accepts any IntoIterator<Item = (&str, ArrayRef)> instead of specifically accepting a Vec<(&str, ArrayRef)>.

// Existing Vec usage continues to work
let df = DataFrame::from_columns(vec![
    ("id", id),
    ("name", name),
])?;

// Arrays can now be used directly
let df = DataFrame::from_columns([
    ("id", id),
    ("name", name),
])?;

Most existing call sites using Vec require no changes. Code that relies on the exact non-generic function signature of DataFrame::from_columns may need to be updated to account for the new generic API.

Aggregate ordering state field names are now namespaced#

Ordering expressions exposed as top-level aggregate state fields now use names derived from the aggregate name and their ordering position.

For example:

first_value(value)[ordering_0]

is now used instead of an unqualified ordering field name such as:

timestamp@0

This guarantees unique aggregate state field names and allows DFSchema::check_names() to be enforced.

Who is affected:

Users or integrations that inspect aggregate state field names directly, including custom UDAFs and FFI integrations.

Physical filter pushdown resolves columns by position#

datafusion_physical_plan::filter_pushdown::ChildFilterDescription::from_child and FilterDescription::from_children now resolve filter columns by position instead of looking them up by name. This prevents incorrect results when a child schema contains duplicate column names. from_child requires the child field at each referenced position to have the same name as the filter column.

ChildFilterDescription::from_child_with_allowed_indices is deprecated but preserves its previous name-based mapping to the first matching child field. Migrate to from_child_with_column_mapping because name resolution is ambiguous when the child schema contains duplicate field names.

Migration guide:

Use from_child (or from_children for multiple children) when the parent and child schemas have matching column positions and names. When a node projects, reorders, or renames columns, use from_child_with_column_mapping with an explicit map from parent output indices to child input indices. Columns absent from the mapping cannot be pushed down.

For example, if the parent outputs [a, b] and the child outputs [b, a], a filter on a@0 must map to child column a@1:

use std::collections::{HashMap, HashSet};
use datafusion_physical_plan::filter_pushdown::ChildFilterDescription;

// Before: allow parent column 0 and resolve "a" by name in the child.
let description = ChildFilterDescription::from_child_with_allowed_indices(
    &parent_filters,
    HashSet::from([0]),
    &child,
)?;

// After: explicitly map parent column 0 to child column 1.
let description = ChildFilterDescription::from_child_with_column_mapping(
    &parent_filters,
    HashMap::from([(0, 1)]),
    &child,
)?;

Default datafusion.sql_parser.recursion_limit raised from 50 to 51#

DataFusion now uses sqlparser 0.63.0, which adds recursion guards to more parse functions (for example, data type and INTERVAL parsing). As a result, the same SQL statement consumes slightly more of the parser recursion budget than it did with previous sqlparser versions. In particular, every expression now needs one additional level of depth.

To avoid rejecting queries that previously parsed successfully, the default value of the datafusion.sql_parser.recursion_limit configuration option, and of DFParserBuilder::with_recursion_limit, has been raised from 50 to 51.

If you set datafusion.sql_parser.recursion_limit explicitly to a value close to the depth of your queries, you may see RecursionLimitExceeded errors after upgrading and should raise the limit accordingly.

See PR devlive-community/knowforge#25278 for details.

ensure_distribution is deprecated in favour of ensure_distribution_with_stats#

datafusion_physical_optimizer::enforce_distribution::ensure_distribution now takes its StatisticsContext from the caller, so one context (and its memoization cache) can be shared across a whole bottom-up traversal. The old two-argument form is kept as a deprecated wrapper that allocates a fresh context per call, which recomputes each shared subtree’s statistics once per ancestor.

// Before
let plan = DistributionContext::new_default(plan)
    .transform_up(|ctx| ensure_distribution(ctx, config))
    .data()?;

// After: one context for the whole walk. `StatsCache` is keyed by raw plan-node
// pointers, so reset it whenever a node's plan pointer actually changed.
let stats_ctx = StatisticsContext::new();
let plan = DistributionContext::new_default(plan)
    .transform_up(|ctx| {
        let before = Arc::clone(&ctx.plan);
        let result = ensure_distribution_with_stats(ctx, config, &stats_ctx)?;
        if !Arc::ptr_eq(&before, &result.data.plan) {
            stats_ctx.reset_cache();
        }
        Ok(result)
    })
    .data()?;

Callers that only need the previous behaviour can keep using the deprecated form, or pass a freshly constructed StatisticsContext per call.

MergeIntoOp requires the SQL-visible target qualifier#

MergeIntoOp now stores the SQL-visible target qualifier separately from the target provider identity. This prevents an aliased target from being confused with a source relation that uses the target table’s name.

The struct is now non-exhaustive. Replace 55.0 struct literals with MergeIntoOp::new:

// DataFusion 55.0
let op = MergeIntoOp { on, clauses };

// DataFusion 56.0
let op = MergeIntoOp::new(target_qualifier, on, clauses);

target_qualifier must be the relation name referenced by the MERGE expressions: the target alias when one is present, otherwise the target table reference. DmlStatement::table_name remains the provider/catalog identity.

MergeIntoOpNode protobuf literals require target_qualifier#

The generated MergeIntoOpNode type has a new optional field. Code constructing the generated type with a struct literal must provide it:

// DataFusion 55.0
let node = protobuf::MergeIntoOpNode { on, clauses };

// DataFusion 56.0
let node = protobuf::MergeIntoOpNode {
    on,
    clauses,
    target_qualifier: Some(protobuf::TableReference::from(target_qualifier)),
};

Wire compatibility is directional:

• A 56.0 reader accepts a 55.0 payload. When target_qualifier is absent, it falls back to DmlNode.table_name, matching the 55.0 representation.
• A 55.0 reader must not consume an alias-preserving 56.0 MERGE payload. It ignores the unknown field but cannot preserve the qualifier required by the expressions, which can cause resolution failure or incorrect rebinding.