Delta Lake Feature Support Status in Apache Gluten (Velox Backend)

This document summarizes the support status of Delta Lake table features when used with Apache Gluten (Velox backend).

Supported Spark / Delta combinations

Native Delta write is supported in both Spark 3.5 and Spark 4.0 profiles. The difference between the two rows above is the Spark/Delta compatibility target (Spark 3.5 + Delta 3.3 vs Spark 4.0 + Delta 4.0), not a native-write capability gap.

Build and runtime notes

Build Gluten with Delta support by enabling -Pdelta together with the Velox backend profile and a Spark profile.

• Spark 3.5 build example:

  • mvn clean package -Pbackends-velox -Pdelta -Pspark-3.5 -DskipTests

• Spark 4.0 build example:

  • mvn clean package -Pbackends-velox -Pdelta -Pspark-4.0 -Pscala-2.13 -Pjava-17 -DskipTests

Native Delta write is controlled by:

• spark.gluten.sql.columnar.backend.velox.delta.enableNativeWrite

  • Default: false
  • Type: experimental

Native change data feed scan offload is controlled by:

• spark.gluten.sql.columnar.backend.velox.delta.enableChangeDataFeedScan

  • Default: true
  • Type: experimental

The native DELETE/UPDATE/MERGE deletion-vector target scan is controlled by:

• spark.gluten.sql.columnar.backend.velox.delta.enableNativeDmlRowIndexScan

  • Default: true
  • Type: experimental
  • When disabled, the DML target scan that produces file paths and row indexes for deletion-vector writes stays on Spark; other scans are unaffected.