Run an XTable sync on Apache Spark

xtable-spark-runtime is a runtime jar that runs an Apache XTable™ (Incubating) sync on an Apache Spark cluster. As with any XTable sync, no data files are rewritten — the sync reads the source table's metadata and writes the target format's metadata alongside the data that is already there.

The jar is meant to be dropped into a Spark job you already run, so you don't need a separate process or a separate cluster to keep a table interoperable across formats.

Getting the jar

The jar is published to Maven Central as org.apache.xtable:xtable-spark-runtime_2.12:0.4.0-incubating. Download it once:

shell

curl -O https://repo1.maven.org/maven2/org/apache/xtable/xtable-spark-runtime_2.12/0.4.0-incubating/xtable-spark-runtime_2.12-0.4.0-incubating.jar

Every engine dependency is provided, so the jar is about 4 MB and reuses the Hudi, Iceberg and Delta libraries your cluster already has. See the Downloads page for the full list of releases.

Adding the sync to a Spark job

This is what the jar is for. Your job already writes a table in one format, and you want that same table to be interoperable with the others as soon as the write finishes.

Add the jar to the spark-submit you already use, alongside your application jar:

shell

$SPARK_HOME/bin/spark-submit \
  --jars xtable-spark-runtime_2.12-0.4.0-incubating.jar \
  --class com.example.OrdersJob \
  orders-job.jar

Then call XTableSyncService after your write. You describe the table with a TableSyncSpec and hand it the session's Hadoop configuration:

• Scala
• Java

OrdersJob.scala

import java.util.{Arrays => JArrays}
import org.apache.xtable.spark.{TableSyncSpec, XTableSyncService}

val basePath = "s3://example-warehouse/db/orders"

// the write your job already does
df.write.format("hudi").options(hudiOptions).mode("append").save(basePath)

// the one call you add
new XTableSyncService().sync(
  TableSyncSpec.builder()
    .key("orders")
    .basePath(basePath)
    .sourceFormat("HUDI")
    .targets(JArrays.asList("ICEBERG", "DELTA"))
    .build(),
  spark.sparkContext.hadoopConfiguration)

To compile against these classes, add the same Maven coordinates to your build with provided scope.

The sync runs incrementally and tracks its own watermark in the target's sync metadata, falling back to a full snapshot whenever an incremental sync isn't safe — the first run, for example. That makes it safe to call after every write.

Tables are addressed by path

The runtime jar identifies tables by path rather than through a catalog, so there is no equivalent of the RunSync Iceberg catalog config (-i) yet. When a table's data files don't sit directly under basePath, set dataPath to wherever they do, because that's the location each target writes its metadata to. Iceberg tables often end up at <basePath>/data, but nothing in Iceberg requires it — write.data.path and object-storage layouts can put data files anywhere — so check where your table actually writes.

Runnable example

demo/spark-runtime is a complete job that syncs both directions and verifies the row counts.

Running a sync as its own job

The jar also ships a spark-submit entry point, which is the equivalent of RunSync for this bundle. Pass the jar as the application jar rather than with --jars:

shell

$SPARK_HOME/bin/spark-submit \
  --class org.apache.xtable.spark.XTableSparkSync \
  --master 'local[*]' \
  xtable-spark-runtime_2.12-0.4.0-incubating.jar \
  --basepath /path/to/hudi_table \
  --sourceformat HUDI \
  --targets ICEBERG,DELTA

To sync several tables in one submit, use --datasetconfig. It takes the same YAML that RunSync uses, so an existing config works unchanged, and unlike RunSync the config itself may live on cloud storage:

my_config.yaml

sourceFormat: HUDI
targetFormats:
  - DELTA
  - ICEBERG
datasets:
  -
    tableBasePath: s3://tpc-ds-datasets/1GB/hudi/call_center
    tableDataPath: s3://tpc-ds-datasets/1GB/hudi/call_center/data
    tableName: call_center
    namespace: my.db
  -
    tableBasePath: s3://tpc-ds-datasets/1GB/hudi/catalog_sales
    tableName: catalog_sales
    partitionSpec: cs_sold_date_sk:VALUE
  -
    tableBasePath: s3://hudi/multi-partition-dataset
    tableName: multi_partition_dataset
    partitionSpec: time_millis:DAY:yyyy-MM-dd,type:VALUE

shell

$SPARK_HOME/bin/spark-submit \
  --class org.apache.xtable.spark.XTableSparkSync \
  xtable-spark-runtime_2.12-0.4.0-incubating.jar \
  --datasetconfig my_config.yaml

--datasetconfig and --basepath are mutually exclusive — pass one or the other.

Supported formats

Paimon and Parquet are read-only sources; XTable does not write either format as a target.

Spark version support

Converting to and from Hudi and Iceberg doesn't require Spark at all, so those run on any of the Spark versions below. Delta is the only engine whose implementation depends on the Spark version, and the jar chooses the right one automatically:

Delta Standalone doesn't run on Spark 3.5, so on 3.5 and newer a Delta source or target is routed through Delta Kernel with no flag needed. If you want Kernel on Spark 3.4 as well, pass --usedeltakernel, or set .useDeltaKernel(true) on the TableSyncSpec.

Next steps

• See the Quickstart for an end-to-end interoperability walkthrough.
• See Apache Spark for the options each format needs when you query a synced table.
• See Installation if you'd rather build the project from source.