Bolt Parquet Write Configuration

This document outlines the Parquet write-time configuration options supported in Bolt. The Bolt Parquet writer is based on the Arrow C++ Parquet library and is primarily exposed through the Hive connector.

In this document, “Support” indicates that a configuration is available and effective when set through Bolt’s public configuration pathways (e.g., Hive session/table properties, QueryConfig).

Spark-Level Configurations (Reference)

These standard Spark properties are not directly used by Bolt’s writer. Instead, Bolt provides equivalent functionality through its own configuration mechanisms.

Arrow Writer Properties in Bolt

These properties correspond to settings within the underlying parquet::arrow::WriterProperties and parquet::arrow::ArrowWriterProperties. Bolt exposes some of these directly or maps them from other configurations.

Parquet-MR Style Settings

This table indicates whether classic parquet-mr Hadoop configurations are effectively supported by Bolt’s writer, typically by mapping to an equivalent WriterOptions or WriterProperties setting.

Bolt-Specific Writer Options and Behaviors

These options and behaviors are specific to Bolt’s implementation and provide more granular control over the writing process.

• Default Flush Policy: By default, the Parquet writer flushes a row group when it reaches approximately 1,048,576 rows or its estimated size exceeds 128 MiB. This is defined in DefaultFlushPolicy in bolt/dwio/parquet/writer/Writer.h.
• enableFlushBasedOnBlockSize : A bool in WriterOptions that changes the flush behavior from the default row/byte count policy to a purely block-size-based policy. When true, it uses WriterOptions.parquet_block_size (defaults to 128 MiB) to control row group size and sets max_row_group_length to a very large value to prevent it from triggering first. The Arrow writer’s NewBufferedRowGroup() is used. Found in bolt/dwio/parquet/writer/Writer.cpp.
• enableRowGroupAlignedWrite : A bool in WriterOptions used for specialized data retention scenarios. It works with expectedRowsInEachBlock to create row groups with an exact number of rows. See bolt/dwio/parquet/writer/Writer.h.
• writeBatchBytes / minBatchSize : Internal heuristics within WriterOptions to manage memory and performance when converting large Bolt VectorPtr batches to Arrow RecordBatch. The writer may split large batches into smaller ones based on these thresholds (defaults: 40 MiB and 512 rows). See bolt/dwio/parquet/writer/Writer.cpp.
• Filename Extension: For Hive INSERT operations, HiveDataSink automatically appends the .parquet extension to output files when the storage format is PARQUET. This is handled in bolt/connectors/hive/HiveDataSink.cpp.
• Timestamp Bridge Unit: The precision of timestamps written to Parquet is controlled by the arrow_bridge_timestamp_unit session property (via HiveConfig), which can be set to SECOND, MILLI, MICRO, or NANO. For legacy compatibility, the internal WriterOptions.writeInt96AsTimestamp (bool) can be set to true to force the deprecated INT96 format, and this setting takes precedence over the bridge unit. See bolt/connectors/hive/HiveConfig.cpp and bolt/dwio/parquet/writer/Writer.h.