KnowForge/Apache DataFusion 简体中文
DataFusion CLI

用法

师成师成· 更新于 2026-09-28· 阅读 16 分钟· 0 次阅读
登录后可跨设备保存划线和私人笔记登录

用法

使用 datafusion-cli --help 查看当前用法：

Command Line Client for DataFusion query engine.

Usage: datafusion-cli [OPTIONS]

Options:
  -p, --data-path <DATA_PATH>
          Path to your data, default to current directory
  -b, --batch-size <BATCH_SIZE>
          The batch size of each query, or use DataFusion default
  -c, --command [<COMMAND>...]
          Execute the given command string(s), then exit. Commands are expected to be non empty.
  -m, --memory-limit <MEMORY_LIMIT>
          The memory pool limitation (e.g. '10g'), default to None (no limit)
  -f, --file [<FILE>...]
          Execute commands from file(s), then exit
  -r, --rc [<RC>...]
          Run the provided files on startup instead of ~/.datafusionrc
      --format <FORMAT>
          [default: automatic] [possible values: csv, tsv, table, json, nd-json, automatic]
  -q, --quiet
          Reduce printing other than the results and work quietly
      --mem-pool-type <MEM_POOL_TYPE>
          Specify the memory pool type 'greedy' or 'fair' [default: greedy]
      --top-memory-consumers <TOP_MEMORY_CONSUMERS>
          The number of top memory consumers to display when query fails due to memory exhaustion. To disable memory consumer tracking, set this value to 0 [default: 3]
      --maxrows <MAXROWS>
          The max number of rows to display for 'Table' format
          [possible values: numbers(0/10/...), inf(no limit)] [default: 40]
      --color
          Enables console syntax highlighting
      --spark
          Enable Apache Spark-compatible SQL dialect, functions, and expression planning
  -d, --disk-limit <DISK_LIMIT>
          Available disk space for spilling queries (e.g. '10g'), default to None (uses DataFusion's default value of '100g')
      --object-store-profiling <OBJECT_STORE_PROFILING>
          Specify the default object_store_profiling mode, defaults to 'disabled'.
          [possible values: disabled, summary, trace] [default: Disabled]
  -h, --help
          Print help
  -V, --version
          Print version

Apache Spark 兼容性

使用 --spark 可为 CLI 会话启用 Apache Spark SQL 方言、表达式规划器和函数库：

datafusion-cli --spark -c "SELECT next_day('2015-07-27'::DATE, 'Sun'::STRING);"

Spark 兼容模式默认处于禁用状态。启用后，会在当前会话期间覆盖 DataFusion 中同名的函数。

命令

DataFusion CLI 中可用的命令如下：

  • Quit
> \q
  • 帮助
> \?
  • ListTables（列出表）
> \d
  • DescribeTable（描述表）
> \d table_name
  • 静默模式（QuietMode）
> \quiet [true|false]
  • 列出函数
> \h
  • 搜索并描述函数
> \h function
  • 对象存储剖析模式
> \object_store_profiling [disabled|summary|trace]

启用后，将查询执行过程中对象存储（I/O）操作的详细信息打印到标准输出（STDOUT）。

> \object_store_profiling trace
ObjectStore Profile mode set to Trace
> select count(*) from 'https://datasets.clickhouse.com/hits_compatible/athena_partitioned/hits_1.parquet';
+----------+
| count(*) |
+----------+
| 1000000  |
+----------+
1 row(s) fetched.
Elapsed 0.552 seconds.

Object Store Profiling
Instrumented Object Store: instrument_mode: Trace, inner: HttpStore
2025-10-17T18:08:48.457992+00:00 operation=Get duration=0.043592s size=8 range: bytes=174965036-174965043 path=hits_compatible/athena_partitioned/hits_1.parquet
2025-10-17T18:08:48.501878+00:00 operation=Get duration=0.031542s size=34322 range: bytes=174930714-174965035 path=hits_compatible/athena_partitioned/hits_1.parquet

Summaries:
+-----------+----------+-----------+-----------+-----------+-----------+-------+
| Operation | Metric   | min       | max       | avg       | sum       | count |
+-----------+----------+-----------+-----------+-----------+-----------+-------+
| Get       | duration | 0.031542s | 0.043592s | 0.037567s | 0.075133s | 2     |
| Get       | size     | 8 B       | 34322 B   | 17165 B   | 34330 B   | 2     |
+-----------+----------+-----------+-----------+-----------+-----------+-------+

支持的 SQL

除了 DataFusion 支持的标准 SQL 之外，datafusion-cli 还支持额外的语句和命令：

SHOW ALL [VERBOSE]

显示配置选项

> show all;

+-------------------------------------------------+---------+
| name                                            | value   |
+-------------------------------------------------+---------+
| datafusion.execution.batch_size                 | 8192    |
| datafusion.execution.coalesce_batches           | true    |
| datafusion.execution.time_zone                  | UTC     |
| datafusion.explain.logical_plan_only            | false   |
| datafusion.explain.physical_plan_only           | false   |
| datafusion.optimizer.filter_null_join_keys      | false   |
| datafusion.optimizer.skip_failed_rules          | true    |
+-------------------------------------------------+---------+

SHOW <OPTION>>

显示特定的配置选项。

> show datafusion.execution.batch_size;

+-------------------------------------------------+---------+
| name                                            | value   |
+-------------------------------------------------+---------+
| datafusion.execution.batch_size                 | 8192    |
+-------------------------------------------------+---------+

SET <OPTION> TO <VALUE>

  • 设置配置选项
> SET datafusion.execution.batch_size to 1024;

配置选项

如上所述，可以使用 SHOW ALL 查看所有可用的配置选项。

你可以通过环境变量修改配置选项。datafusion-cli 会查找与配置名对应的环境变量，方法是将配置名转为全大写，并把所有的 . 替换为 _。

例如，要将 datafusion.execution.batch_size 设置为 1024，你需要相应地设置 DATAFUSION_EXECUTION_BATCH_SIZE 环境变量：

$ DATAFUSION_EXECUTION_BATCH_SIZE=1024 datafusion-cli
DataFusion CLI v12.0.0
> show all;
+-------------------------------------------------+---------+
| name                                            | value   |
+-------------------------------------------------+---------+
| datafusion.execution.batch_size                 | 1024    |
| datafusion.execution.coalesce_batches           | true    |
| datafusion.execution.time_zone                  | UTC     |
| datafusion.explain.logical_plan_only            | false   |
| datafusion.explain.physical_plan_only           | false   |
| datafusion.optimizer.filter_null_join_keys      | false   |
| datafusion.optimizer.skip_failed_rules          | true    |
+-------------------------------------------------+---------+
8 rows in set. Query took 0.002 seconds.

你也可以使用 SET 语句来修改配置选项。

$ datafusion-cli
DataFusion CLI v13.0.0
> show datafusion.execution.batch_size;
+---------------------------------+---------+
| name                            | value   |
+---------------------------------+---------+
| datafusion.execution.batch_size | 8192    |
+---------------------------------+---------+
1 row in set. Query took 0.011 seconds.

> set datafusion.execution.batch_size to 1024;
0 rows in set. Query took 0.000 seconds.

> show datafusion.execution.batch_size;
+---------------------------------+---------+
| name                            | value   |
+---------------------------------+---------+
| datafusion.execution.batch_size | 1024    |
+---------------------------------+---------+
1 row in set. Query took 0.005 seconds.

函数

datafusion-cli 自带一些 DataFusion SQL 引擎中不包含的内置函数，详情请参阅DataFusion CLI 专有函数章节。

评论

登录后参与评论

正在加载评论…