KnowForge快速入门
Docker 演示
师成· 更新于 2026-09-29· 阅读 107 分钟· 0 次阅读登录后可跨设备保存划线和私人笔记登录
使用 Docker 容器的演示
让我们用一个真实的示例来看看 Hudi 的端到端工作流程。为此，会在你计算机的本地 Docker 集群中启动一套自包含的数据基础设施。这需要你已在本地克隆 Hudi 仓库。
这些步骤已在 Mac 笔记本电脑上测试通过。
前置条件
将 Hudi 仓库克隆到你的本地机器。
Docker 安装：对于 Mac，请按照在 Mac 上安装 Docker Desktop 中的步骤操作。要运行 Spark-SQL 查询，请确保为 Docker 分配至少 6 GB 内存和 4 个 CPU（参见 Docker -> Preferences -> Advanced）。否则，Spark-SQL 查询可能会因内存问题被终止。
kcat：用于向 Kafka 主题发布/消费消息的命令行工具。使用
brew install kcat安装 kcat。/etc/hosts：本演示通过主机名引用许多运行在容器中的服务。请将以下配置添加到 /etc/hosts
127.0.0.1 adhoc-1 127.0.0.1 adhoc-2 127.0.0.1 namenode 127.0.0.1 datanode1 127.0.0.1 hiveserver 127.0.0.1 hivemetastore 127.0.0.1 kafkabroker 127.0.0.1 sparkmaster 127.0.0.1 zookeeperJava：Java SE Development Kit 8。
Maven：Java 项目的构建自动化工具。
jq：轻量灵活的命令行 JSON 处理器。使用
brew install jq安装 jq。
另外，此演示尚未在某些环境（如 Windows 上的 Docker）中测试过。
搭建 Docker 集群
构建 Hudi
第一步是构建 Hudi。注意 此步骤使用受支持的 Scala 版本 - 2.12 构建 Hudi。
注意：请确保你已先克隆了 Hudi 仓库。
cd <HUDI_WORKSPACE>
mvn clean package -Pintegration-tests -DskipTests -Dspark3.5 -Dscala-2.12启动演示集群
下一步是运行 Docker compose 脚本并配置相关参数以启动集群。这些文件位于 Hudi 代码仓库中，按照前面的步骤，你本地应该已经克隆了该仓库。
- 注意
- 演示必须基于 master 分支进行构建和运行。
- 当前演示不支持 Presto 和 Trino。
运行以下命令，在本地为本演示构建所需的 Docker 镜像。
cd docker
./build_docker_images.sh这一步将完成 Docker 集群的搭建。
cd docker
./setup_demo.sh
....
....
....
[+] Running 10/13
⠿ Container zookeeper Removed 8.6s
⠿ Container datanode1 Removed 18.3s
⠿ Container spark-worker-1 Removed 16.7s
⠿ Container adhoc-2 Removed 16.9s
⠿ Container graphite Removed 16.9s
⠿ Container kafkabroker Removed 14.1s
⠿ Container adhoc-1 Removed 14.1s
.......
......
[+] Running 13/13
⠿ adhoc-1 Pulled 2.9s
⠿ graphite Pulled 2.8s
⠿ spark-worker-1 Pulled 3.0s
⠿ kafka Pulled 2.9s
⠿ datanode1 Pulled 2.9s
⠿ hivemetastore Pulled 2.9s
⠿ hiveserver Pulled 3.0s
⠿ hive-metastore-postgresql Pulled 2.8s
⠿ namenode Pulled 2.9s
⠿ sparkmaster Pulled 2.9s
⠿ zookeeper Pulled 2.8s
⠿ adhoc-2 Pulled 2.9s
⠿ historyserver Pulled 2.9s
[+] Running 13/13
⠿ Container zookeeper Started 41.0s
⠿ Container kafkabroker Started 41.7s
⠿ Container graphite Started 41.5s
⠿ Container hive-metastore-postgresql Running 0.0s
⠿ Container namenode Running 0.0s
⠿ Container hivemetastore Running 0.0s
⠿ Container historyserver Started 41.0s
⠿ Container datanode1 Started 49.9s
⠿ Container hiveserver Running 0.0s
⠿ Container sparkmaster Started 41.9s
⠿ Container spark-worker-1 Started 50.2s
⠿ Container adhoc-2 Started 38.5s
⠿ Container adhoc-1 Started 38.5s
Copying spark default config and setting up configs
Copying spark default config and setting up configs
$ docker ps此时，Docker 集群已经启动并运行。演示集群会启动以下服务
- HDFS 服务（NameNode、DataNode）
- Spark Master 和 Worker
- Hive 服务（Metastore、HiveServer2 以及 PostgresDB）
- Kafka Broker 和一个 Zookeeper 节点（Kafka 将作为演示的上游数据源）
- 用于运行 Hudi/Hive CLI 命令的临时容器
演示
将使用股票追踪数据来展示不同的 Hudi 查询类型以及 Compaction 的效果。
请查看目录 docker/demo/data。其中有 2 批股票数据——每批的时间粒度为 1 分钟。第一批包含某些股票代码在交易时段第一个小时（上午 9:30 至 10:30）内的股票追踪数据。第二批包含接下来 30 分钟（上午 10:30 至 11:00）的追踪数据。Hudi 将用于将这些批次摄入到一张表中，该表将包含小时粒度的最新股票追踪数据。这些批次被有意设置为时间窗口，使得第二批包含对第一批中部分行的更新。
第一步：将第一批数据发布到 Kafka
将第一批数据上传到 Kafka topic stock ticks
cat demo/data/batch_1.json | kcat -b kafkabroker -t stock_ticks -P
要检查新 topic 是否出现，请使用
kcat -b kafkabroker -L -J | jq .
{
"originating_broker": {
"id": 1001,
"name": "kafkabroker:9092/1001"
},
"query": {
"topic": "*"
},
"brokers": [
{
"id": 1001,
"name": "kafkabroker:9092"
}
],
"topics": [
{
"topic": "stock_ticks",
"partitions": [
{
"partition": 0,
"leader": 1001,
"replicas": [
{
"id": 1001
}
],
"isrs": [
{
"id": 1001
}
]
}
]
}
]
}第 2 步：从 Kafka 主题增量摄取数据
Hudi 附带了一个名为 Hudi Streamer 的工具。该工具可以连接到多种数据源（包括 Kafka）来拉取变更，并使用 upsert/insert 原语将其应用到 Hudi 表中。在这里，我们将使用该工具从 Kafka 主题下载 JSON 数据，并将其摄取到上一步中初始化的 COW 和 MOR 表中。如果表在文件系统中尚不存在，该工具会自动初始化它们。
docker exec -it adhoc-2 /bin/bash
# Run the following spark-submit command to execute the Hudi Streamer and ingest to stock_ticks_cow table in HDFS
spark-submit \
--class org.apache.hudi.utilities.streamer.HoodieStreamer $HUDI_UTILITIES_BUNDLE \
--table-type COPY_ON_WRITE \
--source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
--source-ordering-field ts \
--target-base-path /user/hive/warehouse/stock_ticks_cow \
--target-table stock_ticks_cow --props /var/demo/config/kafka-source.properties \
--schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider
# Run the following spark-submit command to execute the Hudi Streamer and ingest to stock_ticks_mor table in HDFS
spark-submit \
--class org.apache.hudi.utilities.streamer.HoodieStreamer $HUDI_UTILITIES_BUNDLE \
--table-type MERGE_ON_READ \
--source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
--source-ordering-field ts \
--target-base-path /user/hive/warehouse/stock_ticks_mor \
--target-table stock_ticks_mor \
--props /var/demo/config/kafka-source.properties \
--schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider \
--disable-compaction
# As part of the setup (Look at setup_demo.sh), the configs needed for Hudi Streamer is uploaded to HDFS. The configs
# contain mostly Kafa connectivity settings, the avro-schema to be used for ingesting along with key and partitioning fields.
exit你可以使用 HDFS 网页浏览器查看表 http://namenode:9870/explorer.html#/user/hive/warehouse/stock_ticks_cow。
你可以浏览表中新创建的分区文件夹，以及 .hoodie 目录下表示提交成功的 "commit" / "deltacommit" 文件。
浏览 MOR 表 http://namenode:9870/explorer.html#/user/hive/warehouse/stock_ticks_mor 时也会看到类似的结构。
步骤 3：与 Hive 同步
此时，表已经存在于 HDFS 中。我们需要与 Hive 同步，以创建新的 Hive 表并添加分区，从而能够针对这些表运行 Hive 查询。
docker exec -it adhoc-2 /bin/bash
# This command takes in HiveServer URL and COW Hudi table location in HDFS and sync the HDFS state to Hive
/var/hoodie/ws/hudi-sync/hudi-hive-sync/run_sync_tool.sh \
--jdbc-url jdbc:hive2://hiveserver:10000 \
--user hive \
--pass hive \
--partitioned-by dt \
--base-path /user/hive/warehouse/stock_ticks_cow \
--database default \
--table stock_ticks_cow \
--partition-value-extractor org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor
.....
2025-09-26 13:57:58,718 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(281)) - Sync complete for stock_ticks_cow
.....
# Now run hive-sync for the second data-set in HDFS using Merge-On-Read (MOR table type)
/var/hoodie/ws/hudi-sync/hudi-hive-sync/run_sync_tool.sh \
--jdbc-url jdbc:hive2://hiveserver:10000 \
--user hive \
--pass hive \
--partitioned-by dt \
--base-path /user/hive/warehouse/stock_ticks_mor \
--database default \
--table stock_ticks_mor \
--partition-value-extractor org.apache.hudi.hive.SlashEncodedDayPartitionValueExtractor
...
2025-09-26 13:58:36,052 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(281)) - Sync complete for stock_ticks_mor_ro
...
2025-09-26 13:58:36,184 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(281)) - Sync complete for stock_ticks_mor_rt
...
2025-09-26 13:58:36,308 INFO [main] hive.HiveSyncTool (HiveSyncTool.java:syncHoodieTable(281)) - Sync complete for stock_ticks_mor
....
exit执行上述命令后，你会发现：
- 创建了一个名为
stock_ticks_cow的 Hive 表，该表支持对 Copy On Write 表进行快照查询和增量查询。 - 为 Merge On Read 表创建了两个新表
stock_ticks_mor_rt和stock_ticks_mor_ro。前者支持快照查询和增量查询（提供近实时数据），后者支持读优化查询。
步骤 4(a)：运行 Hive 查询
运行一条 Hive 查询，以查找股票代码 'GOOG' 摄入的最新时间戳。你会发现，无论是快照查询（针对 COW 表和 MOR_rt 表），还是读优化查询（针对 MOR_ro 表），结果都是相同的 "10:29 a.m"，因为 Hudi 为第一批数据创建了 Parquet 文件。
docker exec -it adhoc-2 /bin/bash
beeline -u jdbc:hive2://hiveserver:10000 \
--hiveconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat \
--hiveconf hive.stats.autogather=false \
--hiveconf hive.vectorized.input.format.excludes=org.apache.hudi.hadoop.HoodieParquetInputFormat \
--hiveconf parquet.column.index.access=true
# List Tables
0: jdbc:hive2://hiveserver:10000> show tables;
+---------------------+--+
| tab_name |
+---------------------+--+
| stock_ticks_cow |
| stock_ticks_mor |
| stock_ticks_mor_ro |
| stock_ticks_mor_rt |
+---------------------+--+
4 rows selected (1.099 seconds)
0: jdbc:hive2://hiveserver:10000>
# Look at partitions that were added
0: jdbc:hive2://hiveserver:10000> show partitions stock_ticks_mor_rt;
+----------------+--+
| partition |
+----------------+--+
| dt=2018-08-31 |
+----------------+--+
1 row selected (0.24 seconds)
# COPY-ON-WRITE Queries:
=========================
0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG';
+---------+----------------------+--+
| symbol | _c1 |
+---------+----------------------+--+
| GOOG | 2018-08-31 10:29:00 |
+---------+----------------------+--+
Now, run a projection query:
0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG';
+----------------------+---------+----------------------+---------+------------+-----------+--+
| _hoodie_commit_time | symbol | ts | volume | open | close |
+----------------------+---------+----------------------+---------+------------+-----------+--+
| 20250926135641514 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
| 20250926135641514 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085 |
+----------------------+---------+----------------------+---------+------------+-----------+--+
# Merge-On-Read Queries:
==========================
Lets run similar queries against M-O-R table. Lets look at both
ReadOptimized and Snapshot(realtime data) queries supported by M-O-R table
# Run ReadOptimized Query. Notice that the latest timestamp is 10:29
0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG';
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
+---------+----------------------+--+
| symbol | _c1 |
+---------+----------------------+--+
| GOOG | 2018-08-31 10:29:00 |
+---------+----------------------+--+
1 row selected (6.326 seconds)
# Run Snapshot Query. Notice that the latest timestamp is again 10:29
0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG';
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
+---------+----------------------+--+
| symbol | _c1 |
+---------+----------------------+--+
| GOOG | 2018-08-31 10:29:00 |
+---------+----------------------+--+
1 row selected (1.606 seconds)
# Run Read Optimized and Snapshot project queries
0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_ro where symbol = 'GOOG';
+----------------------+---------+----------------------+---------+------------+-----------+--+
| _hoodie_commit_time | symbol | ts | volume | open | close |
+----------------------+---------+----------------------+---------+------------+-----------+--+
| 20250926135725397 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
| 20250926135725397 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085 |
+----------------------+---------+----------------------+---------+------------+-----------+--+
0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_rt where symbol = 'GOOG';
+----------------------+---------+----------------------+---------+------------+-----------+--+
| _hoodie_commit_time | symbol | ts | volume | open | close |
+----------------------+---------+----------------------+---------+------------+-----------+--+
| 20250926135725397 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
| 20250926135725397 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085 |
+----------------------+---------+----------------------+---------+------------+-----------+--+
exit步骤 4 (b)：运行 Spark SQL 查询
与 Hive 一样，Hudi 也支持将 Spark 作为查询处理器。以下是在 spark-sql 中运行的与前面相同的 Hive 查询：
docker exec -it adhoc-1 /bin/bash
$SPARK_INSTALL/bin/spark-shell \
--jars $HUDI_SPARK_BUNDLE \
--master local[2] \
--driver-class-path $HADOOP_CONF_DIR \
--conf spark.sql.hive.convertMetastoreParquet=false \
--deploy-mode client \
--driver-memory 1G \
--executor-memory 3G \
--num-executors 1
...
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.5.3
/_/
Using Scala version 2.12.18 (OpenJDK 64-Bit Server VM, Java 1.8.0_342)
Type in expressions to have them evaluated.
Type :help for more information.
scala> spark.sql("show tables").show(100, false)
+--------+------------------+-----------+
|database|tableName |isTemporary|
+--------+------------------+-----------+
|default |stock_ticks_cow |false |
|default |stock_ticks_mor |false |
|default |stock_ticks_mor_ro|false |
|default |stock_ticks_mor_rt|false |
+--------+------------------+-----------+
# Copy-On-Write Table
## Run max timestamp query against COW table
scala> spark.sql("select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG'").show(100, false)
[Stage 0:> (0 + 1) / 1]SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes#StaticLoggerBinder for further details.
+------+-------------------+
|symbol|max(ts) |
+------+-------------------+
|GOOG |2018-08-31 10:29:00|
+------+-------------------+
## Projection Query
scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG'").show(100, false)
+-------------------+------+-------------------+------+---------+--------+
|_hoodie_commit_time|symbol|ts |volume|open |close |
+-------------------+------+-------------------+------+---------+--------+
|20250926135641514 |GOOG |2018-08-31 09:59:00|6330 |1230.5 |1230.02 |
|20250926135641514 |GOOG |2018-08-31 10:29:00|3391 |1230.1899|1230.085|
+-------------------+------+-------------------+------+---------+--------+
# Merge-On-Read Queries:
==========================
Lets run similar queries against M-O-R table. Lets look at both
ReadOptimized and Snapshot queries supported by M-O-R table
# Run ReadOptimized Query. Notice that the latest timestamp is 10:29
scala> spark.sql("select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG'").show(100, false)
+------+-------------------+
|symbol|max(ts) |
+------+-------------------+
|GOOG |2018-08-31 10:29:00|
+------+-------------------+
# Run Snapshot Query. Notice that the latest timestamp is again 10:29
scala> spark.sql("select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG'").show(100, false)
+------+-------------------+
|symbol|max(ts) |
+------+-------------------+
|GOOG |2018-08-31 10:29:00|
+------+-------------------+
# Run Read Optimized and Snapshot project queries
scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_ro where symbol = 'GOOG'").show(100, false)
+-------------------+------+-------------------+------+---------+--------+
|_hoodie_commit_time|symbol|ts |volume|open |close |
+-------------------+------+-------------------+------+---------+--------+
|20250926135725397 |GOOG |2018-08-31 09:59:00|6330 |1230.5 |1230.02 |
|20250926135725397 |GOOG |2018-08-31 10:29:00|3391 |1230.1899|1230.085|
+-------------------+------+-------------------+------+---------+--------+
scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_rt where symbol = 'GOOG'").show(100, false)
+-------------------+------+-------------------+------+---------+--------+
|_hoodie_commit_time|symbol|ts |volume|open |close |
+-------------------+------+-------------------+------+---------+--------+
|20250926135725397 |GOOG |2018-08-31 09:59:00|6330 |1230.5 |1230.02 |
|20250926135725397 |GOOG |2018-08-31 10:29:00|3391 |1230.1899|1230.085|
+-------------------+------+-------------------+------+---------+--------+第 5 步：将第二批数据上传到 Kafka 并运行 Hudi Streamer 进行摄取
上传第二批数据，并使用 Hudi Streamer 摄取这批数据。由于该批次不会引入新的分区，因此无需运行 hive-sync。
cat demo/data/batch_2.json | kcat -b kafkabroker -t stock_ticks -P
# Within Docker container, run the ingestion command
docker exec -it adhoc-2 /bin/bash
# Run the following spark-submit command to execute the Hudi Streamer and ingest to stock_ticks_cow table in HDFS
spark-submit \
--class org.apache.hudi.utilities.streamer.HoodieStreamer $HUDI_UTILITIES_BUNDLE \
--table-type COPY_ON_WRITE \
--source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
--source-ordering-field ts \
--target-base-path /user/hive/warehouse/stock_ticks_cow \
--target-table stock_ticks_cow \
--props /var/demo/config/kafka-source.properties \
--schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider
# Run the following spark-submit command to execute the Hudi Streamer and ingest to stock_ticks_mor table in HDFS
spark-submit \
--class org.apache.hudi.utilities.streamer.HoodieStreamer $HUDI_UTILITIES_BUNDLE \
--table-type MERGE_ON_READ \
--source-class org.apache.hudi.utilities.sources.JsonKafkaSource \
--source-ordering-field ts \
--target-base-path /user/hive/warehouse/stock_ticks_mor \
--target-table stock_ticks_mor \
--props /var/demo/config/kafka-source.properties \
--schemaprovider-class org.apache.hudi.utilities.schema.FilebasedSchemaProvider \
--disable-compaction
exit对于 Copy-On-Write 表，Hudi Streamer 的第二次摄入会生成一个新版本的 Parquet 文件。请查看 http://namenode:9870/explorer.html#/user/hive/warehouse/stock_ticks_cow/2018/08/31
对于 Merge-On-Read 表，第二次摄入只是将该批次追加到一个未合并的增量（日志）文件中。可以通过查看 HDFS 文件系统来了解其情况：http://namenode:9870/explorer.html#/user/hive/warehouse/stock_ticks_mor/2018/08/31
步骤 6 (a)：运行 Hive 查询
对于 Copy-On-Write 表，由于每次摄入都会创建更新版本的 Parquet 文件，因此只要该批次提交成功，快照（Snapshot）查询就能立即看到第二次批次带来的变更。
对于 Merge-On-Read 表，第二次摄入只是将该批次追加到一个未合并的增量（日志）文件中。此时，ReadOptimized 查询和快照查询将返回不同的结果。ReadOptimized 查询仍会返回 "10:29 am"，因为它只会从 Parquet 文件中读取数据；而快照查询会进行即时合并，返回最新提交的数据，即 "10:59 a.m"。
docker exec -it adhoc-2 /bin/bash
beeline -u jdbc:hive2://hiveserver:10000 \
--hiveconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat \
--hiveconf hive.stats.autogather=false \
--hiveconf hive.vectorized.input.format.excludes=org.apache.hudi.hadoop.HoodieParquetInputFormat \
--hiveconf parquet.column.index.access=true
# Copy On Write Table:
0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG';
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
+---------+----------------------+--+
| symbol | _c1 |
+---------+----------------------+--+
| GOOG | 2018-08-31 10:59:00 |
+---------+----------------------+--+
1 row selected (1.932 seconds)
0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG';
+----------------------+---------+----------------------+---------+------------+-----------+--+
| _hoodie_commit_time | symbol | ts | volume | open | close |
+----------------------+---------+----------------------+---------+------------+-----------+--+
| 20250926135641514 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
| 20250926141521148 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
+----------------------+---------+----------------------+---------+------------+-----------+--+
As you can notice, the above queries now reflect the changes that came as part of ingesting second batch.
# Merge On Read Table:
# Read Optimized Query
0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG';
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
+---------+----------------------+--+
| symbol | _c1 |
+---------+----------------------+--+
| GOOG | 2018-08-31 10:29:00 |
+---------+----------------------+--+
1 row selected (1.6 seconds)
0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_ro where symbol = 'GOOG';
+----------------------+---------+----------------------+---------+------------+-----------+--+
| _hoodie_commit_time | symbol | ts | volume | open | close |
+----------------------+---------+----------------------+---------+------------+-----------+--+
| 20250926135725397 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
| 20250926135725397 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085 |
+----------------------+---------+----------------------+---------+------------+-----------+--+
# Snapshot Query
0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG';
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
+---------+----------------------+--+
| symbol | _c1 |
+---------+----------------------+--+
| GOOG | 2018-08-31 10:59:00 |
+---------+----------------------+--+
0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_rt where symbol = 'GOOG';
+----------------------+---------+----------------------+---------+------------+-----------+--+
| _hoodie_commit_time | symbol | ts | volume | open | close |
+----------------------+---------+----------------------+---------+------------+-----------+--+
| 20250926135725397 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
| 20250926141535482 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
+----------------------+---------+----------------------+---------+------------+-----------+--+
exit步骤 6(b)：运行 Spark SQL 查询
在 Spark SQL 中运行相同的查询：
docker exec -it adhoc-1 /bin/bash
$SPARK_INSTALL/bin/spark-shell \
--jars $HUDI_SPARK_BUNDLE \
--driver-class-path $HADOOP_CONF_DIR \
--conf spark.sql.hive.convertMetastoreParquet=false \
--deploy-mode client \
--driver-memory 1G \
--master local[2] \
--executor-memory 3G \
--num-executors 1
# Copy On Write Table:
scala> spark.sql("select symbol, max(ts) from stock_ticks_cow group by symbol HAVING symbol = 'GOOG'").show(100, false)
+------+-------------------+
|symbol|max(ts) |
+------+-------------------+
|GOOG |2018-08-31 10:59:00|
+------+-------------------+
scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG'").show(100, false)
+----------------------+---------+----------------------+---------+------------+-----------+--+
| _hoodie_commit_time | symbol | ts | volume | open | close |
+----------------------+---------+----------------------+---------+------------+-----------+--+
| 20250926135641514 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
| 20250926141521148 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
+----------------------+---------+----------------------+---------+------------+-----------+--+
As you can notice, the above queries now reflect the changes that came as part of ingesting second batch.
# Merge On Read Table:
# Read Optimized Query
scala> spark.sql("select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG'").show(100, false)
+---------+----------------------+
| symbol | _c1 |
+---------+----------------------+
| GOOG | 2018-08-31 10:29:00 |
+---------+----------------------+
1 row selected (1.6 seconds)
scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_ro where symbol = 'GOOG'").show(100, false)
+----------------------+---------+----------------------+---------+------------+-----------+
| _hoodie_commit_time | symbol | ts | volume | open | close |
+----------------------+---------+----------------------+---------+------------+-----------+
| 20250926135725397 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
| 20250926135725397 | GOOG | 2018-08-31 10:29:00 | 3391 | 1230.1899 | 1230.085 |
+----------------------+---------+----------------------+---------+------------+-----------+
# Snapshot Query
scala> spark.sql("select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG'").show(100, false)
+---------+----------------------+
| symbol | _c1 |
+---------+----------------------+
| GOOG | 2018-08-31 10:59:00 |
+---------+----------------------+
scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_rt where symbol = 'GOOG'").show(100, false)
+----------------------+---------+----------------------+---------+------------+-----------+
| _hoodie_commit_time | symbol | ts | volume | open | close |
+----------------------+---------+----------------------+---------+------------+-----------+
| 20250926135725397 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
| 20250926141535482 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
+----------------------+---------+----------------------+---------+------------+-----------+
exit步骤 7 (a)：COPY-ON-WRITE 表的增量查询
已摄取 2 批数据，接下来展示 Hudi Copy-On-Write 表对增量查询的支持。
我们沿用前面的投影查询示例。
docker exec -it adhoc-2 /bin/bash
beeline -u jdbc:hive2://hiveserver:10000 \
--hiveconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat \
--hiveconf hive.stats.autogather=false \
--hiveconf hive.vectorized.input.format.excludes=org.apache.hudi.hadoop.HoodieParquetInputFormat \
--hiveconf parquet.column.index.access=true
0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG';
+----------------------+---------+----------------------+---------+------------+-----------+--+
| _hoodie_commit_time | symbol | ts | volume | open | close |
+----------------------+---------+----------------------+---------+------------+-----------+--+
| 20250926135641514 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
| 20250926141521148 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
+----------------------+---------+----------------------+---------+------------+-----------+--+从上面的查询中可以看到，按时间线顺序有 2 个提交：20250926135641514 和 20250926141521148。当你按照步骤操作时，会得到不同的提交时间戳，请用实际的时间戳替换上述时间戳。
为了展示增量查询的效果，我们假设读者在摄入第一批数据时已经看到了这些变更。现在，为了让读者看到第二批数据的效果，需要将起始时间戳设置为第一批数据的提交时间（20250926135641514），然后运行增量查询。
Hudi 的增量模式借助 Hudi 托管的元数据过滤掉不包含任何候选行的文件，从而为增量查询提供高效的扫描。
docker exec -it adhoc-2 /bin/bash
beeline -u jdbc:hive2://hiveserver:10000 \
--hiveconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat \
--hiveconf hive.stats.autogather=false \
--hiveconf hive.vectorized.input.format.excludes=org.apache.hudi.hadoop.HoodieParquetInputFormat \
--hiveconf parquet.column.index.access=true
0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_cow.consume.mode=INCREMENTAL;
No rows affected (0.009 seconds)
0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_cow.consume.max.commits=3;
No rows affected (0.009 seconds)
0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_cow.consume.start.timestamp=20250926135641514;采用上述设置，未在提交 20250926141521148 中产生任何更新的 file-id 将在不进行扫描的情况下被过滤掉。以下是增量查询：
0: jdbc:hive2://hiveserver:10000>
0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow where symbol = 'GOOG' and `_hoodie_commit_time` > '20250926135641514';
+----------------------+---------+----------------------+---------+------------+-----------+--+
| _hoodie_commit_time | symbol | ts | volume | open | close |
+----------------------+---------+----------------------+---------+------------+-----------+--+
| 20250926141521148 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
+----------------------+---------+----------------------+---------+------------+-----------+--+
1 row selected (0.83 seconds)
0: jdbc:hive2://hiveserver:10000>第 7 步 (b)：使用 Spark SQL 进行增量查询：
docker exec -it adhoc-1 /bin/bash
$SPARK_INSTALL/bin/spark-shell \
--jars $HUDI_SPARK_BUNDLE \
--driver-class-path $HADOOP_CONF_DIR \
--conf spark.sql.hive.convertMetastoreParquet=false \
--deploy-mode client \
--driver-memory 1G \
--master local[2] \
--executor-memory 3G \
--num-executors 1
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 3.5.3
/_/
Using Scala version 2.12.18 (OpenJDK 64-Bit Server VM, Java 1.8.0_342)
Type in expressions to have them evaluated.
Type :help for more information.
# In the below query, 20250926135641514 is the first commit's timestamp
scala> val hoodieIncViewDF = spark.read.format("org.apache.hudi").option("hoodie.datasource.query.type", "incremental").option("hoodie.datasource.read.begin.instanttime", "20250926135641514").load("/user/hive/warehouse/stock_ticks_cow")
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes#StaticLoggerBinder for further details.
hoodieIncViewDF: org.apache.spark.sql.DataFrame = [_hoodie_commit_time: string, _hoodie_commit_seqno: string ... 15 more fields]
scala> hoodieIncViewDF.registerTempTable("stock_ticks_cow_incr_tmp1")
warning: there was one deprecation warning; re-run with -deprecation for details
scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_cow_incr_tmp1 where symbol = 'GOOG'").show(100, false);
+----------------------+---------+----------------------+---------+------------+-----------+
| _hoodie_commit_time | symbol | ts | volume | open | close |
+----------------------+---------+----------------------+---------+------------+-----------+
| 20250926141521148 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
+----------------------+---------+----------------------+---------+------------+-----------+第 8 步：为 Merge-On-Read 表调度并运行 Compaction
让我们调度并运行一次 compaction，以生成新版本的列式文件，这样读优化型读者就能看到更新鲜的数据。同样，你可以使用 Hudi CLI 手动调度并运行 compaction。
docker exec -it adhoc-1 /bin/bash
root@adhoc-1:/opt# /var/hoodie/ws/packaging/hudi-cli-bundle/hudi-cli-with-bundle.sh
...
Table command getting loaded
HoodieSplashScreen loaded
===================================================================
* ___ ___ *
* /\__\ ___ /\ \ ___ *
* / / / /\__\ / \ \ /\ \ *
* / /__/ / / / / /\ \ \ \ \ \ *
* / \ \ ___ / / / / / \ \__\ / \__\ *
* / /\ \ /\__\ / /__/ ___ / /__/ \ |__| / /\/__/ *
* \/ \ \/ / / \ \ \ /\__\ \ \ \ / / / /\/ / / *
* \ / / \ \ / / / \ \ / / / \ /__/ *
* / / / \ \/ / / \ \/ / / \ \__\ *
* / / / \ / / \ / / \/__/ *
* \/__/ \/__/ \/__/ Apache Hudi CLI *
* *
===================================================================
Welcome to Apache Hudi CLI. Please type help if you are looking for help.
hudi->connect --path /user/hive/warehouse/stock_ticks_mor
14512 [main] WARN org.apache.hadoop.util.NativeCodeLoader [] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14711 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient [] - Loading HoodieTableMetaClient from /user/hive/warehouse/stock_ticks_mor
14711 [main] INFO org.apache.hudi.common.table.HoodieTableConfig [] - Loading table properties from /user/hive/warehouse/stock_ticks_mor/.hoodie/hoodie.properties
14855 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient [] - Finished Loading Table of type MERGE_ON_READ(version=2) from /user/hive/warehouse/stock_ticks_mor
Metadata for table stock_ticks_mor loaded
hoodie:stock_ticks_mor->compactions show all
73614 [main] INFO org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2 [] - Loaded instants upto : Option{val=[20250926141535482__20250926141539083__deltacommit__COMPLETED]}
╔═════════════════════════╤═══════╤═══════════════════════════════╗
║ Compaction Instant Time │ State │ Total FileIds to be Compacted ║
╠═════════════════════════╧═══════╧═══════════════════════════════╣
║ (empty) ║
╚═════════════════════════════════════════════════════════════════╝
# Schedule a compaction. This will use Spark Launcher to schedule compaction
hoodie:stock_ticks_mor->compaction schedule --hoodieConfigs hoodie.compact.inline.max.delta.commits=1
....
Attempted to schedule compaction for stock_ticks_mor
# Now refresh and check again. You will see that there is a new compaction requested
hoodie:stock_ticks_mor->refresh
185420 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient [] - Loading HoodieTableMetaClient from /user/hive/warehouse/stock_ticks_mor
185420 [main] INFO org.apache.hudi.common.table.HoodieTableConfig [] - Loading table properties from /user/hive/warehouse/stock_ticks_mor/.hoodie/hoodie.properties
185443 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient [] - Finished Loading Table of type MERGE_ON_READ(version=2) from /user/hive/warehouse/stock_ticks_mor
Metadata for table stock_ticks_mor refreshed.
hoodie:stock_ticks_mor->compactions show all
216313 [main] INFO org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2 [] - Loaded instants upto : Option{val=[==>20250926143925260__compaction__REQUESTED]}
╔═════════════════════════╤═══════════╤═══════════════════════════════╗
║ Compaction Instant Time │ State │ Total FileIds to be Compacted ║
╠═════════════════════════╪═══════════╪═══════════════════════════════╣
║ 20250926143925260 │ REQUESTED │ 1 ║
╚═════════════════════════╧═══════════╧═══════════════════════════════╝
# Execute the compaction. The compaction instant value passed below must be the one displayed in the above "compactions show all" query
hoodie:stock_ticks_mor->compaction run --compactionInstant 20250926143925260 --parallelism 2 --sparkMemory 1G --schemaFilePath /var/demo/config/schema.avsc --retry 1
....
Compaction successfully completed for 20250926143925260
## Now check if compaction is completed
hoodie:stock_ticks_mor->refresh
282367 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient [] - Loading HoodieTableMetaClient from /user/hive/warehouse/stock_ticks_mor
282367 [main] INFO org.apache.hudi.common.table.HoodieTableConfig [] - Loading table properties from /user/hive/warehouse/stock_ticks_mor/.hoodie/hoodie.properties
282383 [main] INFO org.apache.hudi.common.table.HoodieTableMetaClient [] - Finished Loading Table of type MERGE_ON_READ(version=2) from /user/hive/warehouse/stock_ticks_mor
Metadata for table stock_ticks_mor refreshed.
hoodie:stock_ticks_mor->compactions show all
298704 [main] INFO org.apache.hudi.common.table.timeline.versioning.v2.ActiveTimelineV2 [] - Loaded instants upto : Option{val=[20250926143925260__20250926144127165__commit__COMPLETED]}
╔═════════════════════════╤═══════════╤═══════════════════════════════╗
║ Compaction Instant Time │ State │ Total FileIds to be Compacted ║
╠═════════════════════════╪═══════════╪═══════════════════════════════╣
║ 20250926143925260 │ COMPLETED │ 1 ║
╚═════════════════════════╧═══════════╧═══════════════════════════════╝第 9 步：运行 Hive 查询，包括增量查询
你会发现 ReadOptimized（读优化）查询和 Snapshot（快照）查询显示的都是最新提交的数据。接下来，我们再为 MOR 表运行一次增量查询。从下面的查询输出中可以清楚地看到，该 MOR 表的首次提交时间为 20250926135725397，第二次提交时间为 20250926141535482。
docker exec -it adhoc-2 /bin/bash
beeline -u jdbc:hive2://hiveserver:10000 \
--hiveconf hive.input.format=org.apache.hadoop.hive.ql.io.HiveInputFormat \
--hiveconf hive.stats.autogather=false \
--hiveconf hive.vectorized.input.format.excludes=org.apache.hudi.hadoop.HoodieParquetInputFormat \
--hiveconf parquet.column.index.access=true
# Read Optimized Query
0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG';
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
+---------+----------------------+--+
| symbol | _c1 |
+---------+----------------------+--+
| GOOG | 2018-08-31 10:59:00 |
+---------+----------------------+--+
1 row selected (1.6 seconds)
0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_ro where symbol = 'GOOG';
+----------------------+---------+----------------------+---------+------------+-----------+--+
| _hoodie_commit_time | symbol | ts | volume | open | close |
+----------------------+---------+----------------------+---------+------------+-----------+--+
| 20250926135725397 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
| 20250926141535482 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
+----------------------+---------+----------------------+---------+------------+-----------+--+
# Snapshot Query
0: jdbc:hive2://hiveserver:10000> select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG';
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the future versions. Consider using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
+---------+----------------------+--+
| symbol | _c1 |
+---------+----------------------+--+
| GOOG | 2018-08-31 10:59:00 |
+---------+----------------------+--+
0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_rt where symbol = 'GOOG';
+----------------------+---------+----------------------+---------+------------+-----------+--+
| _hoodie_commit_time | symbol | ts | volume | open | close |
+----------------------+---------+----------------------+---------+------------+-----------+--+
| 20250926135725397 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
| 20250926141535482 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
+----------------------+---------+----------------------+---------+------------+-----------+--+
# Incremental Query:
0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_mor.consume.mode=INCREMENTAL;
No rows affected (0.008 seconds)
# Max-Commits covers both second batch and compaction commit
0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_mor.consume.max.commits=3;
No rows affected (0.007 seconds)
0: jdbc:hive2://hiveserver:10000> set hoodie.stock_ticks_mor.consume.start.timestamp=20250926135725397;
No rows affected (0.013 seconds)
# Query:
0: jdbc:hive2://hiveserver:10000> select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_ro where symbol = 'GOOG' and `_hoodie_commit_time` > '20250926135725397';
+----------------------+---------+----------------------+---------+------------+-----------+--+
| _hoodie_commit_time | symbol | ts | volume | open | close |
+----------------------+---------+----------------------+---------+------------+-----------+--+
| 20250926141535482 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
+----------------------+---------+----------------------+---------+------------+-----------+--+
exit步骤 10：compaction 后使用 Spark-SQL 对 MOR 表执行 Read Optimized 和 Snapshot 查询
docker exec -it adhoc-1 /bin/bash
$SPARK_INSTALL/bin/spark-shell \
--jars $HUDI_SPARK_BUNDLE \
--driver-class-path $HADOOP_CONF_DIR \
--conf spark.sql.hive.convertMetastoreParquet=false \
--deploy-mode client \
--driver-memory 1G \
--master local[2] \
--executor-memory 3G \
--num-executors 1
# Read Optimized Query
scala> spark.sql("select symbol, max(ts) from stock_ticks_mor_ro group by symbol HAVING symbol = 'GOOG'").show(100, false)
+---------+----------------------+
| symbol | max(ts) |
+---------+----------------------+
| GOOG | 2018-08-31 10:59:00 |
+---------+----------------------+
scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_ro where symbol = 'GOOG'").show(100, false)
+----------------------+---------+----------------------+---------+------------+-----------+
| _hoodie_commit_time | symbol | ts | volume | open | close |
+----------------------+---------+----------------------+---------+------------+-----------+
| 20250926135725397 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
| 20250926141535482 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
+----------------------+---------+----------------------+---------+------------+-----------+
# Snapshot Query
scala> spark.sql("select symbol, max(ts) from stock_ticks_mor_rt group by symbol HAVING symbol = 'GOOG'").show(100, false)
+---------+----------------------+
| symbol | max(ts) |
+---------+----------------------+
| GOOG | 2018-08-31 10:59:00 |
+---------+----------------------+
scala> spark.sql("select `_hoodie_commit_time`, symbol, ts, volume, open, close from stock_ticks_mor_rt where symbol = 'GOOG'").show(100, false)
+----------------------+---------+----------------------+---------+------------+-----------+
| _hoodie_commit_time | symbol | ts | volume | open | close |
+----------------------+---------+----------------------+---------+------------+-----------+
| 20250926135725397 | GOOG | 2018-08-31 09:59:00 | 6330 | 1230.5 | 1230.02 |
| 20250926141535482 | GOOG | 2018-08-31 10:59:00 | 9021 | 1227.1993 | 1227.215 |
+----------------------+---------+----------------------+---------+------------+-----------+至此，演示结束。
在本地 Docker 环境中测试 Hudi
您可以启动一个包含 Hadoop、Hive 和 Spark 服务的 Hadoop Docker 环境，该环境支持 Hudi。
$ mvn pre-integration-test -DskipTests上述命令会为所有服务构建 Docker 镜像，其中安装了位于 /var/hoodie/ws 的当前 Hudi 源码，同时使用 compose 文件启动这些服务。目前 Docker 镜像中使用的是 Hadoop（v3.3.4）、Hive（v3.1.3）和 Spark（v3.5.3）。
要停止这些容器
$ cd hudi-integ-test
$ mvn docker-compose:down如果你想启动这些 Docker 容器，请使用
$ cd hudi-integ-test
$ mvn docker-compose:up -DdetachedMode=trueHudi 是一个在更广泛的数据分析/数据接入环境中运行的库，该环境涉及 Hadoop、Hive 和 Spark。与所有这些系统实现互操作是我们的一项关键目标。我们正在 hudi-integ-test/src/test/java 下积极添加利用该 Docker 环境的集成测试（参见 hudi-integ-test/src/test/java/org/apache/hudi/integ/ITTestHoodieSanity.java）。
构建本地 Docker 容器：
演示和运行集成测试所需的 Docker 镜像已经发布在 docker-hub 上。这些 Docker 镜像和 compose 脚本都经过精心设计，以实现双重用途：
- Docker 镜像中内置了 Hudi 的 jar 文件，并有环境变量指向这些 jar（HUDI_HADOOP_BUNDLE 等）
- 为了运行集成测试，我们需要使用本地生成的 jar 来在 Docker 内运行各项服务。docker-compose 脚本（参见
docker/compose/docker-compose_hadoop334_hive313_spark353_arm64.yml）通过将本地 Hudi 工作空间挂载到 Docker 中的对应位置，确保本地 jar 覆盖内置 jar - 由于这些 Docker 容器挂载了本地 Hudi 工作空间，工作空间中发生的任何更改都会自动反映到容器中。这对于没有分布式环境的开发者来说，是一种开发和验证 Hudi 的便捷方式。请注意，集成测试正是以这种方式运行的。
这样可以避免维护多套独立的 Docker 镜像，也省去了在本地构建 Hudi Docker 镜像这一耗时耗力的步骤。不过，如果用户希望在带宽较低的网络环境下测试 Hudi，也可以构建本地镜像：先运行脚本 docker/build_local_docker_images.sh 构建本地 Docker 镜像，然后再运行 docker/setup_demo.sh。
以下是相关命令：
cd docker
./build_local_docker_images.sh
.....
[INFO] Reactor Summary:
[INFO]
[INFO] Hudi ............................................... SUCCESS [ 2.507 s]
[INFO] hudi-common ........................................ SUCCESS [ 15.181 s]
[INFO] hudi-aws ........................................... SUCCESS [ 2.621 s]
[INFO] hudi-timeline-service .............................. SUCCESS [ 1.811 s]
[INFO] hudi-client ........................................ SUCCESS [ 0.065 s]
[INFO] hudi-client-common ................................. SUCCESS [ 8.308 s]
[INFO] hudi-hadoop-mr ..................................... SUCCESS [ 3.733 s]
[INFO] hudi-spark-client .................................. SUCCESS [ 18.567 s]
[INFO] hudi-sync-common ................................... SUCCESS [ 0.794 s]
[INFO] hudi-hive-sync ..................................... SUCCESS [ 3.691 s]
[INFO] hudi-spark-datasource .............................. SUCCESS [ 0.121 s]
[INFO] hudi-spark-common_2.12 ............................. SUCCESS [ 12.979 s]
[INFO] hudi-spark2_2.12 ................................... SUCCESS [ 12.516 s]
[INFO] hudi-spark_2.12 .................................... SUCCESS [ 35.649 s]
[INFO] hudi-utilities_2.12 ................................ SUCCESS [ 5.881 s]
[INFO] hudi-utilities-bundle_2.12 ......................... SUCCESS [ 12.661 s]
[INFO] hudi-cli ........................................... SUCCESS [ 19.858 s]
[INFO] hudi-java-client ................................... SUCCESS [ 3.221 s]
[INFO] hudi-flink-client .................................. SUCCESS [ 5.731 s]
[INFO] hudi-spark3_2.12 ................................... SUCCESS [ 8.627 s]
[INFO] hudi-dla-sync ...................................... SUCCESS [ 1.459 s]
[INFO] hudi-sync .......................................... SUCCESS [ 0.053 s]
[INFO] hudi-hadoop-mr-bundle .............................. SUCCESS [ 5.652 s]
[INFO] hudi-hive-sync-bundle .............................. SUCCESS [ 1.623 s]
[INFO] hudi-spark-bundle_2.12 ............................. SUCCESS [ 10.930 s]
[INFO] hudi-presto-bundle ................................. SUCCESS [ 3.652 s]
[INFO] hudi-timeline-server-bundle ........................ SUCCESS [ 4.804 s]
[INFO] hudi-trino-bundle .................................. SUCCESS [ 5.991 s]
[INFO] hudi-hadoop-docker ................................. SUCCESS [ 2.061 s]
[INFO] hudi-hadoop-base-docker ............................ SUCCESS [ 53.372 s]
[INFO] hudi-hadoop-base-java11-docker ..................... SUCCESS [ 48.545 s]
[INFO] hudi-hadoop-namenode-docker ........................ SUCCESS [ 6.098 s]
[INFO] hudi-hadoop-datanode-docker ........................ SUCCESS [ 4.825 s]
[INFO] hudi-hadoop-history-docker ......................... SUCCESS [ 3.829 s]
[INFO] hudi-hadoop-hive-docker ............................ SUCCESS [ 52.660 s]
[INFO] hudi-hadoop-sparkbase-docker ....................... SUCCESS [01:02 min]
[INFO] hudi-hadoop-sparkmaster-docker ..................... SUCCESS [ 12.661 s]
[INFO] hudi-hadoop-sparkworker-docker ..................... SUCCESS [ 4.350 s]
[INFO] hudi-hadoop-sparkadhoc-docker ...................... SUCCESS [ 59.083 s]
[INFO] hudi-hadoop-presto-docker .......................... SUCCESS [01:31 min]
[INFO] hudi-hadoop-trinobase-docker ....................... SUCCESS [02:40 min]
[INFO] hudi-hadoop-trinocoordinator-docker ................ SUCCESS [ 14.003 s]
[INFO] hudi-hadoop-trinoworker-docker ..................... SUCCESS [ 12.100 s]
[INFO] hudi-integ-test .................................... SUCCESS [ 13.581 s]
[INFO] hudi-integ-test-bundle ............................. SUCCESS [ 27.212 s]
[INFO] hudi-examples ...................................... SUCCESS [ 8.090 s]
[INFO] hudi-flink_2.12 .................................... SUCCESS [ 4.217 s]
[INFO] hudi-kafka-connect ................................. SUCCESS [ 2.966 s]
[INFO] hudi-flink-bundle_2.12 ............................. SUCCESS [ 11.155 s]
[INFO] hudi-kafka-connect-bundle .......................... SUCCESS [ 12.369 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 14:35 min
[INFO] Finished at: 2025-09-26T18:41:27-08:00
[INFO] ------------------------------------------------------------------------评论
登录后参与评论
正在加载评论…