Concepts, Readings, Events#

🧭 Background Concepts#

• 2024-06-13: 2024 ACM SIGMOD International Conference on Management of Data: Apache Arrow DataFusion: A Fast, Embeddable, Modular Analytic Query Engine - Download, Talk, Slides, Recording
• 2024-06-07: Video: SIGMOD 2024 Practice: Apache Arrow DataFusion A Fast, Embeddable, Modular Analytic Query Engine - Slides
• 2023-04-05: Video: DataFusion Architecture Part 3: Physical Plan and Execution - Slides
• 2023-04-04: Video: DataFusion Architecture Part 2: Logical Plans and Expressions - Slides
• 2023-03-31: Video: DataFusion Architecture Part 1: Query Engines - Slides
• 2020-02-27: Online Book: How Query Engines Work

✨ Good Reads#

This is a list of DataFusion related blog posts, articles, and other resources. Please open a PR to add any new resources you create or find

• 2026-07-29 Blog: Bringing DuckLake to DataFusion
• 2026-07-20 Blog: Optimizing for Almost Sorted Data: Sort Pushdown in Apache DataFusion
• 2026-06-03 Blog: What two months with the Comet community got our Spark workload on Amazon EKS
• 2026-05-28 Blog: Writing Agent Skills for an Open Source Project: Lessons from DataFusion Python
• 2026-05-24 Blog: Building Composable Query Engines with Rust Runtimes
• 2026-05-23 Blog: Single-Node Data Engineering: DuckDB, DataFusion, Polars, and LakeSail
• 2026-04-10 Blog: DataFusion and the Rise of Deconstructed Data Systems
• 2026-03-31 Blog: Writing Custom Table Providers in Apache DataFusion
• 2026-03-20 Blog: Turning LIMIT into an I/O Optimization: Inside DataFusion’s Multi-Layer Pruning Stack
• 2026-02-23 Blog: Apache DataFusion: A Data Engineer’s Guide to the Query Engine Reshaping How We Build Data Systems
• 2026-02-09 Blog: Vector search using only Parquet and DataFusion
• 2026-02-02 Blog: Optimizing SQL CASE Expression Evaluation
• 2026-01-12 Blog: Extending SQL in DataFusion: from ->> to TABLESAMPLE
• 2025-12-15 Blog: Optimizing Repartitions in DataFusion: How I Went From Database Noob to Core Contribution
• 2025-12-09 Blog: Apache DataFusion: The Quiet Engine Powering the Next Generation of Data Systems
• 2025-09-21 Blog: Implementing User Defined Types and Custom Metadata in DataFusion
• 2025-09-10 Blog: Dynamic Filters: Passing Information Between Operators During Execution for 25x Faster Queries
• 2025-08-15 Blog: Using External Indexes, Metadata Stores, Catalogs and Caches to Accelerate Queries on Apache Parquet
• 2025-07-22 Blog: Why Startups Are Betting Everything on Apache DataFusion
• 2025-07-14 Blog: Embedding User-Defined Indexes in Apache Parquet Files
• 2025-06-30 Blog: Using Rust async for Query Execution and Cancelling Long-Running Queries
• 2025-06-15 Blog: Optimizing SQL (and DataFrames) in DataFusion, Part 1: Query Optimization Overview
• 2025-06-15 Blog: Optimizing SQL (and DataFrames) in DataFusion, Part 2: Optimizers in Apache DataFusion
• 2025-04-19 Blog: User defined Window Functions in DataFusion
• 2025-04-10 Blog: tpchgen-rs World’s fastest open source TPC-H data generator, written in Rust
• 2025-03-11 Blog: Using Ordering for Better Plans in Apache DataFusion
• 2024-05-07 Blog: Announcing Apache Arrow DataFusion is now Apache DataFusion
• 2024-03-06 Blog: Announcing Apache Arrow DataFusion Comet
• 2025-03-21 Blog: Efficient Filter Pushdown in Parquet
• 2025-03-20 Blog: Parquet Pruning in DataFusion: Read Only What Matters
• 2024-11-22 Blog: Apache Datafusion Comet and the story of my first contribution to it
• 2024-11-21 Blog: DataFusion is featured as one of the coolest 10 open source software tools by CRN
• 2024-11-20 Blog: Apache DataFusion Comet 0.4.0 Release
• 2024-11-19 Blog: Comparing approaches to User Defined Functions in Apache DataFusion using Python
• 2024-11-18 Blog: Apache DataFusion is now the fastest single node engine for querying Apache Parquet files
• 2024-11-18 Blog: Building Databases over a Weekend
• 2024-10-27 Blog: Caching in DataFusion: Don’t read twice
• 2024-10-24 Blog: Parquet pruning in DataFusion: Read no more than you need
• 2024-09-13 Blog: Using StringView / German Style Strings to make Queries Faster: Part 2 - String Operations | Reposted on DataFusion Blog
• 2024-09-13 Blog: Using StringView / German Style Strings to Make Queries Faster: Part 1- Reading Parquet | Reposted on Datafusion Blog
• 2024-08-25 Blog: Pydantic/logfire: We’re changing database
• 2024-08-14 Blog: DataFusion @ UWheel
• 2024-06-17 Blog: Columnar File Readers In-Depth: APIs and Fusion
• 2024-06-14 Talk: 2024 Simplicity in Management of Data (SiMOD): DataFusion: The Case for Building Open Data Systems (Keynote) - Slides
• 2024-05-29 Blog: Query Push Down in Cube’s Semantic Layer
• 2024-06-26 Talk: Microsoft Gray Systems Lab: Building InfluxDB 3.0 (and other systems) - Slides
• 2024-03-26 Talk: DataCouncil 2024: Building InfluxDB 3.0 with Apache Arrow, DataFusion, Flight, and Parquet - Slides, Recording
• 2024-03-18 Blog: Making Recent Value Queries Hundreds of Times Faster
• 2023-10-25 Blog: Flight, DataFusion, Arrow, and Parquet: Using the FDAP Architecture to build InfluxDB 3.0
• 2023-09-26 Blog: 100x Faster Ingest with DataFusion + Better Connectivity with FlightSQL
• 2023-08-15 Blog: Running Window Query in Stream Processing
• 2023-08-05 Blog: Aggregating Millions of Groups Fast in Apache Arrow DataFusion | DataFusion Blog
• 2023-07-28 Blog: Sliding Window Hash Join (SWHJ)
• 2023-07-13 Blog: Probabilistic Data Structures in Streaming: Count-Min Sketch
• 2023-02-20 Blog: General Purpose Stream Joins via Pruning Symmetric Hash Joins
• 2023-09-27 Slides: MIT Database Group: Implementing InfluxDB IOx
• 2023-06-02 Talk: Dutch Seminar on Database System Design: Implementing InfluxDB IOx - Slides, Recording
• 2023-02-15 Slides: Invited Talk at Optum Labs: Building a New Time Series Database
• 2023-01-01 Blog: What I Want from DataFusion 2023
• 2022-12-07 Blog: Querying Parquet with Millisecond Latency
• 2022-06-27 Talk: DataBricks Data+AI Summit: DataFusion and Arrow - Slides, Recording

🎥 Videos#

• 2026-04-04 Video: Generalized Consensus & Native Top-K Joins in ParadeDB

• 2026-03-24 Podcast: The Data Fusion Secret & Why Custom Query Engines Fail with Nikita Lapkov

• 2025-02-12 Video: Alex Kesling on Apache Arrow DataFusion - Papers We Love NYC

• 2025-01-30 Video: Data & Drinks: Building Next-Gen Data Systems with Apache DataFusion

• 2024-10-29 Video: MiDAS Seminar Fall 2024 on “Apache DataFusion” by Andrew Lamb

• 2024-09-23 → 2024-12-02 Talks: Carnegie Mellon University: Database Building Blocks Seminar Series - Fall 2024

  • 2024-11-12 Video: Building InfluxDB 3.0 with the FDAP Stack: Apache Flight, DataFusion, Arrow and Parquet (Paul Dix)
  • 2024-11-04 Video: Synnada: Towards “Unified” Compute Engines: Opportunities and Challenges (Mehmet Ozan Kabak)
  • 2024-10-28 Video: Exon: A Built for Purpose Bioinformatics Database (Trent Hauck)
  • 2024-10-21 Video: Accelerating Data and AI with Spice.ai Open-Source Software (Luke Kim)
  • 2024-10-07 Video: ParadeDB – Postgres for Search and Analytics (Philippe Noël)
  • 2024-09-30 Video: Accelerating Apache Spark Workloads with Apache DataFusion Comet (Andy Grove)
  • 2024-09-23 Video: Apache Arrow DataFusion: A Fast, Embeddable, Modular Analytic Query Engine (Andrew Lamb)

• 2024-09-17 Video: Profiling Apache DataFusion using flamegraph

• 2024-08-15 Video: Faster DataFusion with StringView - Xiangpeng Hao (Aug 15, 2024)

• 2024-04-06 Video: 1 billion row challenge in Rust using Apache Arrow

• 2024-03-20 Video: Profiling DataFusion with Instruments (part of XCode on Mac OSx)

• 2023-05-25 Video: D3L2: Discussing Rust, Ballista, Ray SQL, Data Fusion with Andy Grove

• 2022-05-23 Video: The Data Thread 2022: Apache Arrow and DataFusion - Slides

• 2021-03-10 Video: InfluxData Tech Talk: Query Engine Design and Rust-Based DataFusion in Apache Arrow - Slides

📅 Release Notes & Updates#

• 2026-08-09 Apache DataFusion Ballista 54.1.0 Released
• 2026-08-07 Apache DataFusion Comet 1.0.0 Release
• 2026-07-12 Apache DataFusion Ballista 54.0.0 Released
• 2026-06-20 Apache DataFusion Comet 0.17.0 Release
• 2026-06-12 Apache DataFusion 54.0.0 Released
• 2026-05-26 Introducing Apache DataFusion Java 0.1.0
• 2026-05-24 Apache DataFusion Ballista 53.0.0 Released
• 2026-05-07 Apache DataFusion Comet 0.16.0 Release
• 2026-04-18 Apache DataFusion Comet 0.15.0 Release
• 2026-04-02 Apache DataFusion 53.0.0 Released
• 2026-03-18 Apache DataFusion Comet 0.14.0 Release
• 2026-01-30 Apache DataFusion Comet 0.13.0 Release
• 2026-01-12 Apache DataFusion 52.0.0 Released
• Older release announcements can be found on the Apache DataFusion Blog

🎥 Community Showcase#

The DataFusion Community Showcase is a regular virtual event where community members share what they are building with DataFusion.

• 2026-08-06 Vol. 3: ASAPQuery (Milind Srivastava) & Streamling (Yaroslav Tkachenko)
• 2026-07-23 Vol. 2: DataFusion Comet (Jordan Epstein) & DataFusion Ballista (Phillip LeBlanc)
• 2026-07-09 Vol. 1: SedonaDB (Dewey Dunnington) & Xarray-SQL (Alex Merose)

🌎 Community Events#

• 2026-09-30 London Apache DataFusion Meetup - RSVP
• 2026-09-03 Boston Apache DataFusion Meetup - RSVP
• 2026-08-13 San Francisco Apache DataFusion Meetup - RSVP
• 2026-07-22 Denver Apache DataFusion Meetup - RSVP
• 2026-06-28 Shanghai Apache DataFusion Meetup - RSVP, LinkedIn
• 2026-05-12 New York City Apache DataFusion Meetup - RSVP
• 2026-05-11 San Francisco Apache DataFusion Meetup - RSVP
• 2026-04-23 Seattle Apache DataFusion Meetup - RSVP
• 2026-04-22 Portland Apache DataFusion Meetup - RSVP
• 2026-03-05 Stockholm Apache DataFusion Meetup - RSVP, Recording
• 2026-02-19 San Francisco Apache DataFusion Meetup - RSVP, Recording
• 2025-11-12 Boston Apache DataFusion Meetup - Recording, Slides
• 2025-09-15 New York City Apache DataFusion Meetup - RSVP, Recording
• 2025-01-23 Amsterdam Apache DataFusion Meetup - Slides
• 2025-01-22 Datadog Apache DataFusion Community Meeting - Recording
• 2025-01-15 Boston Apache DataFusion Meetup - Slides
• 2024-12-18 Chicago Apache DataFusion Meetup - Slides, Recording
• 2024-10-14 Seattle Apache DataFusion Meetup
• 2024-09-27 Belgrade Apache DataFusion Meetup - Recap, Slides, Recording
• 2024-06-26 New York City Apache DataFusion Meetup - Slides
• 2024-06-25 San Francisco Bay Area Apache DataFusion Meetup - Slides, Recording
• 2024-03-27 Austin Apache DataFusion Meetup - Slides, Recording