Apache Hamilton (incubating) is a general-purpose framework to write dataflows using regular Python functions. At the core, each function defines a transformation and its parameters indicates its dependencies. Apache Hamilton automatically connects individual functions into a Directed Acyclic Graph (DAG) that can be executed, visualized, optimized, and reported on. Apache Hamilton also comes with a UI to visualize, catalog, and monitor your dataflows.

Apache Hamilton (incubating)

# Using within your ETL System

Conceptually you can integrate Apache Hamilton within your existing ETL system quite easily:

## Compatibility Matrix

| Framework / Scheduler                                                    | Compatibility                                                                                                                                                                                                                          |
|--------------------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| [Airflow](http://airflow.org)                                            | ✅ (see \[airflow example]([https://github.com/apache/hamilton/tree/main/examples/airflow](https://github.com/apache/hamilton/tree/main/examples/airflow)))                                                                             |
| [Dagster](https://dagster.io)                                            | ✅                                                                                                                                                                                                                                      |
| [Prefect](https://prefect.io)                                            | ✅ (see \[prefect example]([https://github.com/apache/hamilton/tree/main/examples/prefect](https://github.com/apache/hamilton/tree/main/examples/prefect)))                                                                             |
| [Kubeflow Pipelines](https://www.kubeflow.org/docs/components/pipelines) | ✅                                                                                                                                                                                                                                      |
| [CRON](https://en.wikipedia.org/wiki/Cron)                               | ✅                                                                                                                                                                                                                                      |
| [dbt](https://getdbt.com)                                                | ✅ (see dbt example)                                                                                                                                                                                                                    |
| [kubernetes](https://kubernetes.io)                                      | ✅ but you need to setup kubernetes to run an image that can run python code - e.g. see [Running a python application on kubernetes](https://medium.com/avmconsulting-blog/running-a-python-application-on-kubernetes-aws-56609e7cd88c) |
| [docker](https://www.docker.com)                                         | ✅ but you need to setup a docker image that can execute python code.                                                                                                                                                                   |
| … in general if it runs python 3.7+ …                                    | ✅                                                                                                                                                                                                                                      |

Title

## ETL Recipe

1. Write Hamilton functions & “driver” code.
2. Publish your Hamilton functions in a package, or import via other means (e.g. checkout a repository & include in python path).
3. Include apache-hamilton as a python dependency
4. Have your ETL system execute your “driver” code.
5. Profit.

Using within your ETL System

KnowForge 新版本 2026.0.6 正式发布！！！

产品

资源

其他

Why use Apache Hamilton?

Install

Your First Dataflow

Learning Resources

Contributing

License

Get Started

Glossary

Functions, nodes & dataflow

Driver

Visualization

Materialization

Function modifiers

Builder

Caching

Function modifiers (Advanced)

Dynamic DAGs/Parallel Execution

UI Overview

Function Naming

Migrating to Apache Hamilton

Code Organization

Common Indices

Output Immutability

Loading Data

Best Practices

Concepts

Jupyter notebooks

Loading data

Feature engineering

Reusing functions and nodes

Model training

LLM workflows

Data quality

Testing Apache Hamilton code

Lineage + Apache Hamilton

Scaling computation

Microservice

Extension autoloading

Wrapping the Driver

Command line interface

pre-commit hooks

User Guide

Apache Hamilton UI

Apache Hamilton VSCode

Language Server

IDE extension

FastAPI

Ibis

Streamlit

Integrations

Kedro

Dagster

LangChain

Airflow

Code Comparisons

Meet-ups

Claude Code Plugin for Hamilton

MCP Server

Ecosystem

check_output*

config.when*

dataloader

datasaver

does

unpack_fields

extract_columns

extract_fields

inject

load_from

parameterize

parameterize_extract_columns

parameterize_frame

parameterize_sources

parameterized_subdag

parameterize_values

pipe family

resolve

save_to

subdag

schema

tag*

with_columns

Decorators

AsyncDriver

Custom Driver

Drivers

Caching logic

Data versioning

Stores

SimplePythonDataFrameGraphAdapter

SimplePythonGraphAdapter

HamiltonGraphAdapter

h_async.AsyncGraphAdapter

h_threadpool.FutureAdapter

CachingGraphAdapter

h_dask.DaskGraphAdapter

h_spark.PySparkUDFGraphAdapter

h_ray.RayGraphAdapter

h_spark.SparkKoalasGraphAdapter

GraphAdapters

lifecycle.ResultBuilder

lifecycle.LegacyResultMixin

lifecycle.api.GraphAdapter

lifecycle.NodeExecutionHook

lifecycle.api.GraphExecutionHook

lifecycle.api.EdgeConnectionHook

lifecycle.api.NodeExecutionMethod

lifecycle.api.StaticValidator

lifecycle.api.GraphConstructionHook

lifecycle.api.TaskSubmissionHook

lifecycle.api.TaskReturnHook

lifecycle.api.TaskExecutionHook

lifecycle.api.TaskGroupingHook

lifecycle.PDBDebugger

lifecycle.PrintLn

plugins.h_tqdm.ProgressBar

plugins.h_rich.RichProgressBar

plugins.h_ddog.DDOGTracer

lifecycle.FunctionInputOutputTypeChecker

plugins.h_slack.SlackNotifier

lifecycle.GracefulErrorAdapter

plugins.h_spark.SparkInputValidator

plugins.h_narhwals.NarwhalsAdapter

plugins.h_mlflow.MLFlowTracker

lifecycle.NoEdgeAndInputTypeChecking

plugins.h_openlineage.OpenLineageAdapter

Lifecycle Adapters

Generic

Numpy

Pandas

Polars

Dask

plugins.h_pyarrow.PyarrowTableResult

Custom ResultBuilder

ResultBuilders

Using Data Adapters

Data Adapters

clear_storage()

copy()

find()

import_module()

inspect()

inspect_module()

install_dependencies_string()

latest_commit()

list()

pull_module()

Dataflows

Framework / Scheduler	Compatibility
Airflow	✅ (see [airflow example](https://github.com/apache/hamilton/tree/main/examples/airflow))
Dagster	✅
Prefect	✅ (see [prefect example](https://github.com/apache/hamilton/tree/main/examples/prefect))
Kubeflow Pipelines	✅
CRON	✅
dbt	✅ (see dbt example)
kubernetes	✅ but you need to setup kubernetes to run an image that can run python code - e.g. see Running a python application on kubernetes
docker	✅ but you need to setup a docker image that can execute python code.
… in general if it runs python 3.7+ …	✅

Using within your ETL System

Using within your ETL System

Compatibility Matrix

ETL Recipe

评论