User Guide

This portion of the documentation goes over the set of common examples for Apache Hamilton usage, so you can apply it to your day-to-day work. Each one corresponds to an example in the examples directory. If there’s an example you want but don’t see, reach out or open an issue on github – we’re always looking to add more.

• Jupyter notebooks

  • 1 - Dynamically create modules within your notebook

    • Use Hamilton Jupyter Magic

      • Importing specific functions into cell modules

    • Using ad_hoc_utils to create a temporary module (e.g. use in google colab)

    • Caveat with this approach:

  • 2 - Importing modules into your notebook

    • Step 1 — Install Jupyter & Apache Hamilton
    • Step 2— Set up the files
    • Step 3— The basic process of iteration

  • Pro-tip: You can use ipython magic to autoreload code

  • Pro-tip: You can import functions directly

• Loading data

• Caching

  • Basics

    • Understanding the cache_key

  • Adding a node

  • Changing inputs

  • Changing code

  • Changing external data

    • Idempotency
    • .with_cache() to specify caching behavior
    • @cache to specify caching behavior
    • When to use @cache vs. .with_cache()?

  • Force recompute all

  • Setting default behavior

  • Materializers

    • Usage patterns

  • Changing the cache format

  • Introspecting the cache

    • Interactively explore runs

  • Managing storage

    • Setting the cache path
    • Instantiating the result_store and metadata_store
    • Deleting data and recovering storage

  • Usage patterns

  • 🚧 INTERNALS

    • Manually retrieve results
    • Decoding the cache_key
    • Manually retrieve metadata

• Feature engineering

  • Offline Feature Engineering

    • Apache Hamilton Example

  • Streaming Feature Engineering

    • Apache Hamilton Example

  • Online Feature Engineering

    • Apache Hamilton Example

  • Write once, run anywhere blog post:

  • Best Egg Platform Blog Post:

  • FAQ

    • Q. Can I use Apache Hamilton for feature engineering with Feast?

• Reusing functions and nodes

  • 1. Reuse a function module across multiple Drivers
  • 2. Override a module to swap same-named functions
  • 3. @subdag – repeat the same transform inside one Driver
  • 4. @parameterized_subdag – many subdags at once (advanced)
  • Choosing between the four patterns
  • Where to go from here

• Model training

• LLM workflows

• Data quality

  • Async validators
  • Disabling validators at runtime

• Testing Apache Hamilton code

  • Prerequisites

  • 1. Unit-testing plain functions

    • Notes

  • 2. Unit-testing decorated functions

  • 3. Integration-testing the DAG

  • 4. In-memory modules for self-contained tests

  • Where to go from here

• Lineage + Apache Hamilton

  • Common Problems (and therefore questions)

  • What is “Lineage”?

  • Apache Hamilton’s Lineage Capabilities

    • Lineage as Code
    • Reproducibility
    • Auditing and Compliance
    • Troubleshooting and Debugging
    • Collaboration

  • Recipe for using Apache Hamilton’s Lineage Capabilities

  • A script you could write to ask questions of your DAGs

• Scaling computation

• Microservice

• Extension autoloading

  • Autoloading behavior

  • Disable autoloading

    • 1. Programmatically
    • 2. Environment variables
    • 3. Configuration file

  • Manually loading extensions

    • 1. Importing the extension
    • 2. Registering the extension

• Wrapping the Driver

• Command line interface

  • Installation
  • hamilton (global)
  • hamilton build
  • hamilton diff
  • hamilton version
  • hamilton view

• pre-commit hooks

  • Use pre-commit hooks for safer Apache Hamilton code changes

  • What are pre-commit hooks?

  • Add pre-commit hooks to your project

    • Steps to get started

  • Custom Apache Hamilton pre-commit hooks

    • Checking dataflow definition
    • Checking dataflow paths

  • Add Apache Hamilton pre-commit to your project