Lifecycle Adapters

Currently a few of the API extensions are still experimental. Note this doesn’t mean they’re not well-tested or thought out – rather that we’re actively looking for feedback. More docs upcoming, but for now fish around the experimental package, and give the extensions a try!

The other extensions live within plugins. These are fully supported and will be backwards compatible across major versions.

Customization

The subsequent documents contain public-facing APIs for customizing Apache Hamilton’s execution. Note that the public-facing APIs are still a work in progress – we will be improving the documentation. We plan for the APIs, however, to be stable looking forward.

• lifecycle.ResultBuilder

  • ResultBuilder

    • ResultBuilder.build_result()
    • ResultBuilder.do_build_result()
    • ResultBuilder.input_types()
    • ResultBuilder.output_type()

• lifecycle.LegacyResultMixin

  • LegacyResultMixin

    • LegacyResultMixin.build_result()
    • LegacyResultMixin.do_build_result()
    • LegacyResultMixin.input_types()
    • LegacyResultMixin.output_type()

• lifecycle.api.GraphAdapter

  • GraphAdapter

    • GraphAdapter.build_result()
    • GraphAdapter.check_input_type()
    • GraphAdapter.check_node_type_equivalence()
    • GraphAdapter.do_build_result()
    • GraphAdapter.do_check_edge_types_match()
    • GraphAdapter.do_node_execute()
    • GraphAdapter.do_validate_input()
    • GraphAdapter.execute_node()
    • GraphAdapter.input_types()
    • GraphAdapter.output_type()

• lifecycle.NodeExecutionHook

  • NodeExecutionHook

    • NodeExecutionHook.post_node_execute()
    • NodeExecutionHook.pre_node_execute()
    • NodeExecutionHook.run_after_node_execution()
    • NodeExecutionHook.run_before_node_execution()

• lifecycle.api.GraphExecutionHook

  • GraphExecutionHook

    • GraphExecutionHook.post_graph_execute()
    • GraphExecutionHook.pre_graph_execute()
    • GraphExecutionHook.run_after_graph_execution()
    • GraphExecutionHook.run_before_graph_execution()

• lifecycle.api.EdgeConnectionHook

  • EdgeConnectionHook

    • EdgeConnectionHook.check_edge_types_match()
    • EdgeConnectionHook.do_check_edge_types_match()
    • EdgeConnectionHook.do_validate_input()
    • EdgeConnectionHook.validate_input()

• lifecycle.api.NodeExecutionMethod

  • NodeExecutionMethod

    • NodeExecutionMethod.do_node_execute()
    • NodeExecutionMethod.run_to_execute_node()

• lifecycle.api.StaticValidator

  • StaticValidator

    • StaticValidator.run_to_validate_graph()
    • StaticValidator.run_to_validate_node()
    • StaticValidator.validate_graph()
    • StaticValidator.validate_node()

• lifecycle.api.GraphConstructionHook

  • GraphConstructionHook

    • GraphConstructionHook.post_graph_construct()
    • GraphConstructionHook.run_after_graph_construction()

• lifecycle.api.TaskSubmissionHook

  • TaskSubmissionHook

    • TaskSubmissionHook.pre_task_submission()
    • TaskSubmissionHook.run_before_task_submission()

• lifecycle.api.TaskReturnHook

  • TaskReturnHook

    • TaskReturnHook.post_task_return()
    • TaskReturnHook.run_after_task_return()

• lifecycle.api.TaskExecutionHook

  • TaskExecutionHook

    • TaskExecutionHook.post_task_execute()
    • TaskExecutionHook.pre_task_execute()
    • TaskExecutionHook.run_after_task_execution()
    • TaskExecutionHook.run_before_task_execution()

• lifecycle.api.TaskGroupingHook

  • TaskGroupingHook

    • TaskGroupingHook.post_task_expand()
    • TaskGroupingHook.post_task_group()
    • TaskGroupingHook.run_after_task_expansion()
    • TaskGroupingHook.run_after_task_grouping()

Available Adapters

In addition to the base classes for lifecycle adapters, we have a few adapters implemented and available for use. Note that some of these are plugins, meaning they require installing additional (external) libraries.

Recall to add lifecycle adapters, you just need to call the with_adapters method of the driver:

dr = (
    driver
    .Builder()
    .with_modules(...)
    .with_adapters(
        Adapter1(...),
        Adapter2(...),
        *more_adapters)
    ...build()
)

• lifecycle.PDBDebugger

  • PDBDebugger

    • PDBDebugger.__init__()
    • PDBDebugger.do_node_execute()
    • PDBDebugger.post_node_execute()
    • PDBDebugger.pre_node_execute()
    • PDBDebugger.run_after_node_execution()
    • PDBDebugger.run_before_node_execution()
    • PDBDebugger.run_to_execute_node()

• lifecycle.PrintLn

  • PrintLn

    • PrintLn.__init__()
    • PrintLn.post_node_execute()
    • PrintLn.pre_node_execute()
    • PrintLn.run_after_node_execution()
    • PrintLn.run_before_node_execution()

• plugins.h_tqdm.ProgressBar

  • ProgressBar

    • ProgressBar.__init__()
    • ProgressBar.post_graph_execute()
    • ProgressBar.post_node_execute()
    • ProgressBar.pre_graph_execute()
    • ProgressBar.pre_node_execute()
    • ProgressBar.run_after_graph_execution()
    • ProgressBar.run_after_node_execution()
    • ProgressBar.run_before_graph_execution()
    • ProgressBar.run_before_node_execution()

• plugins.h_rich.RichProgressBar

  • RichProgressBar

    • RichProgressBar.__init__()
    • RichProgressBar.post_graph_execute()
    • RichProgressBar.post_node_execute()
    • RichProgressBar.post_task_execute()
    • RichProgressBar.post_task_expand()
    • RichProgressBar.post_task_group()
    • RichProgressBar.pre_graph_execute()
    • RichProgressBar.pre_node_execute()
    • RichProgressBar.pre_task_execute()
    • RichProgressBar.run_after_graph_execution()
    • RichProgressBar.run_after_node_execution()
    • RichProgressBar.run_after_task_execution()
    • RichProgressBar.run_after_task_expansion()
    • RichProgressBar.run_after_task_grouping()
    • RichProgressBar.run_before_graph_execution()
    • RichProgressBar.run_before_node_execution()
    • RichProgressBar.run_before_task_execution()

• plugins.h_ddog.DDOGTracer

  • DDOGTracer

    • DDOGTracer.__init__()
    • DDOGTracer.post_graph_execute()
    • DDOGTracer.post_node_execute()
    • DDOGTracer.post_task_execute()
    • DDOGTracer.pre_graph_execute()
    • DDOGTracer.pre_node_execute()
    • DDOGTracer.pre_task_execute()
    • DDOGTracer.run_after_graph_execution()
    • DDOGTracer.run_after_node_execution()
    • DDOGTracer.run_after_task_execution()
    • DDOGTracer.run_before_graph_execution()
    • DDOGTracer.run_before_node_execution()
    • DDOGTracer.run_before_task_execution()

  • AsyncDDOGTracer

    • AsyncDDOGTracer.__init__()
    • AsyncDDOGTracer.post_graph_construct()
    • AsyncDDOGTracer.post_graph_execute()
    • AsyncDDOGTracer.post_node_execute()
    • AsyncDDOGTracer.pre_graph_execute()
    • AsyncDDOGTracer.pre_node_execute()

• lifecycle.FunctionInputOutputTypeChecker

  • FunctionInputOutputTypeChecker

    • FunctionInputOutputTypeChecker.__init__()
    • FunctionInputOutputTypeChecker.post_node_execute()
    • FunctionInputOutputTypeChecker.pre_node_execute()
    • FunctionInputOutputTypeChecker.run_after_node_execution()
    • FunctionInputOutputTypeChecker.run_before_node_execution()

• plugins.h_slack.SlackNotifier

  • SlackNotifier

    • SlackNotifier.__init__()
    • SlackNotifier.post_node_execute()
    • SlackNotifier.pre_node_execute()
    • SlackNotifier.run_after_node_execution()
    • SlackNotifier.run_before_node_execution()

• lifecycle.GracefulErrorAdapter

  • GracefulErrorAdapter

    • GracefulErrorAdapter.__init__()

  • default.accept_error_sentinels()

• plugins.h_spark.SparkInputValidator

  • SparkInputValidator

    • SparkInputValidator.do_validate_input()

• plugins.h_narhwals.NarwhalsAdapter

  • NarwhalsAdapter

    • NarwhalsAdapter.do_node_execute()
    • NarwhalsAdapter.run_to_execute_node()

• plugins.h_narhwals.NarwhalsDataFrameResultBuilder

  • NarwhalsDataFrameResultBuilder

    • NarwhalsDataFrameResultBuilder.__init__()
    • NarwhalsDataFrameResultBuilder.build_result()
    • NarwhalsDataFrameResultBuilder.do_build_result()
    • NarwhalsDataFrameResultBuilder.input_types()
    • NarwhalsDataFrameResultBuilder.output_type()

• plugins.h_mlflow.MLFlowTracker

  • MLFlowTracker

    • MLFlowTracker.__init__()
    • MLFlowTracker.post_graph_construct()
    • MLFlowTracker.post_graph_execute()
    • MLFlowTracker.post_node_execute()
    • MLFlowTracker.pre_graph_execute()
    • MLFlowTracker.pre_node_execute()
    • MLFlowTracker.run_after_graph_construction()
    • MLFlowTracker.run_after_graph_execution()
    • MLFlowTracker.run_after_node_execution()
    • MLFlowTracker.run_before_graph_execution()
    • MLFlowTracker.run_before_node_execution()

• lifecycle.NoEdgeAndInputTypeChecking

  • NoEdgeAndInputTypeChecking

    • NoEdgeAndInputTypeChecking.check_edge_types_match()
    • NoEdgeAndInputTypeChecking.do_check_edge_types_match()
    • NoEdgeAndInputTypeChecking.do_validate_input()
    • NoEdgeAndInputTypeChecking.validate_input()

• plugins.h_openlineage.OpenLineageAdapter

  • OpenLineageAdapter

    • OpenLineageAdapter.__init__()
    • OpenLineageAdapter.post_graph_execute()
    • OpenLineageAdapter.post_node_execute()
    • OpenLineageAdapter.pre_graph_execute()
    • OpenLineageAdapter.pre_node_execute()

  • SQL datasets

    • Migrating to the datasource identity

  • Reuse the conversion

    • sql_datasets()
    • SqlDatasets