Trino Extractor

Overview

The Trino Extractor is a comprehensive metadata extraction utility designed for Apache Atlas integration with Trino. It provides discovery, extraction, and synchronization of Trino metadata including catalogs, schemas, tables, and columns into Apache Atlas for enhanced data governance and metadata management.

Key Features

Metadata Extraction

• Comprehensive Discovery: Automatically discovers and extracts metadata from Trino catalogs, schemas, tables, and columns
• JDBC-Based Connection: Uses standard Trino JDBC driver for reliable connectivity
• Selective Extraction: Supports extraction for specific catalog, schema, or table names

Atlas Integration

• Entity Management: Creates and updates Atlas entities for Trino metadata objects
• Relationship Mapping: Establishes proper hierarchical relationships between catalogs, schemas, tables, and columns
• Synchronization: Maintains consistency by removing Atlas entities that no longer exist in Trino
• Connector Support: Specialized handling for Trino connectors for which Atlas captures the metadata through individual Hook like Hive, Iceberg

Scheduling & Automation

• Cron-based Scheduling: Supports automated periodic extraction using cron expressions
• One-time Execution: Can be run as a single extraction job
• Error Handling: Robust error handling with detailed logging

Architecture

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Trino Cluster │    │ Trino Extractor │    │  Apache Atlas   │
│                 │    │                 │    │                 │
│ ┌─────────────┐ │    │ ┌─────────────┐ │    │ ┌─────────────┐ │
│ │  Catalogs   │◄┼────┼►│ JDBC Client │ │    │ │  Entities   │ │
│ └─────────────┘ │    │ └─────────────┘ │    │ └─────────────┘ │
│ ┌─────────────┐ │    │ ┌─────────────┐ │    │ ┌─────────────┐ │
│ │   Schemas   │ │    │ │ Extraction  │◄┼────┼►│Relationships│ │
│ └─────────────┘ │    │ │   Service   │ │    │ └─────────────┘ │
│ ┌─────────────┐ │    │ └─────────────┘ │    │ ┌─────────────┐ │
│ │   Tables    │ │    │ ┌─────────────┐ │    │ │  Lineage    │ │
│ └─────────────┘ │    │ │Atlas Client │◄┼────┼►│    Data     │ │
│ ┌─────────────┐ │    │ └─────────────┘ │    │ └─────────────┘ │
│ │   Columns   │ │    │                 │    │                 │
│ └─────────────┘ │    └─────────────────┘    └─────────────────┘
└─────────────────┘                           

Quick Start

1. Configuration Setup

Configure the atlas-trino-extractor.properties file:

# Atlas connection
atlas.rest.address=http://localhost:21000/
# Trino connection
atlas.trino.jdbc.address=jdbc:trino://localhost:8080/
atlas.trino.jdbc.user=your-username
# Catalogs to extract
atlas.trino.catalogs.registered=hive_catalog,iceberg_catalog

2. Basic Execution

# Extract all registered catalogs
./bin/run-trino-extractor.sh
# Extract specific catalog
./bin/run-trino-extractor.sh -c my_catalog
# Schedule periodic extraction (every 6 hours)
./bin/run-trino-extractor.sh -cx "0 0 */6 * * ?"

Configuration Properties

Command Line Usage

Available Options

Connector-Specific Processing

For Example: Hive Connector Integration

# Enable Hive hook integration
atlas.trino.catalog.hook.enabled.hive_catalog=true
atlas.trino.catalog.hook.enabled.hive_catalog.namespace=cm

Benefits:

• Links Trino entities with existing Hive entities
• Maintains consistency between Hive and Trino metadata
• Supports environments with Atlas Hive hooks

FAQ

Troubleshooting Questions

Q: Why are some entities not appearing in Atlas?

A: Check catalog registration, permissions, and network connectivity. Review logs for specific errors.

Q: How do I handle large clusters with thousands of tables?

A: Use selective extraction, increase memory allocation, schedule during off-peak hours, and process catalogs individually.

Documentation

• Apache Atlas Documentation
• Trino Documentation
• Atlas REST API Reference