AWS Glue Tutorial: Run a simple ETL job

Summary:

• Learn how to build and run a complete AWS Glue ETL job that transforms CSV data to Parquet format using both the visual editor and PySpark scripts.
• Understand the architectural flow from S3 source through the Data Catalog to your target destination, including IAM role configuration and crawler setup.
• Compare Glue engine options including Spark, Ray, and Visual ETL to select the right approach for your workload requirements and budget.
• Master DynamicFrame operations, ApplyMapping transforms, and job monitoring techniques used by production data engineering teams.

Every data engineering team eventually faces the same inflection point. Manual scripts that once handled modest data volumes begin buckling under production scale, and the overhead of managing Spark clusters becomes a distraction from actual business logic. AWS Glue eliminates that operational burden by providing a fully managed ETL service that auto-scales compute resources while integrating natively with the broader AWS ecosystem.

This AWS Glue tutorial walks you through building a simple ETL job from scratch. It covers both the visual drag-and-drop interface and the programmatic PySpark approach so you can choose the method that fits your team’s workflow. By the end, you will have transformed raw CSV files into optimized Parquet format and gained the foundational knowledge to tackle more complex data pipelines.

The following diagram illustrates the end-to-end architecture you will build throughout this tutorial. It shows how data flows from source to destination through Glue’s core components.

What is AWS Glue and why it matters for ETL workloads

AWS Glue is a serverless data integration service that handles extract, transform, and load operations without requiring you to provision or manage infrastructure. At its core, Glue combines three capabilities. First, a centralized Data Catalog stores metadata about your datasets. Second, managed Spark or Ray runtimes execute transformation logic. Third, crawlers automatically infer schemas from raw data sources. This combination means you can focus on defining business transformations rather than wrestling with cluster sizing, dependency management, or job orchestration mechanics.

For junior and mid-level engineers, Glue provides an accessible entry point into distributed data processing through its visual editor and pre-built transforms. Senior and staff engineers appreciate the service for different reasons. These include predictable cost models based on Data Processing Units (DPUs), native integration with Lake Formation for governance, and the ability to version-control transformation logic alongside application code. The serverless model also eliminates the cold-start delays associated with EMR cluster provisioning, making Glue particularly effective for event-driven or scheduled batch workloads.

Understanding when to use Glue versus alternatives requires evaluating your latency requirements, data volume, and team expertise. Consider the following decision factors before committing to an architecture.

Core components you will use

Before diving into implementation, familiarize yourself with the building blocks that make Glue ETL jobs function. The Data Catalog serves as a persistent metadata repository compatible with the Apache Hive metastore. This means tools like Amazon Athena and Redshift Spectrum can query cataloged tables directly. Crawlers automate schema detection by scanning data sources and populating catalog tables with column names, data types, and partition information. ETL jobs contain your transformation logic and can be authored through the AWS Glue Visual ETL canvas, interactive notebooks, or traditional script files uploaded to S3.

DynamicFrames represent Glue’s extension of Spark DataFrames, adding schema flexibility that handles inconsistent or evolving source data gracefully. Unlike rigid DataFrame schemas that fail on unexpected columns, DynamicFrames track schema variations per-record and provide resolution strategies. This distinction becomes critical when processing data lakes where upstream systems may add fields without coordination. With these concepts established, the next section covers the prerequisites needed before creating your first job.

Prerequisites and environment setup

Completing this tutorial requires an active AWS account with permissions to create IAM roles, S3 buckets, and Glue resources. You will also need the AWS CLI configured locally if you prefer command-line interactions, though all steps can be accomplished through the AWS Management Console. Ensure you have a sample CSV dataset ready for transformation. The AWS Glue documentation provides example datasets, or you can use any structured CSV file containing at least a few hundred records for meaningful testing.

The IAM role configuration deserves careful attention because insufficient permissions cause the majority of first-time Glue job failures. Your Glue service role needs the following trust relationship and policy attachments:

• Trust relationship: Allow the glue.amazonaws.com service principal to assume the role.
• AWSGlueServiceRole: Managed policy providing baseline Glue permissions.
• S3 access: Read permissions on your source bucket and write permissions on your target bucket.

Create two S3 buckets. One is for source data and one is for transformed output. Upload your CSV file to the source bucket, noting the exact path because you will reference it when configuring the crawler. With infrastructure in place, you can now create the Data Catalog entries that describe your source schema.

Creating a crawler to populate the Data Catalog

Navigate to the AWS Glue console and select Crawlers from the left navigation panel. Click Create crawler and provide a descriptive name such as csv-source-crawler. For the data source, choose S3 and specify the path to your uploaded CSV file. Select the IAM role you created earlier, then configure the crawler to run on demand rather than on a schedule for this tutorial. Finally, create a new database in the Data Catalog to store the discovered table metadata.

Run the crawler and wait for completion, which typically takes under two minutes for small datasets. Once finished, navigate to Tables in the Data Catalog section to verify that a new table exists with columns matching your CSV structure. The crawler automatically infers data types, though you may need to adjust classifications for ambiguous fields like dates stored as strings. This cataloged table becomes the input source for your ETL job, enabling Glue to understand the schema without hardcoding column definitions in your transformation code.

The visual representation below shows the crawler configuration workflow and resulting catalog table structure.

Building your first ETL job with Glue Studio visual ETL

Glue Studio provides a drag-and-drop interface that generates PySpark code automatically, making it ideal for rapid prototyping or teams with limited Spark experience. From the Glue console, select Glue Studio and choose Visual with a source and target to start a new job. The canvas opens with a source node already placed. Configure it to read from the Data Catalog table your crawler created. The visual editor displays a live schema preview, allowing you to verify column detection before adding transformations.

Add an ApplyMapping transform node by clicking the plus icon and selecting it from the transform menu. This node lets you rename columns, change data types, and drop unnecessary fields without writing code. For a simple ETL job converting CSV to Parquet, map your source columns to cleaner target names and cast string dates to timestamp types where appropriate. The visual editor updates the downstream schema preview in real-time as you configure mappings.

Finally, add a target node configured to write Parquet format to your output S3 bucket. Enable schema updates in the Data Catalog so the job automatically registers the output table for downstream querying. Configure job details including the IAM role, Glue version, and worker type. For this tutorial, select Glue version 5.0 with G.1X workers and two worker nodes as a cost-effective starting point. Save and run the job, then monitor progress in the Runs tab.

Understanding the generated PySpark code

Glue Studio generates production-ready PySpark code that you can export, modify, and version control. Click the Script tab to view the auto-generated code, which follows Glue’s standard job structure including initialization, transformation, and commit phases. Understanding this code prepares you for scenarios requiring custom logic beyond visual editor capabilities. The following annotated script demonstrates the key patterns.

import sys
from awsglue.transforms import *
from awsglue.utils import getResolvedOptions
from pyspark.context import SparkContext
from awsglue.context import GlueContext
from awsglue.job import Job
from awsglue.dynamicframe import DynamicFrame

# Initialize Glue context and job
args = getResolvedOptions(sys.argv, ['JOB_NAME'])
sc = SparkContext()
glueContext = GlueContext(sc)
spark = glueContext.spark_session
job = Job(glueContext)
job.init(args['JOB_NAME'], args)

# Read from Data Catalog as DynamicFrame
source_dyf = glueContext.create_dynamic_frame.from_catalog(
    database="your_database",
    table_name="your_source_table",
    transformation_ctx="source_dyf"
)

# Apply column mappings and type conversions
mapped_dyf = ApplyMapping.apply(
    frame=source_dyf,
    mappings=[
        ("old_column_name", "string", "new_column_name", "string"),
        ("date_string", "string", "event_date", "timestamp"),
        ("amount", "string", "amount", "double")
    ],
    transformation_ctx="mapped_dyf"
)

# Convert to DataFrame for advanced Spark operations if needed
df = mapped_dyf.toDF()
df_filtered = df.filter(df.amount > 0)
filtered_dyf = DynamicFrame.fromDF(df_filtered, glueContext, "filtered_dyf")

# Write to S3 in Parquet format
glueContext.write_dynamic_frame.from_options(
    frame=filtered_dyf,
    connection_type="s3",
    format="parquet",
    connection_options={"path": "s3://your-output-bucket/transformed/"},
    transformation_ctx="output"
)

job.commit()

Notice the transformation_ctx parameters throughout the script. These context strings enable Glue’s job bookmarking feature, which tracks processed data and prevents reprocessing on subsequent runs. The pattern of converting between DynamicFrame and DataFrame demonstrates how to leverage native Spark operations when Glue’s built-in transforms prove insufficient. With the visual approach covered, the next section explores writing custom scripts for greater control.

Writing custom ETL scripts with PySpark and Scala

Production data pipelines often require logic that exceeds visual editor capabilities. This includes complex joins across multiple sources, custom aggregations, or integration with external libraries. Writing scripts directly provides full access to Spark’s API while maintaining Glue’s managed infrastructure benefits. Create a new job in Glue Studio, but select Spark script editor instead of the visual option. You can author directly in the browser or upload scripts from your local development environment.

The script structure follows a consistent pattern regardless of complexity. Initialize the GlueContext and Job objects, read source data into DynamicFrames, apply transformations, and commit the job upon completion. The commit call signals successful execution and updates job bookmarks. Omitting this call causes Glue to treat the run as failed even if data was written successfully.

For teams with existing Scala codebases, Glue supports Scala scripts with equivalent functionality. The following snippet demonstrates the same CSV-to-Parquet transformation in Scala syntax.

import com.amazonaws.services.glue.GlueContext
import com.amazonaws.services.glue.util.GlueArgParser
import com.amazonaws.services.glue.util.Job
import org.apache.spark.SparkContext
import com.amazonaws.services.glue.DynamicFrame

object GlueETLJob {
  def main(args: Array[String]): Unit = {
    val sc: SparkContext = new SparkContext()
    val glueContext: GlueContext = new GlueContext(sc)
    val spark = glueContext.getSparkSession
    val jobArgs = GlueArgParser.getResolvedOptions(args, Seq("JOB_NAME").toArray)
    Job.init(jobArgs("JOB_NAME"), glueContext, jobArgs.asJava)
    
    val sourceDyf = glueContext.getCatalogSource(
      database = "your_database",
      tableName = "your_source_table"
    ).getDynamicFrame()
    
    val mappedDyf = sourceDyf.applyMapping(
      mappings = Seq(
        ("old_column", "string", "new_column", "string"),
        ("amount", "string", "amount", "double")
      )
    )
    
    glueContext.getSinkWithFormat(
      connectionType = "s3",
      options = JsonOptions(Map("path" -> "s3://output-bucket/transformed/")),
      format = "parquet"
    ).writeDynamicFrame(mappedDyf)
    
    Job.commit()
  }
}

Choosing between PySpark and Scala depends on team expertise and existing codebases. PySpark offers faster iteration cycles and broader library ecosystem access. Scala provides stronger type safety and marginally better runtime performance for compute-intensive transformations. Both languages receive equal support in Glue’s managed environment.

Comparing Glue engines and versions

AWS Glue has evolved significantly since its 2017 launch, with each major version introducing performance improvements and new capabilities. Selecting the appropriate version and engine type impacts both job performance and cost efficiency. The table below summarizes key differences between currently supported options.

Glue for Ray represents a significant architectural departure, replacing Spark’s JVM-based execution with Python-native distributed computing. This engine excels for workloads involving pandas transformations, scikit-learn models, or other Python libraries that serialize poorly to Spark’s execution model. However, Ray lacks equivalents for some Spark-specific features like DynamicFrames and certain Data Catalog integrations. This makes it better suited for compute-heavy transformations than traditional ETL patterns.

Architectural Note: AWS officially announced the end of support for AWS Glue for Ray (restricting new signups in April 2026). For Python-native ML pipelines or pandas-heavy workloads that do not serialize well to Spark, modern architectures now utilize Amazon EKS or Amazon SageMaker Unified Studio rather than Glue.

The visual below compares execution models across engine types to help inform your selection.

Monitoring, debugging, and scheduling jobs

Production ETL pipelines require robust observability to detect failures, diagnose performance bottlenecks, and maintain data freshness SLAs. Glue integrates with Amazon CloudWatch for metrics and logging, providing visibility into job execution without additional instrumentation. Key metrics to monitor include job duration, DPU utilization, bytes read and written, and error counts. Configure CloudWatch alarms on these metrics to receive notifications when jobs exceed expected runtime or fail repeatedly.

Debugging failed jobs starts with the Glue console’s run details page, which displays error messages and links to CloudWatch Logs. Common failure patterns include:

1. OutOfMemoryError: Upgrade the worker type from G.1X to G.2X for more memory per executor, or use Spark repartitioning to break up massive, skewed datasets.
2. Access denied: Verify IAM role permissions and S3 bucket policies.
3. Schema mismatch: Re-run crawlers after source schema changes or use DynamicFrame’s resolveChoice method.
4. Job timeout: Extend the timeout setting or optimize transformation logic to reduce processing time.

Pro tip: Modern AWS Glue versions enable Continuous Logging by default. You no longer have to wait for a job to complete or fail to see the output. You can monitor the real-time CloudWatch log streams to catch errors or track progress while the job is still actively running.

Scheduling jobs uses Glue Triggers, which support time-based schedules (cron expressions), event-based activation (job completion chains), or on-demand execution. For complex pipelines, consider using Glue Workflows to orchestrate multiple crawlers and jobs with dependency management. Alternatively, AWS Step Functions provides more sophisticated orchestration capabilities including conditional branching and human approval steps.

Conclusion

This AWS Glue tutorial demonstrated the complete lifecycle of building a simple ETL job. It covered configuring IAM roles and crawlers through authoring transformations in both the visual editor and PySpark scripts. The key architectural insight is that Glue’s value proposition extends beyond serverless compute. The Data Catalog provides a unified metadata layer that enables downstream analytics tools, while DynamicFrames handle schema evolution gracefully in ways that rigid DataFrame schemas cannot.

Selecting between Glue versions and engines requires matching your workload characteristics to each option’s strengths. Glue 5.0 serves as the sensible default for most batch ETL scenarios. Looking ahead, expect continued convergence between Glue’s visual and programmatic interfaces as AWS expands Glue Studio’s transform library and notebook integration capabilities.

Teams adopting Glue today should establish patterns for version-controlling generated scripts and monitoring job performance baselines. These foundations scale as pipeline complexity grows. Start with the visual editor for rapid prototyping, graduate to custom scripts when business logic demands it, and leverage the Data Catalog as your organization’s single source of schema truth.