How to Setup AWS Data Pipeline

Summary:

• AWS Data Pipeline entered maintenance mode in mid-2024. No new features will be added, though existing pipelines continue to function with full support for bug fixes and security patches.
• This guide walks through the complete setup process, covering IAM roles, pipeline definition file syntax, data nodes, activities, preconditions, and scheduling using both the AWS Console and CLI.
• You will learn production-ready best practices for security, monitoring with CloudWatch, cost optimization, and a clear migration path to modern alternatives like AWS Glue, Step Functions, and MWAA.
• A detailed comparison table helps you decide whether to continue with AWS Data Pipeline or transition to serverless ETL solutions based on your workload requirements.

Building a reliable data movement infrastructure on AWS requires understanding which orchestration tools match your operational maturity and long-term roadmap. AWS Data Pipeline has served as a foundational service for scheduling and executing data-driven workflows since 2012. It enables teams to move data between AWS compute and storage services with dependency-aware execution.

The landscape shifted significantly when AWS announced maintenance mode for this service. This prompted engineering teams to evaluate whether to continue leveraging existing pipelines or migrate to newer alternatives. This guide provides a comprehensive walkthrough for setting up AWS Data Pipeline while equipping you with the strategic context needed to make informed architectural decisions in 2025 and beyond.

Understanding AWS Data Pipeline and its current status

AWS Data Pipeline is a web service designed to automate the movement and transformation of data across AWS services and on-premises data sources. The service operates on a definition-based model where you declare data nodes, activities, schedules, and preconditions in a JSON or console-based pipeline definition. This declarative approach allows the service to handle dependency resolution, retry logic, and failure notifications without requiring custom orchestration code.

The Task Runner component can run on EC2 instances or EMR clusters. It polls for scheduled work and executes the defined activities against your data sources.

AWS placed Data Pipeline into maintenance mode in mid-2024, which carries specific implications for your infrastructure decisions. Maintenance mode means AWS will continue to operate the service, fix bugs, and address security vulnerabilities. However, no new features or regional expansions will occur. Existing pipelines remain fully functional, and you can still create new pipelines in supported regions. This status does not indicate imminent deprecation, but it signals that AWS is directing investment toward newer orchestration services like AWS Glue, Step Functions, and Managed Workflows for Apache Airflow.

For teams with existing Data Pipeline workloads, the maintenance mode status provides runway to plan migrations thoughtfully rather than reactively. The service continues to integrate with IAM for access control, CloudWatch for monitoring, and SNS for alerting. Understanding this context helps you decide whether to invest in new pipeline development or allocate engineering effort toward migration.

The following sections detail the setup process for those who need to work with AWS Data Pipeline today while providing migration guidance for future-proofing your architecture.

Key components and architecture

Before configuring your first pipeline, you need to understand the core abstractions that AWS Data Pipeline uses to model data workflows. These components work together to define what data moves, where it moves, when it moves, and what conditions must be satisfied before execution begins. Mastering these concepts enables you to build pipelines that handle complex dependencies and failure scenarios gracefully.

Pipeline definition and data nodes

The pipeline definition file serves as the blueprint for your entire workflow, expressed in JSON format with specific schema requirements. This definition contains all objects that participate in your data movement, including schedules, resources, data nodes, activities, and preconditions. Each object has a unique identifier, a type declaration, and type-specific fields that configure its behavior. The AWS documentation on pipeline objects provides the complete schema reference for all supported object types.

Data nodes represent the logical data sources and destinations in your pipeline. AWS Data Pipeline supports several data node types:

• S3DataNode: References objects in Amazon S3 buckets, supporting both file paths and directory prefixes with time-based expressions.
• DynamoDBDataNode: Connects to DynamoDB tables for reading or writing item collections.
• SqlDataNode: Interfaces with RDS instances or other JDBC-compatible databases for SQL-based data access.
• RedshiftDataNode: Enables data movement to and from Amazon Redshift clusters.

Activities and Task Runner

Activities define the actual work performed on your data nodes, ranging from simple copy operations to complex EMR-based transformations. The CopyActivity moves data between two data nodes without transformation, while ShellCommandActivity executes arbitrary shell scripts on EC2 resources. For big data workloads, EmrActivity and HiveActivity leverage EMR clusters to run distributed processing jobs.

Each activity references input and output data nodes, creating the dependency graph that the scheduler uses to determine execution order.

Preconditions and scheduling

Preconditions act as gates that must evaluate to true before an activity executes. The DynamoDBDataExists precondition verifies that a DynamoDB table contains data, while S3KeyExists checks for the presence of specific S3 objects. You can also create custom preconditions using ShellCommandPrecondition to run arbitrary validation logic. These mechanisms prevent activities from running against incomplete or missing data, reducing failed executions and downstream data quality issues.

Scheduling in AWS Data Pipeline supports both time-based and on-demand execution patterns. Time-based schedules use cron-like expressions to define recurring execution windows, with support for backfill operations that process historical time slices. The schedule object defines the period, start time, and end time for pipeline execution.

On-demand activation allows you to trigger pipelines programmatically through the API or console. This is useful for event-driven architectures where external systems determine when data processing should occur. With these foundational components understood, you can proceed to the hands-on setup process.

Step-by-step setup guide

Setting up AWS Data Pipeline involves configuring IAM permissions, defining your pipeline structure, and activating the workflow for execution. This section walks through each phase with practical examples that you can adapt to your specific use case. The process applies whether you use the AWS Console, CLI, or infrastructure-as-code tools like Terraform.

Configuring IAM roles

AWS Data Pipeline requires two IAM roles to operate securely. The pipeline role grants the Data Pipeline service permission to access AWS resources on your behalf. The resource role provides permissions to the EC2 instances or EMR clusters that execute your activities.

AWS provides default roles named DataPipelineDefaultRole and DataPipelineDefaultResourceRole that you can use for initial testing. Production deployments should use custom roles with least-privilege permissions.

Create a custom pipeline role with the following trust policy that allows the Data Pipeline service to assume it:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "datapipeline.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

Attach a permissions policy that grants access only to the specific S3 buckets, DynamoDB tables, or other resources your pipeline needs. For the resource role, the trust policy must allow EC2 to assume the role. The permissions should include access to Data Pipeline APIs for task heartbeating and status reporting. This separation of concerns ensures that compromised compute resources cannot escalate privileges beyond their intended scope.

Creating the pipeline definition

The pipeline definition file declares all objects that participate in your workflow. The following example demonstrates a pipeline that copies data from one S3 location to another on a daily schedule. Each object includes an id field for reference, a type field declaring the object class, and type-specific configuration fields.

{
  "objects": [
    {
      "id": "Default",
      "name": "Default",
      "scheduleType": "cron",
      "schedule": { "ref": "DailySchedule" },
      "failureAndRerunMode": "CASCADE",
      "pipelineLogUri": "s3://my-bucket/logs/",
      "role": "DataPipelineCustomRole",
      "resourceRole": "DataPipelineCustomResourceRole"
    },
    {
      "id": "DailySchedule",
      "type": "Schedule",
      "period": "1 day",
      "startAt": "FIRST_ACTIVATION_DATE_TIME"
    },
    {
      "id": "SourceData",
      "type": "S3DataNode",
      "directoryPath": "s3://source-bucket/input/#{format(@scheduledStartTime,'YYYY-MM-dd')}/"
    },
    {
      "id": "DestinationData",
      "type": "S3DataNode",
      "directoryPath": "s3://destination-bucket/output/#{format(@scheduledStartTime,'YYYY-MM-dd')}/"
    },
    {
      "id": "CopyActivity",
      "type": "CopyActivity",
      "input": { "ref": "SourceData" },
      "output": { "ref": "DestinationData" },
      "runsOn": { "ref": "Ec2Resource" }
    },
    {
      "id": "Ec2Resource",
      "type": "Ec2Resource",
      "instanceType": "t3.medium",
      "terminateAfter": "2 hours"
    }
  ]
}

Deploying with AWS CLI

After preparing your pipeline definition, use the AWS CLI to create and activate the pipeline. The following commands demonstrate the complete deployment workflow:

1. Create the pipeline: aws datapipeline create-pipeline --name "DailyS3Copy" --unique-id "daily-s3-copy-001"
2. Upload the definition: aws datapipeline put-pipeline-definition --pipeline-id df-1234567890 --pipeline-definition file://pipeline-definition.json
3. Validate the configuration: aws datapipeline validate-pipeline-definition --pipeline-id df-1234567890 --pipeline-definition file://pipeline-definition.json
4. Activate the pipeline: aws datapipeline activate-pipeline --pipeline-id df-1234567890

The validation step catches syntax errors and missing references before activation, preventing failed deployments. Once activated, the pipeline begins executing according to its schedule. Task Runner instances poll for work and report status back to the service. The next section covers operational best practices for running pipelines reliably in production environments.

Best practices for production deployments

Running AWS Data Pipeline in production requires attention to security hardening, observability, and cost management. These practices differentiate proof-of-concept implementations from enterprise-grade deployments that handle failures gracefully and provide visibility into operational health.

Security and access control

Implement defense-in-depth by combining IAM policies with network controls and encryption. Deploy Task Runner instances in private subnets with no direct internet access, using VPC endpoints for S3 and DynamoDB communication. Enable server-side encryption on all S3 buckets referenced by your pipelines, and use AWS KMS customer-managed keys for sensitive workloads. The pipelineLogUri should point to an encrypted bucket with restricted access to prevent exposure of execution details.

Monitoring and debugging

AWS Data Pipeline integrates with Amazon CloudWatch for metrics and logging. Configure CloudWatch alarms on pipeline execution failures, activity timeouts, and resource provisioning errors. The service emits metrics, including ActivitiesRunning, ActivitiesFailed, and ActivitiesSucceeded, that provide aggregate visibility into pipeline health. For detailed debugging, enable pipeline logging to S3 and use CloudWatch Logs Insights to query execution traces.

Set up SNS notifications for critical events using the onFail and onSuccess fields in your pipeline definition. These notifications should route to operational channels like PagerDuty or Slack for immediate visibility. For complex pipelines, implement custom health checks using ShellCommandActivity that validate data quality after each stage and emit custom CloudWatch metrics for business-level monitoring.

Cost optimization

AWS Data Pipeline pricing includes charges for pipeline execution plus the underlying compute and storage resources. Optimize costs by right-sizing EC2 instance types based on actual workload requirements. Use spot instances for fault-tolerant activities and schedule pipelines during off-peak hours when possible. The terminateAfter field on Ec2Resource objects ensures instances shut down after completing work, preventing runaway costs from orphaned resources.

Review pipeline execution history monthly to identify optimization opportunities. Activities that consistently complete quickly may be over-provisioned, while those that frequently timeout need larger instances or parallelization. This operational maturity positions you well for evaluating migration options, which the next section addresses comprehensively.

Migration guide and service comparison

Given the maintenance mode status of AWS Data Pipeline, understanding migration paths to modern alternatives is essential for long-term architectural planning. Each alternative service offers different trade-offs in terms of complexity, cost, and capability that align with specific workload patterns.

Comparing orchestration services

The following table compares AWS Data Pipeline with its primary alternatives across dimensions that matter for production deployments:

Migration strategies

For simple data movement workloads, AWS Glue provides the most direct migration path. Glue crawlers can automatically discover schema from your data sources, and Glue jobs handle both ETL transformations and simple copy operations. The serverless execution model eliminates Task Runner management overhead. Glue job bookmarks provide built-in incremental processing that replaces manual time-slice handling in Data Pipeline.

For complex orchestration with conditional logic, parallel execution, and human approval steps, AWS Step Functions offers superior workflow modeling capabilities. Step Functions integrates natively with over 200 AWS services, enabling you to compose workflows that combine data processing with application logic. The visual workflow builder accelerates development, and the pay-per-transition pricing model can reduce costs for infrequently executed pipelines.

For teams with existing Apache Airflow expertise or complex dependency graphs, Amazon MWAA provides a managed Airflow environment that supports the full Airflow operator ecosystem. This option offers the most flexibility but requires Airflow-specific knowledge and incurs higher baseline costs due to the always-on environment model. Evaluate your team’s skills and workload complexity when selecting the appropriate migration target.

Conclusion

AWS Data Pipeline remains a viable option for data orchestration workloads, particularly for teams with existing investments in pipeline definitions and operational runbooks. The maintenance mode status provides stability for current deployments while signaling the strategic direction toward serverless alternatives. Setting up new pipelines requires careful attention to IAM configuration, pipeline definition syntax, and operational practices that ensure reliable execution.

The key takeaways from this guide center on three areas. First, understand that maintenance mode means continued support without new features, giving you time to plan migrations thoughtfully. Second, invest in proper IAM role separation and monitoring configuration from the start, as these practices transfer directly to any future orchestration platform. Third, evaluate migration targets based on your specific workload patterns, team expertise, and cost constraints rather than defaulting to the newest service.

Looking ahead, the data orchestration landscape continues evolving toward serverless, event-driven architectures that reduce operational overhead. Whether you continue operating AWS Data Pipeline or migrate to Glue, Step Functions, or MWAA, the fundamental concepts of data nodes, activities, dependencies, and scheduling remain relevant. Master these abstractions, and you will navigate any orchestration platform with confidence.