AWS data pipeline runtimes: Glue 6.0 vs EMR Serverless

AWS Data Pipeline stopped taking new customers in July 2024, so every new AWS data pipeline today is assembled from Glue and EMR Serverless, with Step Functions conducting. Those services are billed on different clocks, and on a 100,000 DPU-hour month the runtime you pick moves the bill by 23,700 USD before you change a line of SQL.

Retiring a service is easy. Replacing it is the work, and the replacement usually gets decided in a design review by whoever is most confident that week. I have sat through enough of those to recognise the assumption underneath: serverless means the bill takes care of itself. It does not. The invoice has structure.

[!NOTE] TL;DR - Glue 6.0 (GA 21 August 2026) cut the DPU-hour rate by 30%, to 0.308 USD. A standard Glue hour now sits within a cent of an equal EMR Serverless worker at 0.303 USD. - Glue Flex takes another 34% off, around 0.203 USD, but AWS reports a 5% to 10% interruption rate at peak hours and warns against using it for jobs with downstream dependencies. - The larger lever is physical: Athena bills 5.00 USD per TB scanned whatever wrote the files, so partition design and compaction decide the bill more than the runtime does.

The runtime question

My main question was narrow, and it took two rewrites of a pricing spreadsheet to ask it properly: for a given Spark job, which runtime should carry it, using numbers I could defend in front of a CFO without mumbling?

Runtimes are not ranked. They are matched to the shape of a job, and that shape is measurable: DPU-equivalents burned per run, plus cadence. Then there is one piece of policy that has nothing to do with compute, namely whether anything downstream waits on the output.

Underneath sits the part that took me longest to accept. There are two meters. One charges compute while the job exists, in DPU-hours on Glue or in vCPU-hours and GB-hours on EMR Serverless. The other charges bytes when somebody reads what the job wrote, at 5.00 USD per TB scanned in Athena. Tuning the first meter while ignoring the second is how a 30% discount dissolves inside a scan amplification nobody measured. I explored the same asymmetry from the inference side in the scale-to-zero trap, and it cost me more than one evening.

The blind DPU default

The obvious move is to put the whole estate on Glue 6.0. It is serverless, it ships with the Data Catalog, and since 21 August 2026 it costs 30% less per DPU-hour. AWS ran TPC-DS at 3 TB scale with 30 G.2X workers and published 5.61 USD against 8.87 USD on Glue 5.1, which it summarises as up to 36% better price performance.

The numbers support the default. What they do not show is where it stops being true.

A DPU is not a neutral unit. It is 4 vCPU and 16 GB sold as a block, and Glue does offer memory-optimised worker types when that ratio is wrong for you. What you cannot do is stop paying for the half of a worker your job never touches. EMR Serverless prices vCPU, memory and storage on three separate axes, so 4 vCPU with 30 GB is a legal job spec there, and a job can grow its own worker count mid-run.

The other half of the trap has nothing to do with the rate. A Glue run does not start on your code. It starts when capacity exists for your DPUs, and engineer-published logs on Glue 4.0 put that gap at 30 to 90 seconds. A DPU is a taxi meter that starts when the driver begins looking for your street. The per-mile rate is the number you compare. The circling is billed at the same tariff.

Glue Flex attacks the rate instead. It runs on spare capacity and costs 34% less, and switching is a parameter on the job, not a rewrite. The AWS FAQ is unusually blunt about the trade: interruption rates of 5% to 10% during peak hours, and an explicit warning not to use Flex for jobs that other processes depend on. A retried run is billed twice.

The small-file tax

This is where the compute argument loses to physics. Athena charges 5.00 USD per TB scanned, rounded to the megabyte, and the file layout decides what a query actually touches. Partition pruning removes whole prefixes. Nothing else removes anything.

Take one day's partition holding 4,000 files of 12 MB, which is roughly what a five-minute micro-batch leaves behind after a year. That is 48 GB on S3, and a query filtering on a recent timestamp reads all of it, because unsorted small files each carry a wide range of values and every row-group statistic overlaps the predicate. Compact the same rows into 96 files of 500 MB, sorted on the filter column, and most of those files become skippable: 12 GB read instead of 48, identical data, four times cheaper for the same answer.

A fragmented table behaves like an archive where every document was photocopied onto its own sheet and filed under a different number. Owning fewer words is not the saving. The saving is that the index stops lying to you about where the words are.

Athena exposes the fix directly: OPTIMIZE ... REWRITE DATA USING BIN_PACK to merge files, VACUUM to expire snapshots and orphans, then Iceberg statistics and Parquet column indexes to prune at page level rather than row-group level. None of that depends on which engine wrote the files, which is the whole point.

The shape of the pipeline

flowchart LR
  S3[S3 raw landing<br/>ingest_date=YYYY-MM-DD] --> EB[EventBridge rule]
  EB --> SF[Step Functions Standard]
  SF --> DM[Distributed Map<br/>ItemBatcher 100]
  DM --> RT{Runtime routing}
  RT -->|dependent output| GS[Glue 6.0 STANDARD]
  RT -->|no dependents| GF[Glue 6.0 FLEX]
  RT -->|tuned Spark| EM[EMR Serverless Graviton]
  GS --> ICE[Iceberg curated tables]
  GF --> ICE
  EM --> ICE
  ICE --> OPT[OPTIMIZE and VACUUM]
  ICE --> ATH[Athena]
  ICE --> RS[Redshift]
  DDB[DynamoDB orders] -->|zero-ETL 15 min| RS

The skeleton matters more than the services slotted into it. Landing zone first, immutable, partitioned on ingestion date. Orchestration in Step Functions Standard, because a fan-out over a prefix is a concurrency problem and a Distributed Map charges one state transition per child, at 0.000025 USD. Curated zone in Iceberg, so Athena, Redshift and EMR read the same files without a copy. I described the same event-driven skeleton for ML workloads in this walkthrough.

Two tables do not belong in that flow at all. Order state lives in DynamoDB, and a zero-ETL integration replicates it into Redshift on a 15-minute cycle with no code of mine in the path. I reach for it when the source is one of the supported engines (DynamoDB, Aurora, RDS, Oracle Database@AWS) and freshness is measured in minutes. It moves rows. It does not model them.

[!WARNING] A zero-ETL integration's target is fixed at creation. AWS states plainly that you cannot modify the target afterwards, and deleting an integration is irreversible. Choose the Redshift namespace or the S3 location as if you were signing a lease.

Since November 2025 the integrations can be declared in CloudFormation and CDK. That is the difference between a prototype and something you can rebuild in a second account at 2 a.m.

Runtime selection as code

Deciding per job in a spreadsheet does not survive contact with 180 jobs. I encode the rule instead, and I make the model refuse the configuration AWS explicitly warns against.

from enum import StrEnum
from pydantic import BaseModel, Field, field_validator, model_validator
class Runtime(StrEnum):
    GLUE_STANDARD = "glue-standard"
    GLUE_FLEX = "glue-flex"
    EMR_SERVERLESS_GRAVITON = "emr-serverless-graviton"
class JobProfile(BaseModel):
    model_config = {"strict": True}
    name: str = Field(..., description="Slug reused as the Step Functions state name.")
    dpu_hours_per_run: float = Field(..., gt=0.0, description="Measured on Glue 5.1. One DPU is 4 vCPU and 16 GB.")
    runs_per_day: int = Field(..., ge=1, le=1440)
    has_downstream_dependency: bool = Field(..., description="True when another job waits on this output.")
    @field_validator("name")
    @classmethod
    def slug_only(cls, value: str) -> str:
        if value != value.lower() or not value.replace("-", "").isalnum():
            raise ValueError("name must be a lowercase slug")
        return value
    @model_validator(mode="after")
    def reject_flex_for_dependents(self) -> "JobProfile":
        if self.has_downstream_dependency and self.runs_per_day > 96:
            raise ValueError("sub-hourly cadence with a dependent: keep it on STANDARD")
        return self
class PricedRuntime(BaseModel):
    runtime: Runtime
    unit_usd: float = Field(..., gt=0.0, description="USD per DPU-equivalent hour, list price, Nov 2026.")
    monthly_usd: float = Field(..., ge=0.0)
    @staticmethod
    def rates() -> dict[Runtime, float]:
        return {Runtime.GLUE_STANDARD: 0.308, Runtime.GLUE_FLEX: 0.203, Runtime.EMR_SERVERLESS_GRAVITON: 0.242424}
    @classmethod
    def price(cls, profile: JobProfile) -> list["PricedRuntime"]:
        total = profile.dpu_hours_per_run * profile.runs_per_day * 30
        return [cls(runtime=r, unit_usd=c, monthly_usd=round(total * c, 2)) for r, c in cls.rates().items()]

One number in that table I had to derive rather than copy. EMR Serverless Graviton bills 0.042094 USD per vCPU-hour and 0.004628 USD per GB-hour, so a DPU-equivalent hour costs 4 x 0.042094 + 16 x 0.004628 = 0.242424 USD. The x86 rate works out at 0.30295 USD, which is why Glue 6.0 and EMR Serverless now land in the same drawer.

The routing itself is one Choice state.

StartAt: Route
States:
  Route:
    Type: Choice
    Choices:
      - Variable: "$.job.flex_eligible"
        BooleanEquals: true
        Next: GlueFlex
    Default: GlueStandard
  GlueFlex:
    Type: Task
    Resource: arn:aws:states:::glue:startJobRun.sync
    Parameters:
      JobName.$: "$.job.name"
      ExecutionClass: FLEX
      Arguments:
        "--run_date.$": "$.run_date"
    Retry:
      - ErrorEquals: ["States.ALL"]
        MaxAttempts: 2
        IntervalSeconds: 60
        BackoffRate: 2.0
    Catch:
      - ErrorEquals: ["States.ALL"]
        Next: GlueStandard
    Next: Compact
  GlueStandard:
    Type: Task
    Resource: arn:aws:states:::glue:startJobRun.sync
    Parameters:
      JobName.$: "$.job.name"
      ExecutionClass: STANDARD
      Arguments:
        "--run_date.$": "$.run_date"
    Next: Compact
  Compact:
    Type: Task
    Resource: arn:aws:states:::athena:startQueryExecution.sync
    Parameters:
      WorkGroup: lakehouse
      QueryString: "OPTIMIZE curated.orders REWRITE DATA USING BIN_PACK"
    End: true

The Choice state costs 0.000025 USD per execution and it is the cheapest component in the pipeline. It also ends the argument, which no amount of documentation had managed.

The arithmetic at list prices, November 2026

Price the same 100,000 DPU-hours a month through each meter and the spread is not a rounding error.

{
  "type": "bar",
  "data": {
    "labels": ["Glue 5.1", "Glue 6.0", "Glue 6.0 Flex", "EMR Serverless x86", "EMR Serverless Graviton", "EMR on EC2 m6g"],
    "datasets": [
      {
        "label": "Monthly USD for 100,000 DPU-hours",
        "data": [44000, 30800, 20300, 30295, 24242, 19300],
        "backgroundColor": ["#ef4444", "#f59e0b", "#10b981", "#3b82f6", "#8b5cf6", "#64748b"]
      }
    ]
  },
  "options": {
    "responsive": true,
    "plugins": { "title": { "display": true, "text": "Same compute, six meters (US East list prices, Nov 2026)" } },
    "scales": { "y": { "title": { "display": true, "text": "USD per month" } } }
  }
}

44,000 USD on Glue 5.1. 30,800 on Glue 6.0. 20,300 on Flex. That last gap is the 23,700 USD I opened with, for a workload nobody touched.

Although the table ranks EMR on EC2 first, that rank is the least useful column and I would not defend it as a conclusion. A transient cluster is a pet until you make it not one, and the engineering time to make it not one is a line item no price list contains. Spot instances are cheap for the same reason a discount airline seat is: the airline may want the seat back.

What the table does support is a rule I now apply without much debate. Anything with a deadline, or anything whose output other jobs consume, stays on STANDARD, and Glue 6.0 is the sensible home for it. Everything that can be a day late moves to Flex, with a bounded timeout and a retry attached. Anything with a native library dependency, or a memory ratio Glue will not sell you, goes to EMR Serverless on Graviton.

What this arithmetic does not cover

Every figure above is a list price from November 2026 in US East, not an invoice. Negotiated discounts, Savings Plans on the EC2 layer of EMR and regional differences will move these columns by tens of percent. Paris is not Virginia.

I did not run the TPC-DS benchmark. The 5.61 against 8.87 USD pair is AWS's number on AWS hardware, and 30 workers over 3 TB of clean Parquet says little about a skewed 400 GB table with three hot partitions.

Nor did I measure cold start myself. The 30 to 90 second range comes from published logs on Glue 4.0, and Glue 6.0 may behave differently.

Two claims I would like to test and have not. The first is whether Flex interruption rates in eu-west-3 track the published 5% to 10%. The second is whether the compaction thresholds I use still hold once a table crosses a few thousand partitions. Both are cheap to measure over a month, and neither is in this article.

If you want a longer treatment of the scan side, the earlier French write-up on Glue and scan costs goes further than I could here.

FAQ

Is AWS Glue cheaper than EMR Serverless in 2026? At list price, one Glue 6.0 DPU-hour costs 0.308 USD and an equivalent EMR Serverless x86 worker costs 0.303 USD, so the compute is effectively a tie. EMR Serverless on Graviton is 20% cheaper at 0.242 USD, and Glue Flex is cheaper again at roughly 0.203 USD but can be interrupted mid-run. The decision usually turns on memory sizing and library control rather than on the rate.

What replaced AWS Data Pipeline? AWS stopped taking new customers for AWS Data Pipeline on 25 July 2024 and points existing workloads at Glue, Step Functions or Amazon MWAA. In practice the ETL logic moves to Glue jobs and the dependency graph moves to Step Functions, which is why the state machine design matters more than the engine choice. The old service keeps running for existing customers but receives no new features.

Do I still need Glue jobs if I use zero-ETL? Yes, for anything that requires transformation. Zero-ETL replicates tables from DynamoDB, Aurora, RDS or supported SaaS sources into Redshift and the lakehouse on a 15-minute cycle without custom code. It does not join, deduplicate or conform schemas across sources. Treat it as a replacement for the extraction job, not for the model.

How many state transitions does a Distributed Map cost? One per child workflow execution at 0.000025 USD, plus whatever each child invokes. A million individual children cost about 24.90 USD in parent transitions alone, while batching 100 items per child drops that to roughly 0.25 USD because the compute does not change, only the fan-out. Set MaxItemsPerBatch before you set MaxConcurrency.

When is Glue Flex a bad idea? When a job has a deadline, or when something downstream waits on its output. AWS reports interruption rates of 5% to 10% at peak and recommends against Flex for jobs that other processes depend on. A retry that restarts a two-hour job also re-bills the DPU-hours, which quietly removes the discount.

More generally, the runtime is the visible decision and the layout is the one nobody schedules a meeting about. A team will argue Glue against EMR for a week, then ship a table with 40,000 small files that costs more every day than the argument ever saved. Pick the meter, then fix the layout: the second decision compounds and the first does not.

If you are re-platforming a Spark estate and want a second pair of eyes on the runtime split, this is exactly the work I do on the AWS data engineering side.