Serverless inference cost: Lambda vs SageMaker, real bills

[!NOTE] TL;DR: At 10 million requests a month and a 100 ms median inference, Lambda with ONNX Runtime on arm64 costs about 23 USD, SageMaker Serverless Inference about 40 USD, and one ml.m5.large real-time endpoint about 84 USD. Serverless wins below roughly 21 million requests a month; the always-on endpoint wins above it. The line item that erases the savings is the tax around the model: CloudWatch Logs ingestion and DynamoDB prediction writes add 9 to 29 USD on top of a 23 USD inference bill.

I replaced an always-on scoring endpoint with a serverless stack. Traffic was spiky: 10 million requests a month, most of them between 6 p.m. and midnight. The endpoint had cost 168 USD a month for two ml.m5.large instances across two AZs. The serverless inference cost came back at 23 USD a month. I sent the client an invoice with a modest amount of pride.

Then I switched on the monitoring. Then I added the prediction log. The bill climbed to 52 USD, and it would have reached 62 in staging, where I was logging full payloads. I spent a full afternoon suspecting my cost model before I suspected my logging volume. The model was not the problem.

The question

My main question was arithmetic. What does one serverless inference request actually cost on AWS, once you count the meters around the model and not only the model itself? I wanted the per-option bill at a fixed volume, the break-even volume between options, the size of the tax that never appears in inference pricing pages, and an honest list of what I could not measure.

Sticker shock, itemized

One spontaneously imagines that serverless means cheap. The price tables say something more precise: serverless means you pay per use, and what counts as use includes far more than the forward pass. These are the unit rates I pulled from the AWS Lambda pricing, SageMaker pricing, Step Functions pricing, and CloudWatch pricing pages (us-east-1, October 2026).

Building on those rates, three traps appear before any architecture is drawn.

First, the always-on endpoint is not always the expensive option. Keeping a real-time endpoint up is like renting a shop that stays open 730 hours a month whether or not a customer walks in. A single ml.m5.large costs about 84 USD a month and never sleeps. SageMaker's own staging case (an ml.m5.xlarge at roughly 166 USD a month replaced by serverless inference at roughly 91 USD) shows serverless winning on spiky test traffic, with about 45 percent savings. Run the same arithmetic at production volume, though, and it inverts: above roughly 21 million requests a month at 100 ms, the always-on instance undercuts Serverless Inference. The savings direction depends on volume, not on the word "serverless".

Second, per-request orchestration is a cost trap.

[!WARNING] Do not put a Standard Step Functions workflow in the per-request path. At 10 million requests a month, six state transitions per request bill 1,500 USD, against about 26 USD for an Express workflow on the same volume.

A Standard workflow with six state transitions per request, at 10 million requests, bills 60 million transitions at 0.000025 USD each: 1,500 USD a month for the orchestrator alone. Express workflows bill 1.00 USD per million requests plus duration (rounded up to 100 ms, memory billed in 64 MB chunks), about 26 USD for the same volume. At request granularity, the orchestrator is the most expensive actor in the diagram. More often than not, the router can branch inline and the state machine can wait for the training path, where executions are rare and transitions are nearly free.

Third, the meters around the model are priced like traffic. CloudWatch Logs ingestion at 0.50 USD per GB, DynamoDB writes at 0.625 USD per million, custom metrics at 0.30 USD per metric-month, alarms at 0.10 USD each. None of these appear on the inference pricing pages.

Although the arithmetic favors serverless below the break-even, the approach genuinely works: with idle periods, you pay nothing when nobody calls. The failure is not the model. It is the assumption that the inference line is the bill.

One router, two inference paths

The architecture I landed on routes by model size and owns two inference paths behind one router. A small CPU model (under 6 GB, under the 15-minute Lambda timeout, with a latency budget that tolerates cold starts) runs as ONNX Runtime inside the router Lambda itself. Anything larger, or anything that needs the SageMaker lifecycle, goes to a Serverless Inference endpoint. Sustained volume above the break-even moves to a real-time endpoint. Async and batch traffic queues in SQS and drains through a separate Lambda into S3. Training runs on a schedule through a Standard workflow, on managed spot instances, and deploys through the Model Registry.

flowchart TB
  client([Client]) --> apigw["API Gateway, HTTP API"]
  apigw --> router["Router Lambda, arm64, 2048 MB, SnapStart"]
  apigw -->|"async or batch"| sqs["SQS standard queue"]
  router -->|"model under 6 GB and CPU"| onnx["ONNX Runtime inside Lambda"]
  router -->|"larger model or SageMaker lifecycle"| ssi["SageMaker Serverless Inference, 2 GB, max concurrency 200"]
  sqs --> batch["Batch Lambda, arm64"]
  batch --> s3[("S3 predictions")]
  router --> ddb[("DynamoDB on-demand, features and predictions")]
  ssi --> ddb
  schedule["EventBridge schedule"] --> sfn["Step Functions, Standard"]
  sfn --> train["SageMaker Training job, ml.m5.xlarge, managed spot"]
  train --> registry["Model Registry"]
  registry --> deploy["Deploy to serverless endpoint"]
  router --> cw["CloudWatch Logs and custom metrics"]
  ssi --> cw
  cw --> alarm["CloudWatch Alarms"]
  alarm --> sns["SNS topic"]
  sns --> oncall([On-call])

The quotas are the real specification. Read them before you draw boxes. Lambda gives 1,000 concurrent executions per region by default, a 6 MB synchronous payload, a 900-second timeout, and 128 MB to 10,240 MB of memory. Serverless Inference caps at 200 concurrent invocations per endpoint (shared 1,000 per region across endpoints in major regions), a 4 MB payload, 60 seconds of processing, and memory options from 1,024 MB to 6,144 MB. For bursty small-model traffic, Lambda scales further without a quota increase. For everything else, the endpoint is the ceiling.

This is the shape of the pipelines I run in production: a router, a queue, a state machine, and one log group per function, correlated by request ID in CloudWatch Logs Insights.

The cost model, then the stack

I built the cost model before the stack, because the decision is arithmetic. The model is a small Pydantic program: you describe the workload, and it returns the monthly bill per option plus the observability tax. All rates are parameters, so it survives the next price change.

from pydantic import BaseModel, Field, field_validator
ARM_GB_SECOND = 0.0000133334
REQUEST_PER_MILLION = 0.20
FREE_REQUESTS = 1_000_000
FREE_GB_SECONDS = 400_000
class Workload(BaseModel):
    model_config = {"strict": True}
    requests_per_month: int = Field(..., ge=1)
    median_duration_s: float = Field(..., gt=0.0, le=900.0)
    memory_gb: float = Field(..., ge=0.125, le=10.0)
    cold_start_rate: float = Field(default=0.0, ge=0.0, le=1.0)
    cold_start_init_s: float = Field(default=0.7, gt=0.0)
    log_kb_per_request: float = Field(default=1.0, gt=0.0)
    log_ingest_usd_per_gb: float = Field(default=0.50, gt=0.0)
    log_free_gb: float = Field(default=5.0, ge=0.0)
    ddb_write_usd_per_million: float = Field(default=0.625, gt=0.0)
    ddb_read_rru_per_request: float = Field(default=0.5, gt=0.0)
    ddb_read_usd_per_million: float = Field(default=0.125, gt=0.0)
    @field_validator("memory_gb")
    @classmethod
    def snap_to_increment(cls, v: float) -> float:
        if abs(v * 1024 - round(v * 1024)) > 1e-9:
            raise ValueError("memory_gb must map to a 1 MB increment")
        return v
    def lambda_monthly_usd(self) -> float:
        warm = self.requests_per_month * self.median_duration_s * self.memory_gb
        cold = self.requests_per_month * self.cold_start_rate * self.cold_start_init_s * self.memory_gb
        billable_gb_seconds = max(warm + cold - FREE_GB_SECONDS, 0.0)
        billable_requests = max(self.requests_per_month - FREE_REQUESTS, 0)
        return billable_gb_seconds * ARM_GB_SECOND + billable_requests * REQUEST_PER_MILLION / 1_000_000
    def serverless_inference_monthly_usd(self, usd_per_second: float = 0.00004, data_gb: float = 10.0, usd_per_gb: float = 0.016) -> float:
        return self.requests_per_month * self.median_duration_s * usd_per_second + data_gb * usd_per_gb
    def realtime_monthly_usd(self, instance_usd_per_hour: float = 0.115, hours: int = 730, azs: int = 1) -> float:
        return instance_usd_per_hour * hours * azs
    def observability_monthly_usd(self) -> float:
        log_gb = self.requests_per_month * self.log_kb_per_request / 1_000_000
        logs = max(log_gb - self.log_free_gb, 0.0) * self.log_ingest_usd_per_gb
        writes = self.requests_per_month * self.ddb_write_usd_per_million / 1_000_000
        reads = self.requests_per_month * self.ddb_read_rru_per_request * self.ddb_read_usd_per_million / 1_000_000
        return logs + writes + reads
workload = Workload(requests_per_month=10_000_000, median_duration_s=0.1, memory_gb=2.0, cold_start_rate=0.05)
print(workload.lambda_monthly_usd(), workload.serverless_inference_monthly_usd(), workload.observability_monthly_usd())

Then the infrastructure, in Terraform. The router runs arm64 with 2,048 MB and SnapStart on published versions. The endpoint, the predictions table, and one alarm wired to SNS are the rest of the moving parts.

resource "aws_iam_role" "lambda" {
  name = "scoring-router-role"
  assume_role_policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{ Effect = "Allow", Principal = { Service = "lambda.amazonaws.com" }, Action = "sts:AssumeRole" }]
  })
}
resource "aws_s3_bucket" "models" {
  bucket = "scoring-models-prod"
}
resource "aws_lambda_function" "router" {
  function_name = "scoring-router"
  role          = aws_iam_role.lambda.arn
  architectures = ["arm64"]
  memory_size   = 2048
  timeout       = 30
  runtime       = "python3.13"
  handler       = "router.handler"
  filename      = "router.zip"
  snap_start {
    apply_on = "PublishedVersions"
  }
  environment {
    variables = {
      MODEL_BUCKET      = aws_s3_bucket.models.id
      PREDICTIONS_TABLE = aws_dynamodb_table.predictions.name
    }
  }
}
resource "aws_sagemaker_endpoint_configuration" "serverless" {
  name = "scoring-serverless"
  production_variants {
    model_name   = "scoring-onnx" # created by the training pipeline, registered in the Model Registry
    variant_name = "AllTraffic"
    serverless_config {
      memory_size     = 2048
      max_concurrency = 200
    }
  }
}
resource "aws_sagemaker_endpoint" "serverless" {
  name                 = "scoring-serverless"
  endpoint_config_name = aws_sagemaker_endpoint_configuration.serverless.name
}
resource "aws_dynamodb_table" "predictions" {
  name         = "predictions"
  billing_mode = "PAY_PER_REQUEST"
  hash_key     = "request_id"
  attribute {
    name = "request_id"
    type = "S"
  }
}
resource "aws_sns_topic" "oncall" {
  name = "scoring-oncall"
}
resource "aws_cloudwatch_metric_alarm" "lambda_throttles" {
  alarm_name          = "scoring-router-throttles"
  namespace           = "AWS/Lambda"
  metric_name         = "Throttles"
  statistic           = "Sum"
  period              = 60
  evaluation_periods  = 1
  threshold           = 10
  comparison_operator = "GreaterThanThreshold"
  alarm_actions       = [aws_sns_topic.oncall.arn]
}

Ten million requests, four bills

The workload: 10 million requests a month, 100 ms median inference, 2 GB of memory, us-east-1, published rates. The results are modeled from the price tables above, not measured on my stack. The chart says what the paragraphs confirm: serverless wins the month.

{
  "type": "bar",
  "data": {
    "labels": ["Lambda ONNX arm64, base", "Lambda ONNX, 5 percent cold starts", "SageMaker Serverless Inference, 2 GB", "Real-time ml.m5.large, 1 AZ", "Real-time ml.m5.large, 2 AZ"],
    "datasets": [
      {
        "label": "Monthly inference cost in USD at 10M requests, 100 ms median",
        "data": [23.13, 32.47, 40.16, 83.95, 167.9],
        "backgroundColor": ["#3b82f6", "#3b82f6", "#f59e0b", "#9ca3af", "#9ca3af"]
      }
    ]
  },
  "options": {
    "responsive": true,
    "plugins": {
      "title": {
        "display": true,
        "text": "Inference bill per option, 10M requests per month, us-east-1 published rates"
      }
    },
    "scales": {
      "y": {
        "title": {
          "display": true,
          "text": "USD per month"
        }
      }
    }
  }
}

The Lambda option: 2.0 million GB-seconds of warm compute plus 0.7 million GB-seconds of cold-start init at a 5 percent cold rate, minus the 400,000 GB-second free tier, at 0.0000133334 USD per GB-second: 30.67 USD. Requests add 1.80 USD after the free million. Total 32.47 USD with cold starts, 23.13 USD without.

The Serverless Inference option: 1 million seconds at 0.00004 USD plus 10 GB of data at 0.016 USD per GB: 40.16 USD. This matches the worked example on the SageMaker pricing page, which is a useful sign that the model is honest.

The real-time option: 730 hours at 0.115 USD per hour: 83.95 USD for one AZ, 167.90 USD for two, plus data. Break-even against a single ml.m5.large: about 20.9 million requests a month for Serverless Inference, about 36 million for the Lambda base case, about 26 million once cold starts are priced in.

The tax. Lean logging at 1 KB per request: 2.50 USD of Logs ingestion (after the 5 GB free tier), 6.25 USD of DynamoDB writes, 0.63 USD of reads: 9.38 USD. Fat logging at 5 KB per request: 22.50 USD of ingestion, for a tax of 29.38 USD. On a 23.13 USD inference bill, the tax is 40 to 127 percent of the line it watches. The tax is the second line item. More importantly, it scales with traffic, so it punishes exactly the growth you wanted.

The observability tax works like a taxi idling outside a stadium: the fare isn't for the ride, but for the engine running while you watch. The busier the stadium, the more taxis idle. The meter climbs whether or not anyone scores.

Two findings I did not expect. First, SnapStart on Python is a latency purchase, not a cost saving! The cache charge is 0.0000015046 USD per GB-second: a 2 GB snapshot held for 31 days costs 8.06 USD a month. The restore charge is 0.0001397998 USD per GB, so at a 5 percent cold rate the restores alone cost 139.80 USD, against 9.33 USD of billed init duration without SnapStart. Java runtimes pay nothing. Python and .NET pay for both cache and restore. Second, the per-request orchestration trap is real but avoidable: the router branches inline, and the state machine only runs where executions are rare.

Although the Lambda option won every line of the arithmetic, my tests confirm it is not free of operational cost. I now own an ONNX export step, a layer build, a memory sweep, and a versioning scheme. Serverless Inference costs about 1.7 times more per request at this size and would have owned all four.

Limits and what remains to be tested

The 100 ms median is an assumption for a small INT8 model, not a measurement of my stack. A published Lambda benchmark measured 590 ms warm for a 24 MB INT8 YOLO model at 2 GB, with 600-900 ms cold starts; my model is smaller, yours may not be. The 5 percent cold-start rate is an assumption too. Measure it: cold starts show up as Init Duration in the REPORT lines, and AWS bills the INIT phase on on-demand invocations. All rates are us-east-1 list prices retrieved on 26 October 2026; other regions differ. Prices change. I modeled the Lambda and CloudWatch free tiers but not the Serverless Inference free tier. I didn't benchmark arm64 against x86 for this model; arm64 is about 20 percent cheaper per GB-second at equal duration. The SnapStart arithmetic is derived from published rates, not from a measured bill; it could easily be tested on a staging function with a forced cold rate. It remains to be seen whether the break-even holds above 6 GB of model, where only real-time endpoints apply.

If your payloads are large or your latency budget is loose, the async path has its own trap, which I covered in the article on the SageMaker async inference scale-to-zero trap. For the pipeline that feeds all of this, see the event-driven serverless MLOps pipeline write-up.

The principle that survives

More generally, a serverless bill is a traffic bill plus a watching bill, and the second grows with the first. Price the model, then price the meters around it, then choose the architecture. The cheapest request is the one you can explain line by line. Cheap is a number, not a feeling.

If you are weighing a serverless inference stack for a spiky workload, I can run this cost model against your real traffic profile, or build the pipeline itself. Reach out through the contact page or see what I ship on the products page.