Every few months the serverless-vs-containers debate flares up again. Most of the takes are wrong — not because they're inaccurate, but because they treat the choice as universal rather than workload-specific.
For AI workloads specifically, the trade-offs look different than for standard API services. Here's how I approach the decision in practice.
The Core Trade-off
Serverless (Lambda, Cloud Functions, Azure Functions): You provide code + dependencies. The platform manages the runtime, scaling, and availability. You pay per invocation. Cold starts are the main downside.
Containers (ECS, Cloud Run, Kubernetes): You provide a Docker image. The platform runs it. You control the environment completely. You pay for runtime (even idle time), but warm instances mean no cold starts.
For typical CRUD APIs, serverless usually wins on simplicity and cost at low-to-medium volume. For AI workloads, it's more nuanced.
Why AI Workloads Are Different
Dependency size matters enormously
A standard Lambda function might have a 10MB deployment package. A Python function that imports torch, transformers, and numpy is looking at 2–3GB of dependencies.
Lambda has a 250MB unzipped deployment limit (10GB via container image, but the cold start impact is significant). Loading a PyTorch model from S3 or EFS on every cold start adds 15–60 seconds depending on model size.
For inference endpoints that need to respond in under 2 seconds, cold starts are a dealbreaker unless you're using Lambda SnapStart (Java only), provisioned concurrency (expensive), or container images with aggressive model caching.
Verdict on dependency size: Containers win for large model workloads. Serverless wins if you can keep dependencies lean (calling an external inference API rather than running the model yourself).
GPU access
Lambda doesn't support GPUs. Period. If your AI workload requires GPU inference — and you're not using a managed inference service — you need containers or VMs.
The practical options for GPU inference:
- AWS SageMaker endpoints (managed, expensive)
- ECS with GPU-enabled EC2 instances (flexible, requires more ops)
- Cloud Run with T4 GPUs (GCP, simpler than ECS, limited GPU types)
- Modal or Replicate (purpose-built ML deployment, abstracts the infra)
- Direct EC2 (most control, most ops work)
Verdict on GPU: If you need GPU, serverless standard is out. Use containers or a managed ML platform.
Latency sensitivity
AI inference — especially LLM inference — can take 2–30 seconds depending on model and output length. This changes the latency calculus.
For synchronous requests where users are waiting on a response:
- A 3-second cold start matters a lot if the inference itself takes 2 seconds (total 5s feels broken)
- A 3-second cold start matters less if inference takes 25 seconds anyway
For async workloads (batch processing, background jobs, webhooks that trigger AI enrichment):
- Cold starts barely matter — the job starts when it starts
- Serverless is often ideal here: invoked by SQS/EventBridge, runs to completion, scales automatically
Verdict on latency: For synchronous user-facing AI, containers with always-warm instances. For async/batch AI, serverless is often the right call.
Decision Framework I Actually Use
Is the workload async/batch?
YES → Serverless (Lambda + SQS/EventBridge)
NO → Continue...
Does it require GPU?
YES → Containers (ECS, Cloud Run) or managed ML platform
NO → Continue...
Are dependencies < 100MB?
YES → Serverless is viable
NO → Continue...
Are you calling an external inference API (OpenAI, Anthropic, Bedrock)?
YES → Serverless — you're just making HTTP calls
NO → Containers (your own model = large deps + potential GPU)
Most LLM-powered applications I build today fall into the "calling external inference API" category — which means serverless is perfectly fine. The LLM provider handles the GPU and the model; I'm just sending requests.
The Pattern I Use Most
For production AI applications in 2026, my default architecture is:
Orchestration layer: AWS Lambda (or Vercel serverless functions for Next.js) — handles request routing, auth, and calls out to inference APIs. Deps are small (just SDK clients), cold starts are under 1s.
AI inference: External API (Anthropic, OpenAI, Bedrock, or Replicate) — managed, scalable, no infra to maintain.
Heavy processing (embeddings generation, batch jobs, data pipelines): ECS Fargate with spot instances — containerized, scales to zero when not needed, GPU available if needed.
Scheduled/event-driven jobs: Lambda triggered by EventBridge or SQS.
This hybrid approach avoids the false binary of "all serverless or all containers." Use each where it fits.
When to Change This Default
Use containers everywhere when:
- Your team has strong Docker/Kubernetes expertise and wants operational consistency
- You're running your own models and need the flexibility
- Regulatory requirements mandate you don't use external inference APIs
- You need consistent low-latency without provisioned concurrency costs
Use serverless everywhere when:
- You're an early-stage team optimising for speed and low ops burden
- All AI inference goes through external APIs
- Your workloads are highly variable (lots of idle time + traffic spikes)
- Cost predictability at low volume is important
Practical Notes on Cold Starts
If you're on serverless and cold starts are a problem, the options in order of preference:
- Reduce package size: Lazy imports, tree-shaking, separate Lambda per function
- Use Provisioned Concurrency for critical paths (adds cost, eliminates cold starts)
- Ping the function on a schedule to keep it warm (hacky but works)
- Move the affected function to a container (Cloud Run with min-instances=1 is effectively always warm at modest cost)
For AI workloads specifically, option 4 is often the right answer — model loading is the dominant cold start cost, and you're better off paying for a container that stays warm than fighting Lambda cold start limits.
The architecture decision isn't about which technology is better. It's about matching the operational characteristics of each to what your workload actually needs. Reach out if you're working through this choice for a specific system.