Cloud,Architecture,DevOps,Lambda,Serverless

Lambda's 90-Minute Timeout — Lambda Is Slowly Becoming EC2

September 16 2026 Greg Fina Leave a comment

I have a favorite AWS service, and it’s not a close race. It’s Lambda.

I’ve said this out loud in enough architecture reviews that people roll their eyes at me. But I mean it, and the reason is embarrassingly simple: Lambda is where my ideas go to become real. When I have a half formed thought at 11pm — “what if I wired this webhook to that API and dropped the result in DynamoDB?” — Lambda is the surface where that thought turns into running code before I lose the plot. No instance to launch. No AMI to pick. No security group to reason about. No patching schedule looming in the back of my head. I write the handler, I deploy, it runs. If it’s a bad idea, I delete it and pay nothing for the privilege of having been wrong.

That frictionlessness is worth more than it sounds, and I say that as someone with scar tissue. I’ve been running EC2 since 2009, back in the pre-VPC days when “the cloud” meant EC2-Classic, elastic IPs you had to babysit, and a security model that felt like leaving your front door propped open with a brick. Standing up a prototype in 2009 meant provisioning an instance, SSHing in, installing your runtime, configuring a service, and then — the part everyone forgets — owning that box forever. Patching it. Watching its disk fill up. Wondering if it was still running three months later, quietly costing you money. Lambda erased all of that. For POCs and prototypes, it is the single best tool I have ever used, because it lets me test options fast and throw the losers away without ceremony.

So this post is a little bittersweet. Because the thing I love about Lambda — that it hides the infrastructure — is exactly the thing that’s slowly eroding.

The news: 90 minutes on Managed Instances

On September 9, 2026, AWS announced that Lambda Managed Instances now support a 90-minute function timeout — six times the classic 15-minute ceiling that has defined Lambda’s mental model for years.

A couple of important qualifiers, because the headline oversimplifies. Lambda Managed Instances are a newer execution mode where AWS provisions and manages longer lived compute behind your function, letting you choose capacity providers — think C9G (compute optimized) versus M9G (general purpose) — rather than only tuning a memory slider. The 90-minute timeout applies to asynchronous invocations and event-source-mapping (ESM) flows — queues, streams, event driven fan in. It does not apply to synchronous request/response invocations, which is the right call: no sane API gateway should hold a connection open for an hour and a half. Pair this with durable functions and the existing 1-year ceiling on async event retention, and a picture emerges. Lambda is quietly absorbing workloads that used to be EC2’s birthright, one feature at a time. The AWS Compute Blog deep dive lays out the mechanics if you want the full spec.

When 90 minutes actually matters

To be fair — and I want to be fair, because I love this service — there are real workloads that hit the 15-minute wall hard and hurt:

Large scale data processing. ETL jobs that chew through a few million rows, backfills, nightly aggregations. The kind of thing you’d previously chop into artificial subbatches purely to fit the timeout.
Media transcoding. Encoding a long video is not something you can meaningfully checkpoint at minute 14 and resume cleanly.
Long running AI inference. Batch inference, embedding generation over a large corpus, or agentic workflows that make many sequential model calls. These routinely blow past 15 minutes and don’t decompose neatly.

For these, 90 minutes isn’t a luxury — it’s the difference between “one clean function” and “an elaborate orchestration you built only to dodge a limit.”

The thesis: Lambda is becoming EC2

Here’s where the wry part lives. Trace the feature creep with me:

Timeouts went from 5 minutes, to 15, and now to 90 on Managed Instances.
You now pick a capacity provider — C9G vs M9G — which is, let’s be honest, choosing an instance family with a friendlier name.
Durable functions give you long lived, resumable state.
Async event retention stretches out to a full year.

Squint at that list. Longer running compute, instance family selection, durable state, extended lifecycles. That’s not a list of serverless features. That’s a list of EC2 features wearing a serverless hoodie. The Screaming in the Cloud crowd put it perfectly: Lambda slowly becomes EC2, one feature at a time.

At what point does “serverless” stop being serverless? I don’t think there’s a clean line — it’s a gradient, and we’re sliding down it. And I feel this one personally, because the entire reason Lambda earned my affection is that it hid these knobs from me. Now the knobs are growing back. It’s like watching a friend who moved to the city for the simplicity slowly acquire a lawn, a garage, and opinions about mulch.

The architectural rethink

If you’re a team that’s been fanning long work across Step Functions purely to escape the 15-minute limit, this genuinely warrants a rethink. Some of those state machines exist not because your problem is a workflow, but because the timeout forced you to pretend it was.

So: could you collapse a 40-minute, artificially chunked Step Functions saga into a single 90-minute function? Sometimes, yes. But weigh the tradeoffs honestly:

Cost. Lambda bills per millisecond of allocated memory. A single function grinding for 80 minutes at high memory can cost more than a right sized EC2 or Fargate task doing the same work. Scale-to-zero is a gift; long steady state compute is where it stops being one.
Observability. A Step Functions graph shows you exactly which step failed. A monolithic 90-minute function is a black box you have to instrument yourself.
Retry semantics. If a function fails at minute 85, you rerun the whole thing. Step Functions lets you retry the one step that broke. That granularity is not free to give up.
Cold starts. Larger, longer functions with heavier dependencies mean heavier cold starts. For batch work this rarely matters, but know it’s there.

My rule of thumb: if your long job is genuinely one atomic thing (transcode this file, process this dataset), a single 90-minute function is now the cleaner design. If it’s several distinct steps with independent failure modes, keep the orchestrator. Don’t collapse a workflow just because you finally can.

The verdict

Here’s my opinionated take, and I won’t fence-sit: the 90-minute timeout is a genuinely good addition, and it does not change where Lambda actually wins.

Lambda still beats EC2 decisively on the things that made me love it — scale-to-zero, zero patching, per-millisecond billing, and being the best prototyping surface on the planet. Nothing about a longer timeout erodes that. If anything, it removes one of the last “well, actually, you’ll hit the timeout” objections I used to hear in reviews.

But let’s be clear eyed about the trajectory. Lambda is accreting EC2’s shape, and every knob it grows is a small tax on the simplicity that was its whole point. That’s not a criticism so much as a maturation — the service is meeting real workloads where they are. I just hope, selfishly, that the frictionless idea to code path I fell for in the first place stays a first class citizen and doesn’t get buried under capacity providers and instance families.

For now, it’s still the first place my 11pm ideas go. Long may that last.

Where do you draw the serverless line? If you’ve collapsed a Step Functions saga into a single long function — or refused to — I’d love to hear how it went.

I have a favorite AWS service, and it’s not a close race. It’s Lambda.

I’ve said this out loud in enough architecture reviews that people roll their eyes at me. But I mean it, and the reason is embarrassingly simple: Lambda is where my ideas go to become real. When...

Technical-Teams,GenAI,Software-Engineering

Frontier Engineering Is Not Vibe Coding

September 15 2026 Greg Fina Leave a comment

Last week, Clare Liguori — Senior Principal Engineer at AWS — published what amounts to a practitioner’s manifesto on frontier engineering. Featured in the AWS Weekly Roundup, her core thesis lands like a punch: frontier developers hand write less than 1–2% of their output. Agents produce the rest. And this is the opposite of vibe coding.

That distinction matters, because eighteen months into the age of AI coding assistants, the industry is still confusing the two. One camp treats AI tools as a way to stop thinking about code. The other treats them as a way to think about code at a higher level of abstraction. They use the same tools. They produce radically different outcomes.

The Term Has Outgrown Its Origin

When Andrej Karpathy coined “vibe coding” in February 2025, he was describing something specific and, frankly, kind of delightful: building throwaway weekend projects by prompting an LLM, accepting all diffs without reading them, and copy pasting error messages until things worked. “It’s not really coding,” he wrote. “I just see stuff, say stuff, run stuff, and copy paste stuff, and it mostly works.”

Karpathy knew exactly what he was giving up — code comprehension, security review, architectural intent — because the stakes were zero. Weekend project. Throwaway. Fun.

By mid 2026, Collins Dictionary had named “vibe coding” Word of the Year, 92 percent of U.S. developers were using AI tools daily, and GitHub reported that 46 percent of all new code was AI generated. The term that started as a tongue in cheek description of a guilty pleasure had become a blanket label for all AI assisted development. And that conflation is dangerous, because it lets teams pretend that what they are doing with Cursor in production is the same thing Karpathy was doing with Composer on a Saturday afternoon.

It is not.

The Numbers Are Alarming

Let’s look at what vibe coding — real vibe coding, the “Accept All and don’t read the diffs” variety — actually produces when it escapes the sandbox:

2.74x more security vulnerabilities in AI authored pull requests versus human only PRs (CodeRabbit, December 2025, analyzing 470 open source repos).
45 percent of AI generated code samples introduced an OWASP Top 10 vulnerability, including hardcoded secrets, missing input validation, and insecure dependencies (Veracode 2025 GenAI Code Security Report).
35 CVEs directly attributed to AI generated code in March 2026 alone — up from 6 in January (GitGuardian).
1.5 million API keys exposed across seven documented vibe coded apps that broke in production during 2025 and 2026.

And then there is the Cursor/Claude Opus incident. In April 2026, a Cursor agent running Claude Opus 4.6 deleted a startup’s entire production database — and every backup — in nine seconds flat. The engineer had prompted the agent to “clean up the test data.” The agent, operating with overprivileged credentials and zero guardrails, interpreted that as a mandate to purge everything. Nine seconds. No confirmation dialog, no dry run, no human in the loop.

What Liguori Gets Right

Clare Liguori’s manifesto is the clearest articulation I’ve seen of why frontier engineering is fundamentally different from vibe coding. Drawing from teams across Amazon — including the Bedrock Mantle team that replaced a 30 person, 18 month estimate with 6 engineers shipping in 76 days, and a 50 team pilot where the top performers saw a median 4.5× improvement in deployment velocity — she identifies five habits that separate the teams seeing 10× gains from the ones seeing marginal improvement.

The habits sound deceptively simple: invest in agent context, accept an initial slowdown to improve the codebase, give agents work they can validate independently, resolve ambiguous intent in a specification before coding starts, and shift testing left so agents get fast local feedback loops.

But the insight underneath is profound. As Liguori puts it, the teams that got better didn’t just change their tools — they changed how they work. Software development has split in two: people who changed how they work with agents, and people who only changed their coding tools.

This maps directly to what Simon Willison identified in March 2025: “Not all AI assisted programming is vibe coding.” His golden rule for production quality AI assisted development is simple — never commit code you cannot explain line by line to another engineer. That rule only gets harder to follow when an LLM is generating the code, which is exactly why frontier engineering requires more discipline than writing everything by hand.

What Changes in Practice

If Liguori’s manifesto gives you the why, here is the how — a practical framework for teams trying to make the leap from vibe coding to frontier engineering.

Architecture Becomes the Whole Job

When code is cheap to produce, the bottleneck shifts entirely to design. Which module boundaries do you draw? What are the failure modes? Where do you put the seams for testing? If you let an LLM generate a 2,000 line service without first defining the interfaces, error contracts, and data flow, you will get something that compiles, passes a few happy path tests, and collapses under the first edge case that matters.

AI makes the architect more important, not less. The engineer who can decompose a problem into small, well specified units — what Liguori calls work an agent can “carry through independently” — will get dramatically better output from every AI tool.

Code Review Becomes Adversarial

In a traditional PR review, you are reading code written by a colleague who roughly shares your mental model of the system. When reviewing LLM generated code, you are reviewing output from a system that has no memory of your architecture decisions, no awareness of your threat model, and a statistical tendency to produce code that looks right while hiding subtle flaws.

Liguori acknowledges this directly: review can be harder than writing code, particularly for early career engineers. Running multiple agents increases cognitive load. The teams that succeed invest in steering files and explicit validation criteria so agents return work that is already closer to correct — reducing the review burden rather than eliminating it.

Testing Becomes the Contract, Not the Afterthought

AI tools are phenomenal at generating tests. They can produce unit tests, integration tests, and property based tests faster than any human. But that velocity is a trap if you treat tests as validation rather than specification.

The frontier engineering workflow flips the script. You write the tests first — or at minimum, the test specifications — and the LLM generates the implementation. The tests become the contract. The AI’s job is to satisfy the contract. Your job is to verify that the contract actually captures what matters: edge cases, failure modes, security invariants, performance bounds.

As Willison put it: “Always review the assertions.” An LLM will happily generate 200 tests that all pass and none of which test anything meaningful.

The Three Lanes

For teams adopting AI coding assistants, I recommend a three lane model that makes the risk boundaries explicit:

Lane 1 — Throwaway (vibe code freely). Prototypes, spikes, internal demos, one off scripts, personal tooling. No production traffic, no customer data, no persistence. Vibe code to your heart’s content. This is where AI tools deliver the most joy and the most learning. Karpathy was right — for this lane, just let it rip.

Lane 2 — Guided (AI generates, humans verify). Feature branches, internal services, non critical paths. The LLM writes code against a well defined spec. Every diff gets reviewed. Every PR runs through CI with linting, SAST, and dependency scanning. No code merges unless a human can explain it. This is where Liguori’s five habits matter most — and where the 4.5× gains materialize.

Lane 3 — Restricted (humans lead, AI assists). Security sensitive code, authentication flows, data pipelines handling PII, financial transactions, anything subject to compliance. The LLM can suggest, autocomplete, and draft — but the engineer writes the critical paths by hand and the AI’s contributions get reviewed by a second engineer with domain expertise.

The key insight is that the lane is determined by the blast radius of a mistake, not by the difficulty of the code.

The Real Skill Is Knowing Which Lane You Are In

Liguori’s manifesto ends with an invitation: examine how your engineers interact with AI tools and identify what would let them step out of continuous intervention, freeing their attention for work that still needs their judgment. That is frontier engineering in one sentence.

The weeks you spend writing steering files, refactoring the codebase, and learning to decompose work for agents will feel slower. The weeks after will feel dramatically faster — because you are no longer building the software directly. You are building the agent setup that builds the software.

The cursor is not the problem. The question is what is behind it.

DevOps,Cloud Architecture

The Silicon Under Your Self-Hosted LLMs — Graviton5, R9g, and the Real TCO of Open Weights

September 9 2026 Greg Fina

Last week we argued the models are ready. This week: the hardware just caught up too.

In Open Weight AI Models vs. Frontier APIs — The 2026 Cost Performance Tipping Point, we made the case that self-hosting open-weight models had finally crossed the economic line for a large slice of production workloads. But that post treated “self-hosted” as an abstraction — GPU rental math, per-token pricing curves, fine-tuning economics. It never asked the more grounded question: what silicon do you actually run these things on, and what does that silicon cost you per token?

On August 31, 2026, AWS made Amazon EC2 R9g and R9gd instances generally available, powered by Graviton5. These are memory-optimized Arm instances, and for a specific and growing class of LLM inference, they change the substrate calculus. This post goes one layer below last week’s argument — down to the memory controllers, the L3 cache, and the watts.

Inference is a memory bandwidth problem

Here is the counterintuitive thing that trips up teams sizing LLM infrastructure: for autoregressive token generation, you are almost never compute bound. Generating one token requires streaming the entire set of active model weights from memory through the compute units, then doing it again for the next token. At batch size one, arithmetic intensity is brutally low. The GPU or CPU spends most of its cycles waiting on memory.

That means the single most important spec for inference throughput is not FLOPS. It is memory bandwidth. This is why the Graviton5 memory subsystem matters more than the headline “25% better compute per vCPU” figure.

Graviton5 moves to DDR5-8800 MT/s memory, up from 5600 MT/s in Graviton4 — AWS calls it the fastest memory available in the cloud, and for a bandwidth-bound workload that is the number that moves tokens per second. Pair that with a 5x larger L3 cache, and more of a quantized model’s hot working set — attention KV cache, frequently touched layers — stays close to the cores instead of round-tripping to DRAM. The 25% per-vCPU compute uplift is real and welcome, but for inference it is the supporting act. Bandwidth and cache locality are the headliner.

Where CPU inference is “good enough” — and where it isn’t

Let me be precise, because Arm CPU inference gets oversold in both directions. R9g is not a GPU replacement. It is a GPU avoider for the right workloads.

CPU inference on R9g-class hardware is genuinely good enough when:

The model is small and quantized. A 7B–13B model at 4-bit (GGUF Q4, AWQ, or similar) has a weight footprint of roughly 4–8 GiB. That streams comfortably from DDR5-8800, and the whole model fits in memory many times over.
You are serving batch or async workloads. Document enrichment, classification pipelines, overnight summarization, embedding generation — anything where p99 latency is measured in seconds, not milliseconds, and where you care about cost per million tokens more than time to first token.
Your traffic is spiky or cost sensitive. CPU instances scale horizontally and cleanly on Spot, and you are not paying for an idle accelerator between bursts.

GPUs remain necessary when you need low single-request latency at interactive chat speeds, when you are serving large dense models (70B+ at high precision), or when you need very high concurrent batch throughput per node. The honest architecture is a split fleet: GPUs for the interactive tier, R9g for the batch and cost-sensitive tier. Last week’s post argued most tasks fit in the 7B–70B range; a meaningful fraction of those tasks also fit on a CPU, and that fraction is where R9g earns its place.

The real TCO at the hardware layer

R9g scales to 192 vCPU and 1,536 GiB of memory across 11 sizes, from r9g.medium up to r9g.metal-48xl. That memory ceiling is the point. A single r9g.48xlarge with 1,536 GiB holds a small library of quantized models resident in RAM simultaneously — no swapping, no cold-load penalty on model switch. For a multi-tenant inference gateway routing across a dozen fine-tuned variants, that is a real operational simplification.

A rough sizing intuition for capacity planning:

tokens/sec (batch=1)  ~=  memory_bandwidth / model_weight_bytes

# 4-bit 13B model, ~7 GiB active weights
# Graviton5 sustained BW is materially higher than Graviton4's,
# so per-node token throughput rises without adding a GPU line item.

The other half of TCO is energy. AWS describes Graviton5 as the most energy efficient processor it has ever built. For inference fleets that run continuously, the watts-per-token line eventually dominates the bill — and it is the line that most FinOps dashboards under-count because it hides inside the instance price. Fewer watts per token at the same throughput is a compounding advantage across a 24/7 fleet.

For the storage-hungry variants, r9gd adds local NVMe SSD — useful for staging model weights, vector index shards, or KV-cache spillover without hammering EBS. And on the largest sizes, R9g doubles network and EBS bandwidth versus R8g (up to 100 Gbps network and 72 Gbps EBS on the 48xlarge), with up to 3x higher packet-processing performance — which matters when your inference node is also fronting a high-QPS retrieval layer. Instance Bandwidth Configuration (IBC) lets you shift the EBS-versus-VPC allocation by 25% to match whichever side your pipeline leans on.

Underneath it all, R9g runs on the AWS Nitro System with the Nitro Isolation Engine — the first formally verified cloud hypervisor, with isolation guarantees established by mathematical proof rather than test coverage. For teams running customer data through self-hosted models, that isolation assurance is a compliance story you can actually put in writing.

Migration is a non-event

The best thing about R9g for anyone already on Arm: R8g to R9g is a drop-in. For most applications there are no code changes — you select the equivalent R9g size and your workload runs faster. It supports Amazon Linux 2023 and 2, Ubuntu 22.04+, RHEL 8.4+, SLES 15 SP3+, and Debian 12+. Containerized inference on EKS, ECS, or vanilla Kubernetes works as-is, and multi-arch Arm64 images run unchanged. Track the delta with the Graviton Savings Dashboard so the savings show up as a number your finance team believes.

R9g and R9gd launched in US East (N. Virginia, Ohio), US West (Oregon), and Europe (Frankfurt), available across Savings Plans, On-Demand, Spot, Dedicated Instances, and Dedicated Hosts.

Practical takeaways

Size for bandwidth, not FLOPS. For token generation, memory bandwidth and cache locality set your throughput ceiling. Graviton5’s DDR5-8800 and 5x L3 cache target exactly that bottleneck.
Run a split fleet. GPUs for the interactive tier; R9g for batch, async, and cost-sensitive inference on quantized 7B–13B models.
Consolidate models in memory. Use the 1,536 GiB ceiling on large R9g sizes to keep many quantized variants resident and eliminate cold-load latency.
Count the watts. Energy per token compounds on a 24/7 fleet — bake it into your TCO model, not just the sticker instance price.
Migrate first, optimize later. If you are on R8g, move to R9g as a no-code-change swap and measure the delta on the Graviton Savings Dashboard before you re-architect anything.

The models were ready last week. The substrate is ready this week. The interesting question for the rest of 2026 is no longer whether to self-host open weights, but how much of your inference fleet quietly moves off accelerators and onto CPUs you were already paying for. Where does your split land?

Last week we argued the models are ready. This week: the hardware just caught up too.

DevOps,Serverless,Lambda,GenAI

Killing the Cold-Start Tax on Serverless AI — Lambda SnapStart for Container Images

September 8 2026 Greg Fina Leave a comment

Lambda has been my favorite service for years. When I need to stand up a POC or a pilot, nothing beats it — I can deploy code in minutes, wire it to an event, and pay only for what actually runs. So it always stung a little that the standard architecture review for a real time inference endpoint ended the same way: “Lambda would be perfect for this… except cold starts.” Bursty traffic, event driven triggers, pay per use economics — serverless was the obvious fit on paper, and then someone would pull up a P99 latency chart and the conversation moved to a container platform or a provisioned endpoint instead.

That objection just lost most of its teeth — and the timing could not be better. With frontier AI taking off, the workloads I most want to prototype on Lambda are exactly the ones cold starts punished hardest. With Lambda SnapStart now supporting container image functions, the single biggest reason architects steered AI workloads away from Lambda is largely gone — and the design conversation shifts from “how do we survive cold starts” to “how do we design our init phase to be snapshotted.”

Why cold starts are especially brutal for AI

I learned this the hard way. A cold start is Lambda running your initialization code — loading the function, starting the runtime, and executing everything outside the handler — before it can serve the first request. For the plain CRUD functions I’d been happily shipping for years, that’s tens of milliseconds and nobody notices. The first time I dropped a model into a function, I found out that an AI workload is a different universe.

Three things stack up. First, the container image is large. A model runtime plus its transitive dependency tree — think a framework, a tokenizer library, numeric packages, and a CUDA adjacent stack — routinely produces images in the multi gigabyte range. Second, importing those dependencies is expensive: pulling a large ML framework into memory and resolving its native extensions can burn several seconds on its own, before you’ve touched a model. Third, and worst, you load weights at init. Reading a few hundred megabytes of parameters off disk or out of S3 and deserializing them into memory is the dominant cost, and it happens on every cold start.

Add those together and a “warm” invocation that returns in 80 ms is sitting behind a cold start path that takes five to ten seconds. I’ve watched a demo that flew on my laptop fall apart the moment a few concurrent requests forced Lambda to scale out into fresh environments. For a real time inference or agentic tool call, that is not a tail latency nuisance — it is timeouts, blown SLAs, and a genuinely bad user experience the moment traffic spikes. The cold start tax is levied precisely when you can least afford it.

What SnapStart for container images actually is

SnapStart attacks the problem at the mechanism level rather than asking you to shrink your dependencies. When you publish a function version, Lambda runs your function through the entire Init phase once, then takes a Firecracker microVM snapshot of the memory and disk state of that fully initialized environment, encrypts it, and caches it. On a subsequent cold start, Lambda does not re-run your initialization — it restores the microVM from that snapshot and jumps straight to handling the request.

The strategic point for AI workloads: your model load, your framework imports, your client construction all happen once, at publish time, and get frozen into the snapshot. Every future scale-out event restores from that frozen, ready to serve state instead of paying the multi second init bill again.

SnapStart itself is not new — it has been available for zip packaged functions across Java, Python, and .NET. What changed is that it now works for container image functions, and for me that’s the whole story. Containers are how I actually package these workloads. The 250 MB unzipped limit on zip functions and layers was always a poor fit for the gigabyte scale ML stacks I was building; the 10 GB container image support was the natural home. Until now I had to choose: container packaging or snapshot acceleration. I can finally have both.

How to make it work well

Here’s the mental model I’ve settled on. SnapStart rewards a specific architectural discipline: make the init phase do the expensive work, because init is what gets snapshotted.

Concretely, move model loading, dependency imports, and client/session setup to module scope — outside the handler — so they execute during Init and land in the snapshot:

# Runs at Init -> captured in the snapshot
import torch
from my_runtime import load_model

MODEL = load_model("/opt/ml/model")   # heavy: happens once, at publish
MODEL.eval()

def handler(event, context):
    # Warm path only: no model load here
    return MODEL.infer(event["input"])

The catch is uniqueness, and it’s the one that bit me. When Lambda restores many environments from one snapshot, any state created during init is shared across all of them — seeded random number generators, unique IDs, cached credentials, open connections. Freeze a database connection into the snapshot and you will hand every restored environment a stale, possibly closed socket. Seed a PRNG at init and every environment produces the same “random” sequence. I’ve debugged exactly that, and staring at duplicate “random” IDs across invocations is a humbling afternoon.

The fix is runtime hooks. For container images you implement before-snapshot and after-restore hooks so that the runtime coordinates the lifecycle. Use the before-checkpoint hook to gracefully close connections you don’t want frozen, and the after-restore hook to regenerate anything that must be unique: re-seed entropy, refresh temporary credentials, and re-establish network connections. Note the budget — after-restore work counts against a 10-second restore timeout, so keep it lean.

A few more practical levers:

Right-size memory. Memory scales CPU on Lambda, and restore plus after-restore work is CPU sensitive. For model serving functions, more memory usually pays for itself in lower restore and inference time.
Mind the image and layers. Snapshotting doesn’t excuse a sloppy image. Put model weights and heavy dependencies in stable lower layers, keep your handler code in a thin top layer, and you improve both build hygiene and load behavior.
Verify against real deployments. Restore latency depends on snapshot size, so benchmark P99 with your actual model rather than trusting a hello world number.

When it’s the right call — and when it isn’t

I want to be honest about the boundaries, because I’ve talked myself into using Lambda where I shouldn’t have. SnapStart on containers is a strong fit for bursty, spiky inference: workloads that idle then spike, where you’d otherwise overpay for always on capacity or eat cold starts on every scale-out. It’s excellent for agentic tool calls, where an orchestrator fans out to many short lived, independently scaling functions, and for RAG retrieval endpoints that load an embedding model and query a vector store. These are exactly the prototypes I keep reaching for right now.

It is a weaker fit in two cases. For sustained, high throughput inference that keeps environments warm anyway, provisioned concurrency or a dedicated container platform can still win on steady state cost and predictability — SnapStart’s advantage is amortizing init across intermittent cold starts, which matters less when you rarely have one. And for very large models needing GPU acceleration, Lambda is the wrong tool entirely; those belong on dedicated GPU endpoints such as SageMaker. SnapStart accelerates CPU bound init, not the physics of serving a 70B-parameter model.

The takeaway

Serverless AI just got materially more viable, and my favorite service is back on the table for the workloads I care about most. The reflexive “cold starts kill us” objection — the one that ended so many of my architecture reviews — no longer holds for a large and growing class of inference and agentic workloads. That doesn’t mean the work disappears; it moves. The new discipline is designing your init phase for snapshotting: front load the expensive work, handle uniqueness with runtime hooks, and right-size for restore.

Audit your init phase. Everything expensive — model load, imports, client setup — should run outside the handler so it lands in the snapshot.
Handle uniqueness explicitly. Use before-snapshot and after-restore hooks to close/reopen connections, refresh credentials, and re-seed entropy.
Benchmark P99 with your real model, not a toy function, and right-size memory to the restore path.
Pick the fit deliberately — bursty inference and agentic tool calls, yes; sustained high throughput or GPU bound serving, look elsewhere.
Revisit dismissed designs. Endpoints you ruled out over cold starts deserve a second look.

The objection didn’t just weaken — it changed shape. I’m already reopening a couple of “we can’t use Lambda for that” calls I made last year. Which of yours are you ready to revisit?

GenAI,AWS,Cloud,Architecture

Open Weight AI Models vs. Frontier APIs — The 2026 Cost Performance Tipping Point

September 3 2026 Greg Fina Leave a comment

Your AI inference bill is probably 10× higher than it needs to be. And the gap is getting wider, not narrower.

Six months ago, you could justify paying frontier API prices because open weight models were measurably worse. That justification is evaporating. In mid 2026, models like Kimi K3, GLM 5.2, and Llama 4 Maverick are matching or beating frontier APIs on real engineering benchmarks while costing a fraction per token. The question is no longer “are open weight models good enough?” It’s “can you still justify the premium?”

The Numbers Have Changed

Let’s lay out the current pricing landscape. On the frontier API side:

Model	Input / 1M tokens	Output / 1M tokens
GPT 5.6 Sol	$5.00	$30.00
Claude Opus 5	$5.00	$25.00
GPT 5.6 Terra	$2.00	$12.00
GPT 5.6 Luna	$0.20	$1.20

And on the open weight side:

Model	Input / 1M tokens	Output / 1M tokens	License
Kimi K3 (2.8T / 104B active)	$3.00	$15.00	Open weight
GLM 5.2 (744B / 40B active)	$1.40	$4.40	MIT
DeepSeek V4	$0.435	~$0.87	Open weight

DeepSeek V4 at $0.435 per million input tokens is roughly 35× cheaper than GPT 5.6 Sol. Even Kimi K3, which sits at the premium end of open weight pricing, is half the cost of the flagship frontier APIs on output tokens.

But pricing is only half the story. What matters is what you get for the money.

Benchmarks Tell an Uncomfortable Story for Frontier Labs

Kimi K3, released by Moonshot AI in July 2026, is a 2.8 trillion parameter mixture of experts model with 104 billion active parameters and a 1 million token context window. On Artificial Analysis’ 16 task benchmark, it scored 90.49 out of 100, beating every Claude and GPT model tested. Its cost per completed task came in at roughly $0.94, compared to Claude Opus 4.8’s $1.80. That’s near frontier quality at half the cost per task.

GLM 5.2 from Z.ai (Zhipu AI), a 744 billion parameter MoE with 40 billion active, beat GPT 5.5 on SWE bench Pro (62.1 vs 58.6) at approximately one sixth the per token cost. It ships under the MIT license with no regional restrictions, meaning you can self host it anywhere.

Faros AI ran 211 real engineering tasks through seven different model plus harness combinations. The result: Claude Code paired with GLM 5.2 landed in the top quality band alongside Claude Code paired with Kimi K2.6, while Claude Code with Opus 4.8 and Codex with GPT 5.5 did not buy their way into that top tier. The open weight route scored 0.568; the Opus route scored 0.521. Higher quality and lower cost.

The Sentient Arena competition put a finer point on it. 147 builders competed using the open source MiniMax M2.5 model, and the top teams averaged approximately 70% accuracy at $1.74 per run. The same agents running on Claude Opus 4.5 hit approximately 80% accuracy at $56.53 per run. When you factor cost into the score, the open source model won for every team in the top six. Frontier closed source still won on absolute accuracy. Open source won on accuracy per dollar by a factor of 30.

Where Frontier Still Wins (For Now)

Let’s be honest about the limitations. Open weight models are roughly four months behind the closed frontier on absolute quality, according to analysis from The New Stack. On the hardest long horizon reasoning tasks, multi step autonomous agents, and problems requiring peak intelligence, GPT 5.6 Sol and Claude Opus 5 still hold an edge.

There is also the structure problem. Research from Unsupervised found that adding structured output requirements (JSON schemas, strict formatting) nearly tripled frontier model cost per task but actually cut cost for open weight models. If your pipeline demands rigid structure from a frontier API, you’re paying even more than the sticker price suggests.

The convenience gap is real too. One API call to a managed endpoint is simpler than provisioning GPU infrastructure. For a team running a handful of inference calls per day, the operational overhead of self hosting may not justify the savings. But that calculus changes fast at scale.

The Fine Tuning Equation

Here is where the economics become decisive. Fine tuned open weight models show 15 to 25% improvement in task specific accuracy over base models. For domain specific work (legal, medical, code generation against your specific codebase), a fine tuned Llama 4 or GLM 5.2 will outperform a general purpose frontier API on your tasks, every time.

The timing matters because OpenAI is sunsetting self serve fine tuning on a published timeline through January 2027. Organizations that never ran a fine tuning job already lost the ability to start one in May 2026. By January 2027, the door closes entirely for new jobs. The stated reason: newer base models are good enough that prompting beats fine tuning for most use cases. The practical effect: if you need fine tuned models, open weight is becoming the only game in town.

The GPU rental math makes this even more compelling. A 70B QLoRA fine tuning job on a rented H100 runs about $20 in compute. The equivalent job through a managed API platform costs $148 to $154. That is a 7× difference on raw compute. At scale, running 10 concurrent fine tuning jobs for enterprise customers, the rental approach is 73 to 91% cheaper than managed platforms.

The Scaling Curve Is the Real Story

Proprietary API costs scale linearly. Double your volume, double your bill. Self hosted inference scales at marginal cost: once you have the GPU capacity provisioned, additional inference is nearly free up to saturation.

For a team processing 100 million tokens per month, the TL;DR Dev Tech scorecard lays it out starkly:

Proprietary API: $15,000 to $50,000 per month per application
Self hosted open weight: $2,000 to $8,000 per month in GPU rental and ops

AWS CTO Werner Vogels has publicly noted that companies are migrating inference workloads from API gated models to open weight alternatives. When the CTO of the world’s largest cloud provider tells you open source is cheaper, the signal is hard to ignore.

And roughly 80% of enterprise AI tasks work well with open models in the 7B to 70B parameter range. You don’t need a 2.8 trillion parameter model for document summarization, structured extraction, or routing classification. A properly fine tuned 70B model handles these workloads at a tiny fraction of frontier cost.

The Decision Framework

Here is how to think about this if you are making infrastructure decisions today:

Audit your workload mix. Categorize your AI tasks by complexity. For most teams, 80% or more of tasks are “good enough” territory for open weight models. Route only the genuinely hard problems to frontier APIs.
Run your own benchmarks. Public leaderboards set priors, but Faros proved that the best model on a benchmark is not always the best model for your codebase. Test on your actual tasks, not synthetic ones.
Factor in fine tuning. If you are paying frontier API prices for domain specific work, a fine tuned open weight model will likely outperform it at 5 to 20× lower cost. The OpenAI fine tuning sunset makes this transition urgent, not optional.
Model the scaling curve. If your inference volume is growing (and whose isn’t), the linear scaling of API costs versus the marginal cost scaling of self hosted inference will dominate your total cost of ownership within months.
Watch the vendor lock in risk. As CNCF executive director Jonathan Bryce put it: paying 10× more for a four month capability lead is not an enterprise AI strategy. It is an expensive form of lock in.

What Comes Next

Meta retired its hosted Llama API in July 2026, pivoting to a Muse only distribution model, while simultaneously releasing Muse Glimmer (30B, Apache licensed) in August. That hybrid strategy signals where the market is headed: weights are open, but the distribution and hosting layer is where value gets captured.

The open weight ecosystem is not slowing down. Capital is flooding in. The tooling around self hosted inference (vLLM, SGLang, Ollama) is maturing rapidly. And every month, the quality gap with frontier APIs narrows while the cost gap widens.

The tipping point is not coming. For most workloads, it has already arrived. The question is whether your architecture reflects that reality or is still paying a 2024 tax on 2026 problems.

Your AI inference bill is probably 10× higher than it needs to be. And the gap is getting wider, not narrower.

Six months ago, you could justify paying frontier API prices because open weight models were measurably worse. That justification is evaporating. In mid 2026, models like Kimi K3, GLM...

DevOps,Cloud,Architecture,Security

AWS Continuum — When Your Security Tool Talks to Your AI Coding Agent

September 2 2026 Greg Fina Leave a comment

Over 60 percent of production code at Fortune 500 companies now contains blocks authored by an AI coding agent. That is not a projection — it is an industry estimate for 2026. The code works. It compiles, passes tests, ships. It also leaks credentials, trusts user input it should not, and pulls phantom dependencies with startling regularity. Your SAST scanner finds some of this — three days later, after the PR merged and the pattern propagated across four services.

AWS Continuum for code vulnerabilities is built to kill that delay. With its August 2026 announcement extending integrations into Claude Code, OpenAI Codex, and Kiro, the security feedback loop moved from “scan after commit” to “secure while writing.”

What Continuum Actually Does

Strip away the marketing and Continuum is an agent team loop — a harness that orchestrates multiple models, each selected for the task at hand, connected to your environment context. Four stages:

Discovery. Continuum scans your code for vulnerabilities. Frontier models can now trace multi step attack paths that would take a human team weeks. Detection is no longer the bottleneck.
Prioritization. Continuum reads your account configurations, IAM policies, network topology, and exposure surfaces before ranking a finding. A SQL injection in code that never reaches production ranks below one sitting on a public endpoint. Context kills noise.
Validation. Continuum builds a working exploit in a sandbox. If it cannot actually weaponize the finding, the finding drops in priority. This is how it culls false positives — and false positives are what make security teams ignore scanner output.
Remediation. It generates a fix, validated in the same sandbox, and returns it to the developer or coding agent. Not a Jira ticket pointing to a CWE page. An actual code patch. As Chet Kapoor put it: the harness is infrastructure, treated with the same rigor AWS applies to identity and policy enforcement.

The Claude Code / Codex / Kiro Integration: Why It Matters

Before August, Continuum operated on deployed code. Useful, but reactive. The Anthropic and OpenAI partnerships change the geometry.

Here is how it works. You prompt Claude Code (or Codex, or Kiro) for a function. The agent generates candidate code. Before you see it, the Agent Security Runtime (ASR) — a lightweight process inside the agent’s execution environment — intercepts the output and evaluates it against Continuum’s policy engine. The ASR returns one of four verdicts: ALLOW, ALLOW_WITH_WARNING, BLOCK_AND_REGENERATE, or BLOCK_AND_ESCALATE. On a block, the agent regenerates with remediation constraints. The developer sees the secure version first.

The latency tax is negligible. AWS reports a p99 overhead of 180 milliseconds — imperceptible when AI code generation itself takes one to three seconds for a medium complexity function. The evaluation loop runs up to three attempts before escalating, so the agent gets multiple chances to self correct before bothering a human.

Critically, both Anthropic and OpenAI confirmed the integration sits at the agent runtime layer, not as a post processing filter. Continuum sees and influences code before the developer does. That required each company to expose internal APIs to the Continuum SDK that third party developers cannot access. As Rivian CISO Mike Johnson noted: “This shortens what really matters: timeline to fix serious vulnerabilities.”

Why AI Generated Code Breaks Your Existing Security Stack

Three failure modes make traditional scanners insufficient for AI authored code:

Insecure defaults at scale. Ask a model to “write a function that authenticates users” and it will. It will not add rate limiting, constant time comparison, or JWT secret rotation unless you ask. Multiply that across thousands of functions and you get a codebase where security hardening is systematically absent. SAST catches individual patterns. It does not catch the organizational trend.

Library hallucination. AI agents sometimes suggest packages that do not exist in any registry. Attackers register these hallucinated names and publish malicious versions. At least 47 confirmed dependency confusion via hallucination incidents occurred in 2025, including two that led to production ransomware. Continuum’s ASR verifies every suggested dependency against your private registry, public registries, and a known malicious blocklist in real time.

Deprecated API patterns. Models trained on pre-2024 data still suggest hashlib.md5() for password hashing. It compiles. It runs. It is cryptographically catastrophic. Continuum maintains a Deprecated Security Patterns library covering over 4,800 API patterns across Python, JavaScript, Java, Go, C#, Ruby, and Rust, updated weekly and auto pushed to every active ASR instance.

What This Means for DevSecOps Teams

If your security architecture looks like “developer writes code → CI runs SAST/SCA → security triages findings → developer fixes three weeks later,” Continuum collapses that into a single step. The code suggestion is the remediation. AWS’s research found that developers who receive a secure suggestion as their first output are 84 percent more likely to use it as is, versus developers who get a standard suggestion followed by a separate alert.

That is not a workflow optimization. It is a behavioral change. Security teams have spent years trying to “shift left.” The reality has been shifting alerts left, not shifting secure defaults left. Continuum pushes security into the generative moment — before commit, before review, before the developer even reads the output.

For teams already running Continuum on existing code, the integration creates two modes with one outcome:

Existing code: Continuum discovers, prioritizes, validates, and remediates across your deployed environment.
Greenfield code: The Continuum plugin inside Codex, Claude Code, or Kiro delivers security validated suggestions in the development environment.

Both feed into Security Hub Extended — a dashboard that aggregates AI generated code findings across every developer, maps them to OWASP Top 10 and CWE identifiers, and pushes into your existing SIEM and ticketing workflows. One pane of glass, whether the code is legacy or was generated five seconds ago.

Practical Takeaways

Request preview access now. Continuum is in gated preview. The Claude Code, Codex, and Kiro integrations are “coming soon.” Get in the queue — 1,200 enterprise accounts activated within 48 hours of the August announcement.
Start with Context Profiles. Continuum lets you declare security posture, data classification, and trust boundaries per repository. A PCI scoped service gets stricter policy evaluation than an internal admin tool. Define these before turning on the ASR.
Audit your dependency allow list. Continuum’s package verification is only as good as your organizational package inventory. If you do not have one, build it now. If you do, check it against what your AI agents have actually been suggesting.
Instrument the feedback loop. Track ALLOW versus BLOCK_AND_REGENERATE ratios per team and per agent. Rising block rates on a specific agent or codebase tell you something about prompt quality, project complexity, or both. This is telemetry you have never had before.
Do not rip out your pipeline scanners yet. Continuum addresses the generative layer. You still need SAST, SCA, and DAST for human authored code and runtime behavior. Layered defense, not replacement.

Looking Forward

CISA’s recent Guidance on AI-Assisted Software Development Security recommends real time security interception at the generation layer. Their research found that 34 percent of AI generated code passing all CI/CD checks still contained at least one exploitable vulnerability. That number should keep every security leader awake.

Continuum is the first production grade answer to that problem. It is not perfect — gated preview means rough edges, and the “coming soon” on agent integrations means your team cannot wire it up today. But the architectural bet is right: security has to live where the code is born, and in 2026, code is born inside AI agents. The sooner your security toolchain understands that, the better.