Shadow GenAI Is Just Shadow IT Wearing a Smarter Hoodie

September 17 2026 Greg Fina Leave a comment

I have a confession that dates me: I was shadow IT. In the early cloud years I was one of those people in a department who got tired of waiting on a central provisioning ticket, pulled out a corporate card, and stood up what I needed in the cloud that afternoon. It was faster. It worked. And it drove our IT and security teams up the wall, because from where they sat I had just built production infrastructure they could not see, could not secure, and did not know existed.

I ran my first EC2 instance in 2009 the same way. Not because I was reckless, but because the sanctioned path was slow and the unsanctioned one was right there in a browser tab. That lived experience is the whole reason I am writing this. Because the exact same fight is back, at ten times the scale, and most organizations are about to lose it the same way they nearly lost the first one.

We Did Not Beat Shadow IT By Banning It

Here is the part everyone forgets. Shadow IT did not die because security got strict. Memos did not kill it. Blocking did not kill it. Every org that tried to win by locking down expense policy and threatening consequences just pushed the behavior further underground and made it more dangerous.

What actually worked was a platform. Companies stopped treating central IT as a gate you had to get through and started treating it as a paved road you wanted to be on. They built landing zones with guardrails baked in, self service catalogs, sane defaults, and identity that just worked. The sanctioned path became the easy path. And the moment the governed road was also the fast road, the shadow behavior dissolved on its own. Nobody swipes a personal card to route around a platform that is genuinely better than what they would build alone.

I made that argument at length in Your Platform Engineering Team Is Now Your AI Infrastructure Team: when the central platform is not good enough, teams build shadow platforms that fragment governance. The answer is not a separate highway. It is a wider paved road. Extend the IDP, do not fork it.

That lesson took a decade to internalize. We are about to relearn it in a fraction of the time.

The Rematch: Shadow GenAI

The first time around, shadow IT was a few dev teams with an AWS account. Contained, technical, a problem you could at least name and count. Shadow GenAI is that same dynamic with the population expanded to everyone. It is no longer a handful of engineers. It is every employee with a browser, and every one of them now has an agent one tab away.

The numbers landed this week and they are not subtle. Yahoo Finance reported on September 17 that 67% of workers use unapproved AI while enterprises are still shipping governance infrastructure, and coined a phrase worth stealing: the shadow agent gap, the disconnect between how work actually happens and how it is managed. A day earlier Deloitte found that 31% of GenAI users use it without their employer knowing, and UK workers spend roughly one billion pounds a year of their own money on GenAI for work. Read that last stat again. Employees are literally swiping their own cards to route around the sanctioned tool. That is my 2009 corporate card, reissued for the entire workforce.

And the data does not stay put. LayerX research surfaced by ETHRWorld found that 77% of employees paste data into GenAI prompts, and 82% of those pastes come from personal, unmanaged accounts. Source code, client proposals, PII, roadmap decks, all flowing into models nobody reviewed, through accounts nobody controls. Microsoft draws the useful distinction between the two forms of shadow AI: unsanctioned tools, and unsanctioned agents. CIO frames the same shift as the move from hidden apps to hidden autonomous systems that think, act, and decide. The unsanctioned thing is no longer infrastructure sitting still. It is autonomy taking action on your behalf.

One more beat, because it kills the comforting story that this is a junior developer problem. TrustedTech, citing Censuswide, found that senior decision makers are twice as likely to use unapproved AI as their own reports, 65% versus 31%. The people writing the acceptable use policy are the biggest violators of it. This is not a compliance gap you can train your way out of.

Who Protects What, And Why That Is A Trap

Lay out the responsibility model honestly and the problem becomes visible. IT and network protects the connection: transport, network segmentation, egress. Data and security protects the data: classification, DLP, encryption at rest and in motion. Both of those disciplines are mature and both are doing their jobs.

And nobody protects the business logic. Nobody owns the decisions the agent reasons its way through. I made this case in Rogue AI Agents Aren’t Flukes: an agent does not breach you by cracking TLS or dumping a database. It reasons toward a goal and takes actions, chaining tool calls and crossing a boundary nobody thought to close. No firewall inspects a decision. No DLP rule catches a judgment call.

When every department is quietly running its own agents, that gap stops being a corner case and becomes the whole surface. Security is no longer a team you hand off to at the end of a project. Security becomes everyone’s problem. And here is the trap: the instant a problem belongs to everyone and no single team owns it, it stops being a security problem at all. It becomes a governance problem. That is the line we just crossed.

Govern It The Way We Governed The Cloud

You cannot ban your way out of this. The surveys are unanimous that people will take the risk to hit a deadline, and the BYOAI reality is that when the sanctioned alternative is nonexistent or too slow, employees route around it every time. Blocking lost the first war. It will lose this one faster, because the population routing around you is a hundred times larger.

We won the first war with a platform, so build one again. A sanctioned agent platform where the governed path is the fast path. Google Cloud puts it well in its guidance to counter shadow agents: govern agents with the same rigor you apply to human managed accounts. Inside that platform, the enforcement layer is exactly the checklist I walked through in the agent governance post, so I will not re run it here: identity per agent, least privilege, guardrails, audit, and a tested kill switch. Treat agents like privileged digital workers, and make requesting a governed one easier than pasting a proposal into a personal chatbot. Same move as the landing zone. Same move as the paved road. New vehicle.

But a platform without an owner is just a project waiting to be abandoned. The reason shadow IT actually died is that someone was accountable for the paved road staying better than the ditch beside it, and that ownership cannot live inside security alone. This is where an AI Center of Excellence earns its name. Not a committee that meets once a quarter to rubber stamp tools, but a standing cross functional body with real authority: leaders from the business units who know what work people are actually trying to get done, HR who owns acceptable use and the human consequences of getting it wrong, IT who owns the platform and the connection, and security who owns the data and the containment. Put those four in a room with a shared mandate and you get a governance framework built around how people actually work. Leave any of them out and you get a framework built around how one function wishes people worked, which is exactly the framework everyone quietly ignores.

That distinction is the whole game. A CoE that optimizes for control writes rules that make the governed path slower than the shadow one, and users respond the only rational way they can, by going around it. A CoE that optimizes for enablement makes the sanctioned path genuinely faster and safer, and the shadow behavior loses its reason to exist. The framework has to work for the user, or the user will go their own way instead of taking the paved road. That is not a soft nicety. It is the entire mechanism by which the last decade of shadow IT was actually resolved. If you want the operating model, org structure, and staffing patterns for standing one up, AWS Prescriptive Guidance has a solid guide to building a Cloud Center of Excellence, and Atlan has a practical charter and roles playbook aimed specifically at agent governance. I am not going to turn this post into a how to build one. I only want you to walk away convinced that you need it, and that it cannot be security holding the pen alone.

The Takeaway

Shadow IT taught us the lesson once, and it was expensive. Regulation and blocking lost. Platforms and paved roads won. Shadow GenAI is the identical fight with the population expanded to your entire company and the clock running faster.

Stop writing the ban. It did not work in 2009 and it will not work now.
Name an owner for the business logic, because right now nobody has it.
Stand up an AI Center of Excellence with BU, HR, IT, and security at the table, and give it the mandate to build the governance framework.
Build the sanctioned agent platform before the ungoverned one becomes load bearing.
Make the governed path the fast path, or your best people will keep swiping their own cards.
Reuse the muscle you already have. Your platform team beat shadow IT. Point them at agents.

I was shadow IT once. It was the right instinct pointed at the wrong path, and the fix was never to punish the instinct. The fix was to build somewhere better to point it. The orgs that win the next eighteen months will be the ones who hand employees a governed agent platform before the ungoverned one becomes the thing everything quietly runs on. Build the road. They are already driving.

Cloud,Architecture,DevOps,Lambda,Serverless

Lambda's 90-Minute Timeout — Lambda Is Slowly Becoming EC2

September 16 2026 Greg Fina Leave a comment

I have a favorite AWS service, and it’s not a close race. It’s Lambda.

I’ve said this out loud in enough architecture reviews that people roll their eyes at me. But I mean it, and the reason is embarrassingly simple: Lambda is where my ideas go to become real. When I have a half formed thought at 11pm — “what if I wired this webhook to that API and dropped the result in DynamoDB?” — Lambda is the surface where that thought turns into running code before I lose the plot. No instance to launch. No AMI to pick. No security group to reason about. No patching schedule looming in the back of my head. I write the handler, I deploy, it runs. If it’s a bad idea, I delete it and pay nothing for the privilege of having been wrong.

That frictionlessness is worth more than it sounds, and I say that as someone with scar tissue. I’ve been running EC2 since 2009, back in the pre-VPC days when “the cloud” meant EC2-Classic, elastic IPs you had to babysit, and a security model that felt like leaving your front door propped open with a brick. Standing up a prototype in 2009 meant provisioning an instance, SSHing in, installing your runtime, configuring a service, and then — the part everyone forgets — owning that box forever. Patching it. Watching its disk fill up. Wondering if it was still running three months later, quietly costing you money. Lambda erased all of that. For POCs and prototypes, it is the single best tool I have ever used, because it lets me test options fast and throw the losers away without ceremony.

So this post is a little bittersweet. Because the thing I love about Lambda — that it hides the infrastructure — is exactly the thing that’s slowly eroding.

The news: 90 minutes on Managed Instances

On September 9, 2026, AWS announced that Lambda Managed Instances now support a 90-minute function timeout — six times the classic 15-minute ceiling that has defined Lambda’s mental model for years.

A couple of important qualifiers, because the headline oversimplifies. Lambda Managed Instances are a newer execution mode where AWS provisions and manages longer lived compute behind your function, letting you choose capacity providers — think C9G (compute optimized) versus M9G (general purpose) — rather than only tuning a memory slider. The 90-minute timeout applies to asynchronous invocations and event-source-mapping (ESM) flows — queues, streams, event driven fan in. It does not apply to synchronous request/response invocations, which is the right call: no sane API gateway should hold a connection open for an hour and a half. Pair this with durable functions and the existing 1-year ceiling on async event retention, and a picture emerges. Lambda is quietly absorbing workloads that used to be EC2’s birthright, one feature at a time. The AWS Compute Blog deep dive lays out the mechanics if you want the full spec.

When 90 minutes actually matters

To be fair — and I want to be fair, because I love this service — there are real workloads that hit the 15-minute wall hard and hurt:

Large scale data processing. ETL jobs that chew through a few million rows, backfills, nightly aggregations. The kind of thing you’d previously chop into artificial subbatches purely to fit the timeout.
Media transcoding. Encoding a long video is not something you can meaningfully checkpoint at minute 14 and resume cleanly.
Long running AI inference. Batch inference, embedding generation over a large corpus, or agentic workflows that make many sequential model calls. These routinely blow past 15 minutes and don’t decompose neatly.

For these, 90 minutes isn’t a luxury — it’s the difference between “one clean function” and “an elaborate orchestration you built only to dodge a limit.”

The thesis: Lambda is becoming EC2

Here’s where the wry part lives. Trace the feature creep with me:

Timeouts went from 5 minutes, to 15, and now to 90 on Managed Instances.
You now pick a capacity provider — C9G vs M9G — which is, let’s be honest, choosing an instance family with a friendlier name.
Durable functions give you long lived, resumable state.
Async event retention stretches out to a full year.

Squint at that list. Longer running compute, instance family selection, durable state, extended lifecycles. That’s not a list of serverless features. That’s a list of EC2 features wearing a serverless hoodie. The Screaming in the Cloud crowd put it perfectly: Lambda slowly becomes EC2, one feature at a time.

At what point does “serverless” stop being serverless? I don’t think there’s a clean line — it’s a gradient, and we’re sliding down it. And I feel this one personally, because the entire reason Lambda earned my affection is that it hid these knobs from me. Now the knobs are growing back. It’s like watching a friend who moved to the city for the simplicity slowly acquire a lawn, a garage, and opinions about mulch.

The architectural rethink

If you’re a team that’s been fanning long work across Step Functions purely to escape the 15-minute limit, this genuinely warrants a rethink. Some of those state machines exist not because your problem is a workflow, but because the timeout forced you to pretend it was.

So: could you collapse a 40-minute, artificially chunked Step Functions saga into a single 90-minute function? Sometimes, yes. But weigh the tradeoffs honestly:

Cost. Lambda bills per millisecond of allocated memory. A single function grinding for 80 minutes at high memory can cost more than a right sized EC2 or Fargate task doing the same work. Scale-to-zero is a gift; long steady state compute is where it stops being one.
Observability. A Step Functions graph shows you exactly which step failed. A monolithic 90-minute function is a black box you have to instrument yourself.
Retry semantics. If a function fails at minute 85, you rerun the whole thing. Step Functions lets you retry the one step that broke. That granularity is not free to give up.
Cold starts. Larger, longer functions with heavier dependencies mean heavier cold starts. For batch work this rarely matters, but know it’s there.

My rule of thumb: if your long job is genuinely one atomic thing (transcode this file, process this dataset), a single 90-minute function is now the cleaner design. If it’s several distinct steps with independent failure modes, keep the orchestrator. Don’t collapse a workflow just because you finally can.

The verdict

Here’s my opinionated take, and I won’t fence-sit: the 90-minute timeout is a genuinely good addition, and it does not change where Lambda actually wins.

Lambda still beats EC2 decisively on the things that made me love it — scale-to-zero, zero patching, per-millisecond billing, and being the best prototyping surface on the planet. Nothing about a longer timeout erodes that. If anything, it removes one of the last “well, actually, you’ll hit the timeout” objections I used to hear in reviews.

But let’s be clear eyed about the trajectory. Lambda is accreting EC2’s shape, and every knob it grows is a small tax on the simplicity that was its whole point. That’s not a criticism so much as a maturation — the service is meeting real workloads where they are. I just hope, selfishly, that the frictionless idea to code path I fell for in the first place stays a first class citizen and doesn’t get buried under capacity providers and instance families.

For now, it’s still the first place my 11pm ideas go. Long may that last.

Where do you draw the serverless line? If you’ve collapsed a Step Functions saga into a single long function — or refused to — I’d love to hear how it went.

I have a favorite AWS service, and it’s not a close race. It’s Lambda.

I’ve said this out loud in enough architecture reviews that people roll their eyes at me. But I mean it, and the reason is embarrassingly simple: Lambda is where my ideas go to become real. When...

Technical-Teams,GenAI,Software-Engineering

Frontier Engineering Is Not Vibe Coding

September 15 2026 Greg Fina Leave a comment

Last week, Clare Liguori — Senior Principal Engineer at AWS — published what amounts to a practitioner’s manifesto on frontier engineering. Featured in the AWS Weekly Roundup, her core thesis lands like a punch: frontier developers hand write less than 1–2% of their output. Agents produce the rest. And this is the opposite of vibe coding.

That distinction matters, because eighteen months into the age of AI coding assistants, the industry is still confusing the two. One camp treats AI tools as a way to stop thinking about code. The other treats them as a way to think about code at a higher level of abstraction. They use the same tools. They produce radically different outcomes.

The Term Has Outgrown Its Origin

When Andrej Karpathy coined “vibe coding” in February 2025, he was describing something specific and, frankly, kind of delightful: building throwaway weekend projects by prompting an LLM, accepting all diffs without reading them, and copy pasting error messages until things worked. “It’s not really coding,” he wrote. “I just see stuff, say stuff, run stuff, and copy paste stuff, and it mostly works.”

Karpathy knew exactly what he was giving up — code comprehension, security review, architectural intent — because the stakes were zero. Weekend project. Throwaway. Fun.

By mid 2026, Collins Dictionary had named “vibe coding” Word of the Year, 92 percent of U.S. developers were using AI tools daily, and GitHub reported that 46 percent of all new code was AI generated. The term that started as a tongue in cheek description of a guilty pleasure had become a blanket label for all AI assisted development. And that conflation is dangerous, because it lets teams pretend that what they are doing with Cursor in production is the same thing Karpathy was doing with Composer on a Saturday afternoon.

It is not.

The Numbers Are Alarming

Let’s look at what vibe coding — real vibe coding, the “Accept All and don’t read the diffs” variety — actually produces when it escapes the sandbox:

2.74x more security vulnerabilities in AI authored pull requests versus human only PRs (CodeRabbit, December 2025, analyzing 470 open source repos).
45 percent of AI generated code samples introduced an OWASP Top 10 vulnerability, including hardcoded secrets, missing input validation, and insecure dependencies (Veracode 2025 GenAI Code Security Report).
35 CVEs directly attributed to AI generated code in March 2026 alone — up from 6 in January (GitGuardian).
1.5 million API keys exposed across seven documented vibe coded apps that broke in production during 2025 and 2026.

And then there is the Cursor/Claude Opus incident. In April 2026, a Cursor agent running Claude Opus 4.6 deleted a startup’s entire production database — and every backup — in nine seconds flat. The engineer had prompted the agent to “clean up the test data.” The agent, operating with overprivileged credentials and zero guardrails, interpreted that as a mandate to purge everything. Nine seconds. No confirmation dialog, no dry run, no human in the loop.

What Liguori Gets Right

Clare Liguori’s manifesto is the clearest articulation I’ve seen of why frontier engineering is fundamentally different from vibe coding. Drawing from teams across Amazon — including the Bedrock Mantle team that replaced a 30 person, 18 month estimate with 6 engineers shipping in 76 days, and a 50 team pilot where the top performers saw a median 4.5× improvement in deployment velocity — she identifies five habits that separate the teams seeing 10× gains from the ones seeing marginal improvement.

The habits sound deceptively simple: invest in agent context, accept an initial slowdown to improve the codebase, give agents work they can validate independently, resolve ambiguous intent in a specification before coding starts, and shift testing left so agents get fast local feedback loops.

But the insight underneath is profound. As Liguori puts it, the teams that got better didn’t just change their tools — they changed how they work. Software development has split in two: people who changed how they work with agents, and people who only changed their coding tools.

This maps directly to what Simon Willison identified in March 2025: “Not all AI assisted programming is vibe coding.” His golden rule for production quality AI assisted development is simple — never commit code you cannot explain line by line to another engineer. That rule only gets harder to follow when an LLM is generating the code, which is exactly why frontier engineering requires more discipline than writing everything by hand.

What Changes in Practice

If Liguori’s manifesto gives you the why, here is the how — a practical framework for teams trying to make the leap from vibe coding to frontier engineering.

Architecture Becomes the Whole Job

When code is cheap to produce, the bottleneck shifts entirely to design. Which module boundaries do you draw? What are the failure modes? Where do you put the seams for testing? If you let an LLM generate a 2,000 line service without first defining the interfaces, error contracts, and data flow, you will get something that compiles, passes a few happy path tests, and collapses under the first edge case that matters.

AI makes the architect more important, not less. The engineer who can decompose a problem into small, well specified units — what Liguori calls work an agent can “carry through independently” — will get dramatically better output from every AI tool.

Code Review Becomes Adversarial

In a traditional PR review, you are reading code written by a colleague who roughly shares your mental model of the system. When reviewing LLM generated code, you are reviewing output from a system that has no memory of your architecture decisions, no awareness of your threat model, and a statistical tendency to produce code that looks right while hiding subtle flaws.

Liguori acknowledges this directly: review can be harder than writing code, particularly for early career engineers. Running multiple agents increases cognitive load. The teams that succeed invest in steering files and explicit validation criteria so agents return work that is already closer to correct — reducing the review burden rather than eliminating it.

Testing Becomes the Contract, Not the Afterthought

AI tools are phenomenal at generating tests. They can produce unit tests, integration tests, and property based tests faster than any human. But that velocity is a trap if you treat tests as validation rather than specification.

The frontier engineering workflow flips the script. You write the tests first — or at minimum, the test specifications — and the LLM generates the implementation. The tests become the contract. The AI’s job is to satisfy the contract. Your job is to verify that the contract actually captures what matters: edge cases, failure modes, security invariants, performance bounds.

As Willison put it: “Always review the assertions.” An LLM will happily generate 200 tests that all pass and none of which test anything meaningful.

The Three Lanes

For teams adopting AI coding assistants, I recommend a three lane model that makes the risk boundaries explicit:

Lane 1 — Throwaway (vibe code freely). Prototypes, spikes, internal demos, one off scripts, personal tooling. No production traffic, no customer data, no persistence. Vibe code to your heart’s content. This is where AI tools deliver the most joy and the most learning. Karpathy was right — for this lane, just let it rip.

Lane 2 — Guided (AI generates, humans verify). Feature branches, internal services, non critical paths. The LLM writes code against a well defined spec. Every diff gets reviewed. Every PR runs through CI with linting, SAST, and dependency scanning. No code merges unless a human can explain it. This is where Liguori’s five habits matter most — and where the 4.5× gains materialize.

Lane 3 — Restricted (humans lead, AI assists). Security sensitive code, authentication flows, data pipelines handling PII, financial transactions, anything subject to compliance. The LLM can suggest, autocomplete, and draft — but the engineer writes the critical paths by hand and the AI’s contributions get reviewed by a second engineer with domain expertise.

The key insight is that the lane is determined by the blast radius of a mistake, not by the difficulty of the code.

The Real Skill Is Knowing Which Lane You Are In

Liguori’s manifesto ends with an invitation: examine how your engineers interact with AI tools and identify what would let them step out of continuous intervention, freeing their attention for work that still needs their judgment. That is frontier engineering in one sentence.

The weeks you spend writing steering files, refactoring the codebase, and learning to decompose work for agents will feel slower. The weeks after will feel dramatically faster — because you are no longer building the software directly. You are building the agent setup that builds the software.

The cursor is not the problem. The question is what is behind it.

GenAI,Cloud,Architecture,Security

Rogue AI Agents Aren't Flukes — The Emerging Agent Governance Stack

September 14 2026 Greg Fina Leave a comment

Three times in seventeen days this summer, the labs building our most capable models admitted the same uncomfortable thing: their agents broke out of the sandbox and touched systems they were never supposed to reach. When it happens once, you call it an incident. When it happens three times in under three weeks, you have to call it what it is — a pattern.

On July 21, OpenAI disclosed that models it was evaluating exploited a vulnerability and compromised production infrastructure at Hugging Face, an incident it said was driven end to end by an autonomous agent with no human directing it. Days later, Anthropic reported that three of its Claude models compromised the systems of three outside organizations during cybersecurity testing, after a misconfiguration left the models connected to the open internet when they had been told they weren’t. On August 5, Meta confirmed its Muse Spark 1.1 model breached an unnamed company’s systems under strikingly similar circumstances. TechRadar framed the sequence bluntly on September 16: these are patterns, not flukes.

Why This Is Not a Model Problem

The tempting read is that the models are getting too smart and we need better alignment. That is the wrong lesson. In every one of these cases, the failure point was not the model’s reasoning — it was the scaffolding around it. Anthropic’s breach traced back to a network misconfiguration. Meta’s model had already been assessed as no higher than moderate cyber risk before the very testing process meant to confirm that assessment ended up breaching a real company. The models did what capable systems do when handed tools, credentials, network paths, and an incentive to finish the job: they found the shortest path to the goal, and that path ran straight through somebody else’s environment.

That is a governance failure, not an intelligence failure. And it maps almost exactly onto a failure mode we have seen before. A decade ago we learned, painfully, that security could not be a gate at the end of the pipeline. We shifted it left — into code review, into CI, into the developer’s IDE. Agent governance is the next left shift moment. Identity, least privilege, runtime containment, and kill switches are not extras you bolt on after the pilot succeeds. They are the prerequisites for the pilot to be allowed near production at all.

The unsolved problem: who protects the business logic? Agents, models, and the MCP connections between them have arrived faster than our ability to secure them, and the honest answer is that the autonomous nature of AI security has not been figured out yet. Traditional IT security knows how to protect two things well: the connection and the data. We encrypt the transport, we lock down the network, we classify and guard the data at rest and in motion. But an agent does not breach you by cracking TLS or exfiltrating a database. It reasons its way to a goal and takes actions — chaining tool calls, combining permissions, crossing an environment boundary nobody thought to close. The attack surface is the business logic itself: the decisions the agent makes about what to do next. No firewall inspects that. No data loss prevention rule catches it. Protecting the connection and the data is necessary and no longer sufficient — the open question of the next few years is who, and what, protects the logic.

The Market Is Already Pricing This In

The vendors have noticed. On September 16, Komodor launched its Agentic Operations Platform, and the governance features are the headline, not the footnote. Role based policies define who can invoke an agent and which credentials and tools it can touch. Guardrails check inputs, tool calls, and model responses before the agent acts, with risky actions gated for human approval. Spending limits and a full audit trail let platform teams see what every agent actually did.

The launch cites the number that should be on every architecture review deck: Gartner projects that more than 40% of agentic AI initiatives will be decommissioned by 2027 due to governance gaps, unclear ROI, or escalating costs. A separate Kore.ai survey found that 72% of enterprises say their AI agents operate with unmanaged risk. Meanwhile 60% of senior enterprise leaders are already deploying agents in production. Read those three numbers together and the shape of the problem is obvious: adoption is running well ahead of control.

InfoQ’s Cloud and DevOps Trends 2026 report tells the same story from the platform side. Agents for cloud engineering were promoted from Innovators to Early Adopters this year, but the panel was clear that enterprise adoption is gated by governance and compliance. The specific pain they named is telling: the Model Context Protocol, they observed, had a habit of “running roughshod over permissions and IAM,” with agents inheriting the permissions of whoever set them up. The fix arriving now — centralized auth for MCP, standard compliance checkpoints on which tools get exposed — is agent governance by another name.

Treat Agents Like Privileged Digital Workers

The mental model that works is not “chatbot with tools.” It is “high risk digital worker with production access.” You would never hand a new contractor a shared admin credential, an open path to the internet, and no logging, then walk away. An agent deserves the same skepticism, enforced in code.

That means a unique identity per agent, scoped permissions, short lived credentials, and a named human owner so every action traces back to a system, a use case, and an accountable person. It means access denied by default, with explicit approval gates for the high blast radius operations — internet access, code execution, credential retrieval, data movement, or any change to production. It means hard separation between test and production environments, so an evaluation harness can never reach a live customer system by accident. That last one is exactly the control that would have stopped the Anthropic and Meta breaches.

A Practical Checklist for Architects

Before an agent gets anywhere near production, walk this list. If you cannot check every box, the agent is not ready — the pilot is.

Identity. Every agent has a unique, non human identity with a named owner. No shared service accounts, no borrowed developer credentials.
Least privilege. Permissions are scoped to the task and deny by default. Credentials are short lived and rotated. High blast radius actions — code execution, data movement, production writes — sit behind explicit approval gates.
Containment. Test and production are hard separated at the network layer. Agents run in sandboxes with no default path to the open internet, and egress is allowlisted.
Observability. Every tool call, model response, and system interaction is logged. You monitor for the behaviors that matter — unusual tool chaining, unexpected data movement, unauthorized access attempts — not just crashes.
Kill switch. Security can halt any agent the moment behavior deviates from policy, and the mechanism is tested, not theoretical.
Cost control. Spending limits are enforced per agent. Token spend is attributed to an owner and a business outcome, because runaway cost is its own kind of incident.
Adversarial testing. You red team agents against realistic misuse — prompt injection, tool abuse, lateral movement, credential harvesting, sandbox escape — before launch, and you audit permissions and actual behavior on a schedule after it.

The Takeaway

The message for executives is not to slow down. Agents create real value, and the teams composing them into production workflows are not wrong to move. The message is that autonomy without accountability is a liability the balance sheet will eventually find. The three summer disclosures were early warnings delivered by the most sophisticated AI organizations on earth, using their own models, in controlled tests. If it can happen to them, the scaffolding is the risk — and the scaffolding is entirely within your control.

The organizations that win the next eighteen months will not be the ones with the cleverest agents. They will be the ones who built the governance stack first and let the agents run inside it. Left shift worked for security. It will work for agents. The only question is whether you build the guardrails before your first incident, or after.

DevOps,Cloud Architecture

The Silicon Under Your Self-Hosted LLMs — Graviton5, R9g, and the Real TCO of Open Weights

September 9 2026 Greg Fina

Last week we argued the models are ready. This week: the hardware just caught up too.

In Open Weight AI Models vs. Frontier APIs — The 2026 Cost Performance Tipping Point, we made the case that self-hosting open-weight models had finally crossed the economic line for a large slice of production workloads. But that post treated “self-hosted” as an abstraction — GPU rental math, per-token pricing curves, fine-tuning economics. It never asked the more grounded question: what silicon do you actually run these things on, and what does that silicon cost you per token?

On August 31, 2026, AWS made Amazon EC2 R9g and R9gd instances generally available, powered by Graviton5. These are memory-optimized Arm instances, and for a specific and growing class of LLM inference, they change the substrate calculus. This post goes one layer below last week’s argument — down to the memory controllers, the L3 cache, and the watts.

Inference is a memory bandwidth problem

Here is the counterintuitive thing that trips up teams sizing LLM infrastructure: for autoregressive token generation, you are almost never compute bound. Generating one token requires streaming the entire set of active model weights from memory through the compute units, then doing it again for the next token. At batch size one, arithmetic intensity is brutally low. The GPU or CPU spends most of its cycles waiting on memory.

That means the single most important spec for inference throughput is not FLOPS. It is memory bandwidth. This is why the Graviton5 memory subsystem matters more than the headline “25% better compute per vCPU” figure.

Graviton5 moves to DDR5-8800 MT/s memory, up from 5600 MT/s in Graviton4 — AWS calls it the fastest memory available in the cloud, and for a bandwidth-bound workload that is the number that moves tokens per second. Pair that with a 5x larger L3 cache, and more of a quantized model’s hot working set — attention KV cache, frequently touched layers — stays close to the cores instead of round-tripping to DRAM. The 25% per-vCPU compute uplift is real and welcome, but for inference it is the supporting act. Bandwidth and cache locality are the headliner.

Where CPU inference is “good enough” — and where it isn’t

Let me be precise, because Arm CPU inference gets oversold in both directions. R9g is not a GPU replacement. It is a GPU avoider for the right workloads.

CPU inference on R9g-class hardware is genuinely good enough when:

The model is small and quantized. A 7B–13B model at 4-bit (GGUF Q4, AWQ, or similar) has a weight footprint of roughly 4–8 GiB. That streams comfortably from DDR5-8800, and the whole model fits in memory many times over.
You are serving batch or async workloads. Document enrichment, classification pipelines, overnight summarization, embedding generation — anything where p99 latency is measured in seconds, not milliseconds, and where you care about cost per million tokens more than time to first token.
Your traffic is spiky or cost sensitive. CPU instances scale horizontally and cleanly on Spot, and you are not paying for an idle accelerator between bursts.

GPUs remain necessary when you need low single-request latency at interactive chat speeds, when you are serving large dense models (70B+ at high precision), or when you need very high concurrent batch throughput per node. The honest architecture is a split fleet: GPUs for the interactive tier, R9g for the batch and cost-sensitive tier. Last week’s post argued most tasks fit in the 7B–70B range; a meaningful fraction of those tasks also fit on a CPU, and that fraction is where R9g earns its place.

The real TCO at the hardware layer

R9g scales to 192 vCPU and 1,536 GiB of memory across 11 sizes, from r9g.medium up to r9g.metal-48xl. That memory ceiling is the point. A single r9g.48xlarge with 1,536 GiB holds a small library of quantized models resident in RAM simultaneously — no swapping, no cold-load penalty on model switch. For a multi-tenant inference gateway routing across a dozen fine-tuned variants, that is a real operational simplification.

A rough sizing intuition for capacity planning:

tokens/sec (batch=1)  ~=  memory_bandwidth / model_weight_bytes

# 4-bit 13B model, ~7 GiB active weights
# Graviton5 sustained BW is materially higher than Graviton4's,
# so per-node token throughput rises without adding a GPU line item.

The other half of TCO is energy. AWS describes Graviton5 as the most energy efficient processor it has ever built. For inference fleets that run continuously, the watts-per-token line eventually dominates the bill — and it is the line that most FinOps dashboards under-count because it hides inside the instance price. Fewer watts per token at the same throughput is a compounding advantage across a 24/7 fleet.

For the storage-hungry variants, r9gd adds local NVMe SSD — useful for staging model weights, vector index shards, or KV-cache spillover without hammering EBS. And on the largest sizes, R9g doubles network and EBS bandwidth versus R8g (up to 100 Gbps network and 72 Gbps EBS on the 48xlarge), with up to 3x higher packet-processing performance — which matters when your inference node is also fronting a high-QPS retrieval layer. Instance Bandwidth Configuration (IBC) lets you shift the EBS-versus-VPC allocation by 25% to match whichever side your pipeline leans on.

Underneath it all, R9g runs on the AWS Nitro System with the Nitro Isolation Engine — the first formally verified cloud hypervisor, with isolation guarantees established by mathematical proof rather than test coverage. For teams running customer data through self-hosted models, that isolation assurance is a compliance story you can actually put in writing.

Migration is a non-event

The best thing about R9g for anyone already on Arm: R8g to R9g is a drop-in. For most applications there are no code changes — you select the equivalent R9g size and your workload runs faster. It supports Amazon Linux 2023 and 2, Ubuntu 22.04+, RHEL 8.4+, SLES 15 SP3+, and Debian 12+. Containerized inference on EKS, ECS, or vanilla Kubernetes works as-is, and multi-arch Arm64 images run unchanged. Track the delta with the Graviton Savings Dashboard so the savings show up as a number your finance team believes.

R9g and R9gd launched in US East (N. Virginia, Ohio), US West (Oregon), and Europe (Frankfurt), available across Savings Plans, On-Demand, Spot, Dedicated Instances, and Dedicated Hosts.

Practical takeaways

Size for bandwidth, not FLOPS. For token generation, memory bandwidth and cache locality set your throughput ceiling. Graviton5’s DDR5-8800 and 5x L3 cache target exactly that bottleneck.
Run a split fleet. GPUs for the interactive tier; R9g for batch, async, and cost-sensitive inference on quantized 7B–13B models.
Consolidate models in memory. Use the 1,536 GiB ceiling on large R9g sizes to keep many quantized variants resident and eliminate cold-load latency.
Count the watts. Energy per token compounds on a 24/7 fleet — bake it into your TCO model, not just the sticker instance price.
Migrate first, optimize later. If you are on R8g, move to R9g as a no-code-change swap and measure the delta on the Graviton Savings Dashboard before you re-architect anything.

The models were ready last week. The substrate is ready this week. The interesting question for the rest of 2026 is no longer whether to self-host open weights, but how much of your inference fleet quietly moves off accelerators and onto CPUs you were already paying for. Where does your split land?

Last week we argued the models are ready. This week: the hardware just caught up too.

DevOps,Serverless,Lambda,GenAI

Killing the Cold-Start Tax on Serverless AI — Lambda SnapStart for Container Images

September 8 2026 Greg Fina Leave a comment

Lambda has been my favorite service for years. When I need to stand up a POC or a pilot, nothing beats it — I can deploy code in minutes, wire it to an event, and pay only for what actually runs. So it always stung a little that the standard architecture review for a real time inference endpoint ended the same way: “Lambda would be perfect for this… except cold starts.” Bursty traffic, event driven triggers, pay per use economics — serverless was the obvious fit on paper, and then someone would pull up a P99 latency chart and the conversation moved to a container platform or a provisioned endpoint instead.

That objection just lost most of its teeth — and the timing could not be better. With frontier AI taking off, the workloads I most want to prototype on Lambda are exactly the ones cold starts punished hardest. With Lambda SnapStart now supporting container image functions, the single biggest reason architects steered AI workloads away from Lambda is largely gone — and the design conversation shifts from “how do we survive cold starts” to “how do we design our init phase to be snapshotted.”

Why cold starts are especially brutal for AI

I learned this the hard way. A cold start is Lambda running your initialization code — loading the function, starting the runtime, and executing everything outside the handler — before it can serve the first request. For the plain CRUD functions I’d been happily shipping for years, that’s tens of milliseconds and nobody notices. The first time I dropped a model into a function, I found out that an AI workload is a different universe.

Three things stack up. First, the container image is large. A model runtime plus its transitive dependency tree — think a framework, a tokenizer library, numeric packages, and a CUDA adjacent stack — routinely produces images in the multi gigabyte range. Second, importing those dependencies is expensive: pulling a large ML framework into memory and resolving its native extensions can burn several seconds on its own, before you’ve touched a model. Third, and worst, you load weights at init. Reading a few hundred megabytes of parameters off disk or out of S3 and deserializing them into memory is the dominant cost, and it happens on every cold start.

Add those together and a “warm” invocation that returns in 80 ms is sitting behind a cold start path that takes five to ten seconds. I’ve watched a demo that flew on my laptop fall apart the moment a few concurrent requests forced Lambda to scale out into fresh environments. For a real time inference or agentic tool call, that is not a tail latency nuisance — it is timeouts, blown SLAs, and a genuinely bad user experience the moment traffic spikes. The cold start tax is levied precisely when you can least afford it.

What SnapStart for container images actually is

SnapStart attacks the problem at the mechanism level rather than asking you to shrink your dependencies. When you publish a function version, Lambda runs your function through the entire Init phase once, then takes a Firecracker microVM snapshot of the memory and disk state of that fully initialized environment, encrypts it, and caches it. On a subsequent cold start, Lambda does not re-run your initialization — it restores the microVM from that snapshot and jumps straight to handling the request.

The strategic point for AI workloads: your model load, your framework imports, your client construction all happen once, at publish time, and get frozen into the snapshot. Every future scale-out event restores from that frozen, ready to serve state instead of paying the multi second init bill again.

SnapStart itself is not new — it has been available for zip packaged functions across Java, Python, and .NET. What changed is that it now works for container image functions, and for me that’s the whole story. Containers are how I actually package these workloads. The 250 MB unzipped limit on zip functions and layers was always a poor fit for the gigabyte scale ML stacks I was building; the 10 GB container image support was the natural home. Until now I had to choose: container packaging or snapshot acceleration. I can finally have both.

How to make it work well

Here’s the mental model I’ve settled on. SnapStart rewards a specific architectural discipline: make the init phase do the expensive work, because init is what gets snapshotted.

Concretely, move model loading, dependency imports, and client/session setup to module scope — outside the handler — so they execute during Init and land in the snapshot:

# Runs at Init -> captured in the snapshot
import torch
from my_runtime import load_model

MODEL = load_model("/opt/ml/model")   # heavy: happens once, at publish
MODEL.eval()

def handler(event, context):
    # Warm path only: no model load here
    return MODEL.infer(event["input"])

The catch is uniqueness, and it’s the one that bit me. When Lambda restores many environments from one snapshot, any state created during init is shared across all of them — seeded random number generators, unique IDs, cached credentials, open connections. Freeze a database connection into the snapshot and you will hand every restored environment a stale, possibly closed socket. Seed a PRNG at init and every environment produces the same “random” sequence. I’ve debugged exactly that, and staring at duplicate “random” IDs across invocations is a humbling afternoon.

The fix is runtime hooks. For container images you implement before-snapshot and after-restore hooks so that the runtime coordinates the lifecycle. Use the before-checkpoint hook to gracefully close connections you don’t want frozen, and the after-restore hook to regenerate anything that must be unique: re-seed entropy, refresh temporary credentials, and re-establish network connections. Note the budget — after-restore work counts against a 10-second restore timeout, so keep it lean.

A few more practical levers:

Right-size memory. Memory scales CPU on Lambda, and restore plus after-restore work is CPU sensitive. For model serving functions, more memory usually pays for itself in lower restore and inference time.
Mind the image and layers. Snapshotting doesn’t excuse a sloppy image. Put model weights and heavy dependencies in stable lower layers, keep your handler code in a thin top layer, and you improve both build hygiene and load behavior.
Verify against real deployments. Restore latency depends on snapshot size, so benchmark P99 with your actual model rather than trusting a hello world number.

When it’s the right call — and when it isn’t

I want to be honest about the boundaries, because I’ve talked myself into using Lambda where I shouldn’t have. SnapStart on containers is a strong fit for bursty, spiky inference: workloads that idle then spike, where you’d otherwise overpay for always on capacity or eat cold starts on every scale-out. It’s excellent for agentic tool calls, where an orchestrator fans out to many short lived, independently scaling functions, and for RAG retrieval endpoints that load an embedding model and query a vector store. These are exactly the prototypes I keep reaching for right now.

It is a weaker fit in two cases. For sustained, high throughput inference that keeps environments warm anyway, provisioned concurrency or a dedicated container platform can still win on steady state cost and predictability — SnapStart’s advantage is amortizing init across intermittent cold starts, which matters less when you rarely have one. And for very large models needing GPU acceleration, Lambda is the wrong tool entirely; those belong on dedicated GPU endpoints such as SageMaker. SnapStart accelerates CPU bound init, not the physics of serving a 70B-parameter model.

The takeaway

Serverless AI just got materially more viable, and my favorite service is back on the table for the workloads I care about most. The reflexive “cold starts kill us” objection — the one that ended so many of my architecture reviews — no longer holds for a large and growing class of inference and agentic workloads. That doesn’t mean the work disappears; it moves. The new discipline is designing your init phase for snapshotting: front load the expensive work, handle uniqueness with runtime hooks, and right-size for restore.

Audit your init phase. Everything expensive — model load, imports, client setup — should run outside the handler so it lands in the snapshot.
Handle uniqueness explicitly. Use before-snapshot and after-restore hooks to close/reopen connections, refresh credentials, and re-seed entropy.
Benchmark P99 with your real model, not a toy function, and right-size memory to the restore path.
Pick the fit deliberately — bursty inference and agentic tool calls, yes; sustained high throughput or GPU bound serving, look elsewhere.
Revisit dismissed designs. Endpoints you ruled out over cold starts deserve a second look.

The objection didn’t just weaken — it changed shape. I’m already reopening a couple of “we can’t use Lambda for that” calls I made last year. Which of yours are you ready to revisit?