The Migration Nobody Wanted to Do Is Now a Sprint: SQL to DynamoDB with Zero Downtime and a GenAI Copilot

Every team I have worked with hates PostgreSQL and MySQL maintenance. It does not matter how you run it, self managed or fully managed, it always comes with downtime. Someone breaks an index and causes an outage. And let’s not even talk about the DDL change that ran fine in test, then hung the production database during peak traffic.

But the minute someone proposes DynamoDB, the room changes. Everyone starts worrying about the heavy lift: reworking the application, rethinking every query, and moving column data into JSON or Ion payloads to store as items. The migration looks bigger than the pain, so it goes back in the backlog.

So next month the team suffers through another database upgrade. And another DDL outage.

The case for DynamoDB on high scale read and write paths was never the problem. You get consistent single digit millisecond latency, data replicated across multiple Availability Zones, and on demand capacity with no servers to patch. The problem was the work. What has changed is that GenAI coding tools now make most of that work cheap. They do not make the judgment calls cheap, and that is where I want to spend most of this post.

Why teams avoided it

Two things kept this migration in the backlog.

First, single table design forces you to think in access patterns instead of joins. In SQL you normalize the data and let the query planner figure out the rest. In DynamoDB you list every question the application asks, then design partition keys, sort keys, and global secondary indexes (GSIs) so each question is answered by a single Query or GetItem. That is a different mental model, and most teams have never written their access patterns down.

Second, the data access layer rewrite is long and boring. Every repository method, ORM call, and hand tuned query has to be translated. On a mature codebase that was months of work, and months of work with no new features is a hard sell.

The AWS Prescriptive Guidance on modernizing RDBMS to DynamoDB lays out this process in detail, and it is honest about how much of it is analysis rather than code.

The zero downtime pattern

The cutover pattern itself is well understood. The steps:

  1. Full load plus change data capture. AWS DMS copies the existing data and then streams ongoing changes from the source into DynamoDB. The DMS MySQL to DynamoDB walkthrough covers the object mapping you need to reshape rows into items.
  2. Dual writes behind a feature flag. The application writes to both systems. SQL stays the source of truth.
  3. Shadow reads. Serve from SQL, also read from DynamoDB, compare results, and log every mismatch. This is where you find the access pattern you forgot.
  4. Gradual cutover. Move reads to DynamoDB by percentage or tenant, then flip the source of truth, then retire the dual write.

There is now a better managed option for the backfill step. On November 5, 2026, AWS made bulk put items for DynamoDB generally available. You export your transformed data to S3, point bulk put items at it, and DynamoDB writes it into your existing table without impacting live workloads. The key detail for migrations: each item can carry its own condition expression. That means you can run the backfill as insert only (attribute_not_exists(PK)) so it never overwrites a fresher record from CDC or dual writes, or use a version guard so only older data loses. AWS lists backfills and migrations as explicit use cases, and the November 9 weekly roundup highlights it too. I would use it for the historical backfill and keep CDC or dual writes for live changes.

Where GenAI actually helps

This is where the cost curve bends. Four jobs that used to eat weeks are now hours of review:

  • Building the access pattern inventory. Feed the model your slow query log, ORM call sites, and API routes. It will cluster them into a list of patterns with frequency and latency targets. You still review it, but you are editing, not writing.
  • Drafting key designs and GSIs. Given the inventory, the model proposes partition keys, sort key prefixes, and GSI overloading. Treat this as a first draft from a junior engineer who read every DynamoDB talk.
  • Generating the repository layer rewrite. Interface stays the same, implementation swaps. This is mechanical translation, which is exactly what these tools are good at.
  • Producing parity tests and reconciliation scripts. Shadow read comparators, row count checks, and checksum jobs are tedious to write and easy to verify.

A version guarded write the copilot might draft for the dual write path:

table.put_item(
    Item={"PK": f"ORDER#{order_id}", "SK": "META", "version": new_version, **attrs},
    ConditionExpression="attribute_not_exists(PK) OR version < :v",
    ExpressionAttributeValues={":v": new_version},
)

Hype check: where AI does not help

Deciding which access patterns matter is a business decision. The model can list what the code does today. It cannot tell you which patterns the product will need next year, which ones carry revenue, or which can tolerate eventual consistency. It will also confidently design a hot partition. I have watched a copilot propose STATUS#OPEN as a partition key for an orders table, which sends every open order to one partition. Reviewing key cardinality is still your job.

Transactions and ad hoc reporting may stay in SQL. DynamoDB supports transactions, but if a workflow depends on complex multi table transactions or analysts running arbitrary joins, forcing it into DynamoDB is a mistake. The hybrid design guidance is explicit that keeping both is a valid architecture. Reporting can flow to an analytics store through zero ETL integrations instead.

Search is the wrong job for DynamoDB. A Scan reads the whole table, which is slow and billed by the data read. AI tools love writing Scans because a Scan with a filter works perfectly on a 500 row dev table. In production it becomes your biggest bill and your worst p99. The right pattern: stream data into OpenSearch through the DynamoDB zero ETL integration or Streams plus a pipeline, run the search there, return only primary keys, then fetch the authoritative record with GetItem or BatchGetItem.

flowchart LR
    App[Application] -->|writes| DDB[(DynamoDB table)]
    DDB -->|DynamoDB Streams / zero ETL integration| OS[(OpenSearch index)]
    App -->|1. search query: text, attributes| OS
    OS -->|2. returns matching primary keys only| App
    App -->|3. GetItem / BatchGetItem by key| DDB
    DDB -->|4. authoritative item, single digit ms| App

OpenSearch finds the item, DynamoDB serves the item. Never Scan the table to search.

The index can lag slightly behind the table, which is exactly why you return keys and read the item from DynamoDB rather than trusting the search document.

Practical playbook

  1. Write the access pattern inventory before any schema. Every pattern gets a name, a frequency, a latency target, and an owner who decides whether it matters.
  2. Have a human approve every partition key with an explicit cardinality estimate.
  3. Flag every Scan in generated code; route search access patterns to OpenSearch. Make this a lint rule, not a code review hope.
  4. Backfill with bulk put items using insert only or version guarded conditions, and keep CDC or dual writes for live traffic.
  5. Do not cut over until shadow reads show zero unexplained mismatches for a full business cycle.

A prompt template I use for step 1:

You are helping migrate a relational schema to DynamoDB.
Inputs: [slow query log], [ORM call sites], [API route list].
Produce a table of access patterns with: name, triggering API route,
source SQL, read or write, estimated requests per second, latency target,
and consistency requirement (strong or eventual).
Flag any pattern that is full text search, ad hoc filtering, or a
multi table join used only for reporting. Do not propose keys yet.

That last line matters. Keep the model listing before it starts designing.

Closing

The migration that lived in next half for three years now fits in a sprint or two, not because DynamoDB changed, but because the boring parts got cheap. What did not get cheap is deciding what the system should do. If you have run one of these with a copilot, I would like to hear where it saved you time and where it confidently led you into a hot partition.

Every team I have worked with hates PostgreSQL and MySQL maintenance. It does not matter how you run it, self managed or fully managed, it always comes with downtime. Someone breaks an index and causes an outage. And let’s not even talk about the DDL change that ran fine in...

Who Becomes Your Next Principal If Agents Do the Junior Work?

Over the years I have hired a lot of junior engineers. Some of my favorite moments as a leader were watching them grow: the first time one of them caught a bug I missed, the first promotion, and eventually the goodbye when they moved on to bigger jobs somewhere else. Every one of them climbed the same ladder. The bottom rungs were small features, bug fixes, and test coverage nobody else wanted.

These days I spend most of my time in customer meetings, and I find myself looking around the table and wondering about that ladder. The principal engineers are still there, sharp and opinionated. The junior engineers are there too, but I am less sure how they are getting the reps their predecessors got. Agents now draft most of that bottom rung work, and the data says those rungs are disappearing.

A Census Bureau Center for Economic Studies working paper found that early career employment in the most AI exposed industries fell 12% over the 10 quarters after ChatGPT launched, while less exposed industries stayed stable. The posting data rhymes: the share of tech jobs open to engineers with three years or less of experience fell from 43% in 2018 to 28% in 2024, according to Stanford Digital Economy Lab data summarized by HCAMag. It is tempting to read that as a 2026 hiring story. I read it as a 2031 principal story. The people who should be making staff and principal calls five years from now are the people we are not hiring, or not growing, today.

Be Honest About the Cause

Before we blame the agents, the hype check. Lambert and Schindler (2026) argue that remote work is a real confound. Distributed teams made mentoring harder and quietly made junior hires look riskier, and that shift started before generative AI was writing anybody’s code. HCAMag’s summary of the research puts it bluntly: the real reason junior hiring is collapsing may be your remote work policy.

I think both explanations are partly right, and I do not think it matters much which one wins the academic argument. The outcome for an engineering organization is identical. Fewer early career engineers are getting fewer reps on the kind of work that builds judgment. Whether the rep disappeared because an agent took it or because nobody was in the room to hand it over, the gap in your future leadership bench looks the same.

If you are a leader, that is the useful framing. You cannot fix macroeconomics or settle the remote work debate. You can decide how judgment gets built inside your own team.

What the Grunt Work Secretly Taught

Nobody designed the junior engineer curriculum. It emerged as a side effect of the work that needed doing.

Fixing a bug in a service you did not write taught you to read unfamiliar code, form a hypothesis, and test it against reality. Getting paged at 2 a.m. taught you what actually breaks in production, which is rarely what the architecture diagram suggested. Being wrong in code review, publicly and repeatedly, taught you to anticipate the objection before a senior engineer raised it. Writing the tests nobody wanted taught you where the edge cases live.

I saw this happen over and over with the engineers I hired. Nobody became senior because of one big project. They became senior because of hundreds of small, unglamorous fixes that each taught them something about how the system really behaved.

None of that was the point of the ticket. The ticket was “fix the null pointer in the billing job.” The lesson was how systems fail and how experienced people reason about failure.

When an agent drafts the fix, the ticket still gets closed. The side effect does not happen. We removed the work and, without noticing, removed the training program that was riding along with it. I wrote earlier this year that when building gets cheap, deciding gets expensive. The uncomfortable corollary is that we are making deciding more valuable at the same moment we are dismantling the pipeline that produced people who can decide.

The AI Drag Problem

The obvious response is “give the juniors agents too.” That is not a plan, and the evidence says it can make things worse.

Mark Russinovich and Scott Hanselman have described what they call AI drag, as covered in the InfoQ piece on the pipeline crisis. AI tools speed up senior engineers because seniors can steer the model and verify the output. The same tools can slow down juniors, who cannot yet tell a plausible answer from a correct one. A senior reads a generated diff and spots the race condition. A junior reads it, sees green tests, and ships.

I see a version of this in customer meetings. When a design question comes up, the principal engineer in the room usually asks the question that matters: what happens when this fails, who owns it, what does it cost at ten times the load. The junior engineers often have the generated answer ready faster than anyone, but they are not yet the ones asking that question. That gap is not about talent. It is about reps.

Blazity’s analysis of whether coding agents can replace junior developers lands in a similar place: agents can produce junior level output, but they do not produce junior level growth. Handing juniors an agent and walking away widens the gap between the people who already have judgment and the people who need to build it.

Build Judgment on Purpose

If the side effect is gone, the curriculum has to become explicit. Here are four practices I would put in place, each of which uses agents rather than pretending they do not exist.

1. Verification rotations

Make juniors the owners of verification for agent output. They write the evals, review the agent generated pull requests, and sign off on whether the change is correct. A senior engineer cosigns. The junior is no longer writing the fix, but they are doing the harder cognitive task: deciding whether a fix is actually right and explaining why. That is exactly the skill AI drag says they lack, so train it directly.

A simple rotation looks like this:

verification_rotation:
  owner: junior_engineer
  cosigner: senior_engineer
  duration_weeks: 2
  responsibilities:
    - write_or_extend_evals_for_agent_tasks
    - review_agent_prs_with_written_rationale
    - log_one_caught_defect_per_week
  exit_criteria: cosigner_agrees_on_90_percent_of_calls

2. Incident ownership

Put juniors on call early, in shadow mode, then have them lead the postmortem for an incident they shadowed. Agents can summarize logs, but leading a postmortem forces someone to build a causal story, defend it in front of peers, and propose a change that will survive scrutiny. That is principal work in miniature.

3. Design review apprenticeships

In every design review, assign a junior to write the “alternatives considered” section, especially the rejected options, and defend it live. Explaining why you did not choose an approach is where architectural judgment actually forms. It is also work an agent can draft but cannot own, because the defense happens in the room.

4. Incentives that count judgment

If your promotion packet still rewards PR volume, you are promoting people for the work agents now do. Count architecture decision records, evals that caught real defects, postmortems led, and design alternatives defended. Guillermo Power makes a related argument about what might actually work: the organizations that keep growing talent will be the ones that change what they measure.

None of these practices is expensive. Each one costs senior time, which is the point. Senior time spent growing judgment is the investment; agents are what free it up.

A Test for Principals

Before you set 2027 headcount, try one exercise: list every engineer with less than three years of experience on your team and write down the last time each of them owned a decision that could have gone wrong. If that list is short or empty, the agents aren’t the problem. The problem is that nobody is designing the pipeline.

The agents will keep getting better at junior work. Whether anyone gets better at principal work is still up to us.

Over the years I have hired a lot of junior engineers. Some of my favorite moments as a leader were watching them grow: the first time one of them caught a bug I missed, the first promotion, and eventually the goodbye when they moved on to bigger jobs somewhere else....

LLMjacking Is the New Cryptojacking — and That Should Terrify Your Security Team

There is a pattern no one names out loud, and it has repeated so reliably for fifteen years that I now treat it as a law of cloud physics: attackers monetize every new cloud resource in the same order. First they want raw capacity for whatever pays — spam relays, DDoS, credential stuffing. Then, when a new asset class gets valuable enough, they mine it. We lived through the cryptojacking era, where a leaked key meant your account quietly spun up a fleet of GPU instances to mint Monero on your dime. The novelty this time is not the attack. It is the speed. AI inference is the newest substrate, and it did not arrive as an emerging threat. It arrived pre industrialized.

The pattern, and why this iteration is different

If you have been doing this long enough, you can recite the sequence from memory: raw compute abuse, then cryptomining, then credential abuse and data theft, and now unauthorized AI inference. Each new layer rides on the same root cause — exposed credentials and internet facing endpoints that get scanned within hours of going live. What should concern your security team is not that AI inference joined the list. It is how fast it commercialized. Cryptojacking took years to develop a resale market. LLMjacking skipped the awkward adolescent phase entirely and showed up with a supply chain, a pricing model, and a storefront.

Sysdig coined the term “LLMjacking” in May 2024, and the framing was deliberate: this is the new cryptomining. Attackers are not primarily after your data. They are after your inference cycles — your paid access to frontier models — and the bill lands on you. Sysdig called it early. What happened next proved them right faster than anyone expected.

What changed: Operation Bizarre Bazaar

In early 2026, Sysdig and Pillar Security reported Operation Bizarre Bazaar, the first LLMjacking campaign with systematic attribution, a full supply chain, and confirmed marketplace monetization. The numbers are not subtle. Honeypots captured roughly 35,000 attack sessions between December 2025 and January 2026 — about 972 a day. This is not a proof of concept in a researcher’s lab. It is a running business.

The operation breaks into three stages, and each one is professionalized:

  1. Automated recon. Shodan and Censys sweeps for exposed endpoints — unauthenticated Ollama on port 11434, unauthenticated vLLM on port 8000, accessible MCP endpoints, and the forgotten dev and staging AI environments that nobody put behind auth.
  2. Quality based validation. Captured access is graded. Which models can this credential reach? What is the quota? A stolen key to a GPT class or Claude class model is worth more than one to a toy endpoint, and the operators price accordingly.
  3. Commercial resale. A marketplace called “silver.inc” resells access to 30+ LLM providers at 40 to 60 percent discounts, brokered through Telegram and Discord and paid in crypto and PayPal.

Read that last line again. There is a storefront. The Cloud Security Alliance research note documents the same marketplace dynamics. This is the detail that should change how you brief your leadership: your stolen Bedrock access is a SKU.

Why AI beats crypto as a target

Two things make AI inference a better target than cryptomining ever was, and both are economic.

First, the exposure is uniquely steep. With cryptojacking, your blast radius was bounded by how many instances the attacker could launch before you noticed, and the yield per instance was mediocre. Inference has no such ceiling. Sysdig estimated up to roughly $46,000 a day for Bedrock credential theft at peak quota, and one assessment projects north of $100,000 a day against the most capable frontier models. There is no mining rig that burns money that fast.

Second — and this is the part that keeps me up at night — it hides in plain sight at the API layer. A cryptomining fleet looks anomalous: unusual instance types, sustained 100 percent CPU, outbound traffic to mining pools. A bedrock:InvokeModel call from a compromised but valid IAM identity is indistinguishable at the API layer from legitimate use. There is no malformed packet, no weird binary, no new instance family. Fortinet’s threat research lays out the typical chain, and it is depressingly clean: a leaked long lived IAM key, a freshly created IAM user, a Marketplace model subscription, and then invocation. Every step uses legitimate APIs. Your tooling sees an authorized principal doing authorized things. The only tell is the bill.

The MCP twist: from inflated bill to lateral movement

Here is where a billing problem becomes a breach problem. By late January 2026, a separate MCP focused campaign accounted for roughly 60 percent of honeypot attack traffic. That shift matters because of what MCP servers do: they bridge AI models to file systems, databases, and internal tooling.

An exposed MCP endpoint is not just unauthorized inference. It is a foothold. Compromise one and you are not stealing cycles anymore — you are reading files and querying databases through the model’s tool connections. The “internal tooling” you wired up to make your agents useful is exactly the thing that turns LLMjacking into lateral movement. The economic abuse was the opening move. The MCP vector is the one that gets you a breach disclosure, not just a surprise invoice.

Hype check

So is this just cryptojacking repainted for the AI news cycle? I went in expecting to say yes, and mostly I have to say no.

The mechanism is identical — exposed credentials, internet facing endpoints, automated recon. If that is your definition, then fine, it is the same attack. But two things are genuinely worse. The economics are an order of magnitude steeper, because inference meters money faster than any mining rig. And the MCP blast radius means the worst case is no longer a cleanup and a credit from your account team — it is data exfiltration through a trusted bridge.

Where the hype deflates is the defense. There is no exotic new control to buy. The fundamentals that would have stopped cryptojacking in 2019 stop LLMjacking in 2026. That is the uncomfortable truth: we keep rebuilding the same house on the same unlocked foundation and acting surprised when a new kind of burglar walks in.

Practical takeaways

  1. Fix credential hygiene first. Short lived, role assumed credentials everywhere. No never expiring admin keys. The entire Bizarre Bazaar supply chain starts with a leaked long lived key — kill the input and you kill the campaign.
  2. Assume breach for anything internet exposed. If an Ollama, vLLM, or MCP endpoint is reachable, treat it as already scanned. It will be found within hours. Put auth in front of it or take it off the internet.
  3. Treat cost anomalies as security signals. Wire FinOps to SecOps. A sudden Bedrock spend spike is not a billing ticket — it is potentially your earliest and sometimes only detection signal. Alert on it like an IDS hit.
  4. Govern your model and MCP endpoints. Inventory them, authenticate them, rate limit them, and monitor them. An MCP server with filesystem access and no auth is a breach waiting for a Shodan hit.
  5. Turn on the logs that are off by default. CloudTrail everywhere, and Bedrock model invocation logging, which is not enabled by default. If an invocation is indistinguishable at the API layer from legitimate use, the invocation logs are the only place you will ever see it.

The attackers have already built the marketplace. The question for your security team is not whether your AI resources are interesting to them — they are, and they are priced. The question is whether the boring fundamentals are in place before the recon sweep finds you. They almost never are. Start there.

There is a pattern no one names out loud, and it has repeated so reliably for fifteen years that I now treat it as a law of cloud physics: attackers monetize every new cloud resource in the same order. First they want raw capacity for whatever pays — spam relays,...

Your Repo Was Built for Humans. Agents Commit Now.

Last quarter one of your teams opened 140 pull requests in a week. Eleven engineers wrote nine of them. The rest came from coding agents, and your CI bill and your senior reviewers both noticed before you did.

This is no longer a thought experiment. Kiro workflows, announced this week, let multiple agents carry a complex task from start to finish with less supervision. Git 2.56 shipped git add --resolved, which refuses to stage files that still contain conflict markers, and git branch --delete-merged for bulk cleanup of landed topic branches. Those are exactly the safety rails an unattended agent needs to merge and tidy up on its own. The tooling now assumes machines commit. Most repositories, pipelines, and review processes still assume humans do. That gap creates two costs that principals should own: CI capacity and reviewer attention. Both are finite budgets, and agents will spend them as fast as you let them.

Section 1: Control pipeline cost before it controls you

Why agent volume inflates CI spend

Agents work differently from people. They push in small increments, retry when a test fails, open draft PRs to check their own work, and iterate in tight loops. A human might push four times on a feature branch. An agent might push forty. If every push triggers your full pipeline, including integration suites and end to end tests against ephemeral environments, you pay full price for every intermediate guess.

Tier the pipeline by PR state

Not every push deserves the whole pipeline. Split CI into tiers:

  • Every push: lint, type checks, unit tests, secrets scanning. Minutes, not tens of minutes.
  • PR marked ready for review: integration tests, end to end suites, performance checks.
  • Merge queue: the final gate against the current main branch.

In GitHub Actions, the ready state is an event you can gate on:

on:
  pull_request:
    types: [opened, synchronize, ready_for_review]

concurrency:
  group: ci-$
  cancel-in-progress: true

jobs:
  fast:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: make lint unit
  full:
    if: github.event.pull_request.draft == false
    needs: fast
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: make integration e2e

Agents iterate cheaply in draft. They pay for the expensive suites only when they claim they are done.

Budgets, quotas, and queue priority

Give each agent identity, or each repository, a monthly CI minute cap. Put agent jobs in a separate runner pool with a concurrency limit so a burst of agent retries cannot starve a human who is waiting to ship a hotfix. When the queue is contended, human PRs go first. An agent that hits its quota should stop and report, not silently queue more work.

Smarter triggers

Three techniques cut most of the waste:

  1. Test what changed. Use path filters or a build system with dependency graphs (Bazel, Nx, Gradle) to run only affected targets.
  2. Cache aggressively. Dependency caches and remote build caches mean the fortieth push reuses the work of the thirty ninth.
  3. Cancel superseded runs. The concurrency block above kills the in flight run when the agent pushes again to the same branch. No one needs results for a commit that is already obsolete.

Make the cost visible

You cannot manage what the bill hides. Tag every pipeline run with the author type: agent or human, plus the agent identity. If you run builds on AWS CodeBuild or self hosted runners on EC2, propagate those tags into cost allocation tags so Cost Explorer shows what agents actually cost per merged change. That number turns a vague concern into a budget conversation.

Section 2: Automated review, human approval

The review problem

Line by line human review does not scale to agent volume. Teams that try anyway get one of two outcomes. Reviewers burn out, or they start skimming and approving, which is worse because it looks like control. Removing humans entirely is not an option for production systems; someone has to own the decision to ship. The answer is to change what the human reviews.

Automation does the first pass

Before a human sees an agent PR, it should clear an automated gauntlet:

  • An AI review agent that reads the diff against the linked task and comments on logic, edge cases, and missing tests.
  • Static analysis and type checking.
  • Security scanning and secrets detection.
  • License and dependency checks for every new or bumped package.
  • Policy as code for infrastructure. Use AWS CloudFormation Guard for templates or OPA for Terraform plans and Kubernetes manifests.
cfn-guard validate \
  --data templates/ \
  --rules policies/org-baseline.guard

Give the human a review brief

The output of that first pass is not a wall of bot comments. It is a structured brief posted at the top of the PR:

REVIEW BRIEF
What changed:   Adds retry with backoff to payments client
Why:            Task PAY-412, intermittent 503s from processor
Risk tier:      HIGH (touches payments/, IAM policy)
Sensitive files: infra/iam/payments-role.yaml
Coverage delta: +3.1% (12 new tests)
Flags:          IAM policy adds kms:Decrypt on wildcard resource

The reviewer reads the brief, then goes straight to the flagged areas. Attention lands where the risk is.

Humans still approve

Automation can block. Automation can flag. Automation never approves a merge to a production path. Require approval from a named owner, routed by CODEOWNERS and by risk tier, enforced with branch protection. That preserves accountability: when something breaks, there is a person who decided it was ready, not a pipeline that shrugged.

Risk tiers set review depth

  • Low risk (docs, tests, minor dependency bumps that pass all checks): the human confirms the brief matches the intent and approves.
  • High risk (IaC, authentication and authorization, data schemas, production configuration): deep human review, but focused on the regions automation narrowed down.

Keep the reviewers honest

A human approval step is only valuable if it is real. Measure it:

  • Time to approve. A high risk PR approved in 40 seconds is a signal.
  • Flag override rate. How often do reviewers dismiss automated flags, and were they right?
  • Audit sampling. Pull a random sample of approved PRs each sprint for a second review.
  • Escaped defects. Track production issues the automated reviewer missed and feed them back into rules and prompts.

Hype check: is AI reviewing AI the blind leading the blind?

Partly true, mostly misdirected. Automated reviewers are excellent at patterns: known vulnerability classes, policy violations, leaked secrets, missing tests, license conflicts. They are consistent and they do not get tired at 4 PM on Friday. They are weak at intent. Does this change solve the right problem? Does it fit the architecture the team agreed on? Will it create an operational burden in six months? Those questions need a human with context, and accountability for production needs a human with a name.

So the real risk is not that a model reviews another model’s code. The real risk is a human approval step that quietly decays into a rubber stamp while everyone believes it is still a control. Design for that failure mode, measure it, and you get the volume benefits of agents without giving up ownership.

Practical takeaways

  1. Tier your pipeline so agent drafts run only fast checks; reserve integration and end to end suites for ready PRs.
  2. Set CI minute caps and concurrency limits per agent, and prioritize human PRs in the queue.
  3. Tag pipeline runs by author type so agent CI cost appears on the bill.
  4. Require an automated first pass and a structured review brief before any human review.
  5. Keep named human approval for production paths, routed by risk tier, and audit it with real metrics.

Agents will keep getting faster at writing code. The scarce resources in your organization are now compute budget and human judgment. Spend both deliberately. How is your team handling agent PR volume today?

Last quarter one of your teams opened 140 pull requests in a week. Eleven engineers wrote nine of them. The rest came from coding agents, and your CI bill and your senior reviewers both noticed before you did.

This is no longer a thought experiment. Kiro workflows, announced this...

The Principal's Filter: Sorting Agent Hype from What Ships

Last month a customer walked me through a slide deck with eleven “agents” on it. By the end of the hour we had found one real agent, three workflows, two chatbots, and five ideas that nobody had scoped yet. Nobody in that room was being dishonest. The word “agent” has simply stretched to cover almost anything with a model behind it.

That is the real job of a principal right now. Customers do not need another person cheering for agents. They need someone who can sort what is viable from what is not, then help them build the right thing, which is sometimes an agent and often is not.

Why This Matters Now

The gap between buzz and production is wide. A Gartner 2026 CIO survey found that only 17% of organizations have deployed agents, while more than 60% expect to within two years. The Sinequa State of Enterprise Agentic AI 2026 report is even more sobering: only 24% have deployed a true agent, and only 10% have true agentic capabilities. Gartner also forecasts that more than 40% of agentic AI projects will be canceled by the end of 2027 because of rising cost, unclear value, and weak risk controls.

Notice what is missing from that list: model quality. In my experience, most agent projects do not fail because the model is weak. They fail because of how they are governed and run. The value was never clear, the costs crept up, the risk controls were thin, and nobody was assigned to manage the agent once it went live.

So I use a filter. Five questions, asked early, before anyone writes a line of orchestration code.

Filter 1: Is It Really an Agent?

The industry now has a name for the relabeling problem: agent washing. Chatbots, assistants, and RPA tools are being renamed as agents, and buyers are paying agent prices for them.

My test is simple. A real agent does three things:

  • Plans. It breaks a goal into steps it was not explicitly given.
  • Uses tools. It calls APIs, queries data, or takes actions in other systems.
  • Acts on its own. It decides the next step based on what it just observed, without a human or a fixed script choosing for it.

If the steps are known in advance, you have a workflow. Build it as a workflow. A state machine with a model call in one or two steps is cheaper, easier to test, and easier to explain to an auditor. I tell customers this is not a downgrade. It is the correct design.

Fixed steps, known branches   -> Workflow (Step Functions, plus a model call)
Answer questions, no actions  -> Assistant / RAG
Repetitive UI clicks          -> RPA
Open ended goal, tool choice  -> Agent

Filter 2: What Happens When It Fails?

Every agent demo works. That is what demos are for. The question is what happens on day thirty when the input looks different. Agents that shine in demos break on new input formats, loop on vague instructions, or quietly make up data. The last one is the dangerous one, because a confident wrong answer does not trigger an alarm.

I ask the team to describe three failure paths out loud:

  1. Catching it. How do you know the agent failed? Schema validation on outputs, checks against a system of record, and confidence thresholds that route to a human.
  2. Containing it. What is the blast radius? Scoped credentials, read only access by default, step and spend limits per task, and an approval gate before any irreversible action.
  3. Recovering from it. Can you replay the trace, see which tool call went wrong, and roll back what the agent did?

If the answer to any of these is “the model is pretty good, so it should be fine,” the project is not ready.

Filter 3: Can You Measure and Test It?

You cannot improve what you cannot measure, and you cannot ship what you cannot test. Before production, I want to see an evaluation suite built on the customer’s real edge cases, not the vendor’s happy path examples.

A useful starting point is fifty to two hundred real tasks pulled from tickets, logs, or past work, each with a known good outcome. Run the agent against them on every prompt change, model change, and tool change. Something as plain as this works:

results = []
for case in eval_cases:
    outcome = agent.run(case["input"], max_steps=15)
    results.append({
        "id": case["id"],
        "passed": grade(outcome, case["expected"]),
        "steps": outcome.step_count,
        "cost_usd": outcome.cost_usd,
    })

pass_rate = sum(r["passed"] for r in results) / len(results)
print(f"pass rate: {pass_rate:.1%}")

Then carry the same idea into production. Sample live traces, grade them, and track pass rate, step count, and escalation rate over time. Offline evaluation tells you whether to ship. Production monitoring tells you when to stop.

Filter 4: What Does Each Completed Task Cost?

Token pricing is the wrong unit. Customers get excited about a fraction of a cent per call, then discover that one finished task took nine calls, two retries, a reasoning loop, and fifteen minutes of a person checking the result.

The number I ask for is cost per completed task:

cost per completed task =
  (model calls + tool calls + retries + loops
   + infrastructure + human review time)
  / tasks completed correctly

The denominator matters as much as the numerator. If the agent completes 70% of tasks correctly, the other 30% still cost money and still need a person to finish them. Compare the result to what the work costs today. If the agent is not clearly cheaper, faster, or better at the same quality bar, the value case is not there yet. This is exactly the “rising cost, unclear value” pattern behind those cancellation forecasts.

Filter 5: Who Manages It Once It Is Live?

This is the question that kills the most projects, and the one teams skip most often. An agent is not a feature you ship and forget. It behaves more like a new team member who needs a manager.

Someone has to:

  • Define the work. What tasks are in scope, and which are not.
  • Set the quality bar. What counts as done, and what error rate is acceptable.
  • Step in when things go wrong. Review escalations, pause the agent, and fix the prompt, tools, or data.
  • Own the evaluation suite. Add new edge cases as production reveals them.

I ask for a name, not a team. If no single person owns that job, with time actually allocated to it, the project is not viable, regardless of how good the demo looked.

Practical Takeaways

  1. Classify before you build. Label every proposed “agent” as a workflow, assistant, RPA, or agent. Most will not be agents.
  2. Write the failure paths first. Catch, contain, and recover, documented before the first sprint.
  3. Build the evaluation suite from real data. Use the customer’s edge cases and rerun it on every change.
  4. Price the finished task, not the token. Include retries, loops, and human review.
  5. Name the owner. No operational owner, no production launch.

Saying “Not Yet” Is the Job

Principals earn trust by being the person in the room willing to say “not yet,” or “a simpler build will work better.” Customers remember who saved them from an expensive cancellation far longer than who sold them the shiniest demo.

And when the filter says an agent really is the answer, the right primitives exist. As I wrote in Agents Aren’t Web Requests, agents need runtimes built for long running, stateful, isolated work. Get the governance right first, and the infrastructure part becomes the easy part.

What questions are in your filter? I would like to hear what you ask customers before you let an agent near production.

Last month a customer walked me through a slide deck with eleven “agents” on it. By the end of the hour we had found one real agent, three workflows, two chatbots, and five ideas that nobody had scoped yet. Nobody in that room was being dishonest. The word “agent” has...

Agents Aren't Web Requests — Why AWS Rebuilt the AgentCore Runtime

Years ago I sat with a customer who needed to orchestrate multiple agents inside their software products — agents that coordinate, hand work to each other, and stay alive across a workflow, not one model answering one call. We wrote a PRFAQ together, took it into an EBC, and a VP agreed to build it. The document was right. It was also complex — it described, in detail, nearly every capability you can find in AgentCore today. The service team then rewrote it, more than once, not to correct it but to break a correct and complex vision into something they could actually execute and build. The primitives got names and edges they did not have in our draft. But the shape of the thing never changed, because we had the mental model right from the start: these are not web requests. They are sessions.

That is the part I am proud of, and it is the only part that matters here: the vision was correct and complete, and the one idea holding all of it together was that an agent is a session, not a request.

Here is the mental model that quietly breaks the moment you ship an agent to production: “an agent is just a web request that happens to call an LLM.” You wire up an HTTP handler, it fires off a prompt, you stream tokens back, the function returns, the compute is reclaimed. Clean. Stateless. Familiar.

It is also wrong in ways that will cost you. An agent is not a request. It is a process — one that reasons over many turns, accumulates intermediate results, calls tools that mutate state, occasionally reaches for a GPU, and sometimes hands work to other agents. The shape of that workload has almost nothing in common with the shape of an HTTP handler. Amazon’s answer, Amazon Bedrock AgentCore Runtime, is best understood not as “serverless with a bigger timeout” but as a deliberate rethinking of what compute for agents should look like.

What breaks when you run an agent like an HTTP handler

Serverless web infrastructure makes three assumptions that are load bearing for request response traffic and actively hostile to agents.

It assumes statelessness. An HTTP handler is supposed to carry nothing between invocations; that is what lets the platform scale it horizontally without thinking. But an agent’s whole value is the state it carries — conversation history, a scratchpad of intermediate reasoning, the output of a tool it called two steps ago, a half built file on disk. If every invocation lands on a fresh execution environment with an empty filesystem, you are forced to serialize and rehydrate the entire context on every single turn. That is slow, lossy, and fragile.

It assumes short duration. Request response compute is tuned for work that completes in milliseconds to seconds, with hard timeouts measured in minutes. Agents routinely run for minutes to hours, and some genuinely need to run continuously for days — a transformation job grinding through a corpus, an automation loop that pauses and resumes, a research agent that keeps working while a human is asleep.

It assumes weak isolation is fine. When requests are stateless and ephemeral, co tenancy is mostly an efficiency detail. But agents run privileged, nondeterministic code on a user’s behalf — they get shell access, read and write files, and invoke tools with real credentials. The isolation boundary stops being a performance concern and becomes a security boundary. You cannot have user A’s agent able to observe anything left behind by user B’s.

How AgentCore answers it: isolated sessions, not shared handlers

AgentCore Runtime’s primitive is the session, not the request. Each user session runs in its own dedicated microVM with isolated compute, memory, and filesystem, plus shell access. One user’s agent cannot reach another user’s data, and when the session ends the entire microVM is torn down and memory is sanitized — no cross session residue. That is a deterministic isolation boundary wrapped around a deliberately nondeterministic workload, which is exactly the property enterprises need.

Inside that boundary, the model does the thing HTTP handlers refuse to do: it lets you safely reuse context across invocations. You generate a session ID, pass the same ID on every related call, and each invocation builds on the environment the previous one left behind — same filesystem, same in memory state, same tool context. Multi turn conversations and multi step workflows stop being a serialization problem and become what they always should have been: successive calls into a living environment.

The honest part of the design is that this state is ephemeral by default. In memory and on disk data lives for the session’s lifetime and no longer. For anything that must outlive the session — learned preferences, durable conversation history, workspace files — you pair the runtime with AgentCore Memory for long term recall and Amazon EBS for persistent volumes. The runtime handles scaling, session management, isolation, patching, and observability so you spend your time on agent logic, not plumbing.

Two compute shapes, one set of APIs

The sharpest architectural decision is that AgentCore exposes two compute shapes under the same runtime APIs, so you match the shape of the compute to the shape of the traffic.

Serverless microVMs are the default: fast cold starts, scale to zero, sessions up to 8 hours. This is the right home for request response, spiky, or latency sensitive agents — the chat assistant, the API driven tool, the thing that needs to start instantly and cost nothing while idle.

Runtime instances, generally available since August 6 2026, are AWS managed EC2 running in your own account, defined by a reusable capacity provider (operating system, allowed instance types, networking, storage, IAM roles). Sessions here run up to 14 days, support GPU instance families with drivers provisioned for you, and can stop and restart to save cost — hibernate Monday night, resume Wednesday morning with volumes reattached and data intact. This is the home for long running, continuous, GPU bound, or multi agent work.

Because the instances live in your account, your data and your account controls stay put, and you can apply existing Savings Plans, Reserved Instances, and ODCRs. AgentCore still owns the lifecycle — provisioning, patching, scaling, teardown — so you get EC2 economics without EC2 babysitting.

Multiple agents on one host

Runtime instances change the unit of collaboration. In the microVM model, one runtime hosts one agent. On an instance, a single session can host many agents that share a filesystem. AWS’s own demo makes the point: a writer agent generates code into a shared session directory, and a reviewer agent reads that same file and critiques it — no message passing, no inter agent API calls, no data transfer. They collaborate through the filesystem.

That composes cleanly with the two shapes. A lightweight orchestrator on a microVM can take fast, spiky, API driven traffic and dispatch heavier work to specialized worker agents running on instances — code compilation, security scanning, GUI automation — that need persistent state and direct OS access. And none of this locks you into a framework: bring CrewAI, LangGraph, LlamaIndex, or Strands, and any model. Packaging is minimal — an @app.entrypoint decorator plus a zip or a container image.

The caveat worth saying out loud

Here is where I will push back on the easy narrative. A 14 day session is not a durable workflow. If your process waits days for a human approval, or moves money, or must survive infrastructure failure with transactional guarantees, a single long lived session is a fragile place to park that state. Runtime instances raise the ceiling for continuous work; they do not turn one invocation into a saga. For orchestration that must be durable and recoverable, reach for a real orchestrator — AWS Step Functions over a transactional store — and let the agent be a step inside it, not the system of record.

Two more things to keep honest. Instances do not scale to zero; an idle instance still bills until you stop the session. And the whole point of the two shapes is that neither is universally correct. Match the compute shape to the traffic shape or you will overpay for idle capacity or starve a long job of the persistence it needs.

Takeaways

  1. Model agents as sessions, not requests. Design around a stable session ID and context that lives across invocations, not stateless handlers that rehydrate everything each turn.
  2. Pick the compute shape from the traffic shape. Spiky and latency sensitive goes on microVMs; continuous, GPU bound, or multi agent goes on instances. The APIs are the same, so switching costs are low.
  3. Separate ephemeral session state from durable state. Use the session filesystem for working data, AgentCore Memory and EBS for anything that must outlive it.
  4. Do not confuse long sessions with durable workflows. For human approvals, money movement, or anything needing transactional recovery, put a Step Functions orchestrator in charge and make the agent a step.
  5. Watch idle cost on instances. No scale to zero means you stop sessions deliberately.

The deeper story is that agents are a genuinely new compute primitive, and infrastructure is catching up to that fact. For a decade we bent every workload to fit the web request. AgentCore is a bet that it is time to bend the compute to fit the workload instead. If your agents still look like HTTP handlers in disguise, that is probably the first thing to rethink.

Years ago I sat with a customer who needed to orchestrate multiple agents inside their software products — agents that coordinate, hand work to each other, and stay alive across a workflow, not one model answering one call. We wrote a PRFAQ together, took it into an EBC, and a...