Last week we argued the models are ready. This week: the hardware just caught up too.

In Open Weight AI Models vs. Frontier APIs — The 2026 Cost Performance Tipping Point, we made the case that self-hosting open-weight models had finally crossed the economic line for a large slice of production workloads. But that post treated “self-hosted” as an abstraction — GPU rental math, per-token pricing curves, fine-tuning economics. It never asked the more grounded question: what silicon do you actually run these things on, and what does that silicon cost you per token?

On August 31, 2026, AWS made Amazon EC2 R9g and R9gd instances generally available, powered by Graviton5. These are memory-optimized Arm instances, and for a specific and growing class of LLM inference, they change the substrate calculus. This post goes one layer below last week’s argument — down to the memory controllers, the L3 cache, and the watts.

Inference is a memory bandwidth problem

Here is the counterintuitive thing that trips up teams sizing LLM infrastructure: for autoregressive token generation, you are almost never compute bound. Generating one token requires streaming the entire set of active model weights from memory through the compute units, then doing it again for the next token. At batch size one, arithmetic intensity is brutally low. The GPU or CPU spends most of its cycles waiting on memory.

That means the single most important spec for inference throughput is not FLOPS. It is memory bandwidth. This is why the Graviton5 memory subsystem matters more than the headline “25% better compute per vCPU” figure.

Graviton5 moves to DDR5-8800 MT/s memory, up from 5600 MT/s in Graviton4 — AWS calls it the fastest memory available in the cloud, and for a bandwidth-bound workload that is the number that moves tokens per second. Pair that with a 5x larger L3 cache, and more of a quantized model’s hot working set — attention KV cache, frequently touched layers — stays close to the cores instead of round-tripping to DRAM. The 25% per-vCPU compute uplift is real and welcome, but for inference it is the supporting act. Bandwidth and cache locality are the headliner.

Where CPU inference is “good enough” — and where it isn’t

Let me be precise, because Arm CPU inference gets oversold in both directions. R9g is not a GPU replacement. It is a GPU avoider for the right workloads.

CPU inference on R9g-class hardware is genuinely good enough when:

• The model is small and quantized. A 7B–13B model at 4-bit (GGUF Q4, AWQ, or similar) has a weight footprint of roughly 4–8 GiB. That streams comfortably from DDR5-8800, and the whole model fits in memory many times over.
• You are serving batch or async workloads. Document enrichment, classification pipelines, overnight summarization, embedding generation — anything where p99 latency is measured in seconds, not milliseconds, and where you care about cost per million tokens more than time to first token.
• Your traffic is spiky or cost sensitive. CPU instances scale horizontally and cleanly on Spot, and you are not paying for an idle accelerator between bursts.

GPUs remain necessary when you need low single-request latency at interactive chat speeds, when you are serving large dense models (70B+ at high precision), or when you need very high concurrent batch throughput per node. The honest architecture is a split fleet: GPUs for the interactive tier, R9g for the batch and cost-sensitive tier. Last week’s post argued most tasks fit in the 7B–70B range; a meaningful fraction of those tasks also fit on a CPU, and that fraction is where R9g earns its place.

The real TCO at the hardware layer

R9g scales to 192 vCPU and 1,536 GiB of memory across 11 sizes, from r9g.medium up to r9g.metal-48xl. That memory ceiling is the point. A single r9g.48xlarge with 1,536 GiB holds a small library of quantized models resident in RAM simultaneously — no swapping, no cold-load penalty on model switch. For a multi-tenant inference gateway routing across a dozen fine-tuned variants, that is a real operational simplification.

A rough sizing intuition for capacity planning:

tokens/sec (batch=1)  ~=  memory_bandwidth / model_weight_bytes

# 4-bit 13B model, ~7 GiB active weights
# Graviton5 sustained BW is materially higher than Graviton4's,
# so per-node token throughput rises without adding a GPU line item.

The other half of TCO is energy. AWS describes Graviton5 as the most energy efficient processor it has ever built. For inference fleets that run continuously, the watts-per-token line eventually dominates the bill — and it is the line that most FinOps dashboards under-count because it hides inside the instance price. Fewer watts per token at the same throughput is a compounding advantage across a 24/7 fleet.

For the storage-hungry variants, r9gd adds local NVMe SSD — useful for staging model weights, vector index shards, or KV-cache spillover without hammering EBS. And on the largest sizes, R9g doubles network and EBS bandwidth versus R8g (up to 100 Gbps network and 72 Gbps EBS on the 48xlarge), with up to 3x higher packet-processing performance — which matters when your inference node is also fronting a high-QPS retrieval layer. Instance Bandwidth Configuration (IBC) lets you shift the EBS-versus-VPC allocation by 25% to match whichever side your pipeline leans on.

Underneath it all, R9g runs on the AWS Nitro System with the Nitro Isolation Engine — the first formally verified cloud hypervisor, with isolation guarantees established by mathematical proof rather than test coverage. For teams running customer data through self-hosted models, that isolation assurance is a compliance story you can actually put in writing.

Migration is a non-event

The best thing about R9g for anyone already on Arm: R8g to R9g is a drop-in. For most applications there are no code changes — you select the equivalent R9g size and your workload runs faster. It supports Amazon Linux 2023 and 2, Ubuntu 22.04+, RHEL 8.4+, SLES 15 SP3+, and Debian 12+. Containerized inference on EKS, ECS, or vanilla Kubernetes works as-is, and multi-arch Arm64 images run unchanged. Track the delta with the Graviton Savings Dashboard so the savings show up as a number your finance team believes.

R9g and R9gd launched in US East (N. Virginia, Ohio), US West (Oregon), and Europe (Frankfurt), available across Savings Plans, On-Demand, Spot, Dedicated Instances, and Dedicated Hosts.

Practical takeaways

1. Size for bandwidth, not FLOPS. For token generation, memory bandwidth and cache locality set your throughput ceiling. Graviton5’s DDR5-8800 and 5x L3 cache target exactly that bottleneck.
2. Run a split fleet. GPUs for the interactive tier; R9g for batch, async, and cost-sensitive inference on quantized 7B–13B models.
3. Consolidate models in memory. Use the 1,536 GiB ceiling on large R9g sizes to keep many quantized variants resident and eliminate cold-load latency.
4. Count the watts. Energy per token compounds on a 24/7 fleet — bake it into your TCO model, not just the sticker instance price.
5. Migrate first, optimize later. If you are on R8g, move to R9g as a no-code-change swap and measure the delta on the Graviton Savings Dashboard before you re-architect anything.

The models were ready last week. The substrate is ready this week. The interesting question for the rest of 2026 is no longer whether to self-host open weights, but how much of your inference fleet quietly moves off accelerators and onto CPUs you were already paying for. Where does your split land?