Below the Ice — Why Agentic AI Had to Rebuild Silicon from Scratch

The evening deep-dive from The Penguin Alley. One topic, told properly. Listen to tonight's episode

---

NVIDIA launched Vera Rubin today. The headlines will talk about benchmark numbers, production partners, and the sheer size of the NVL72 rack. Those numbers are real. But they're the surface.

Tonight we go under — to the question no benchmark answers: why did agentic AI force a ground-up rethink of silicon that was already the best in the world?

What it is

Vera Rubin is NVIDIA's new computing platform, named after the astronomer who found the first compelling evidence for dark matter. The woman who discovered the universe is mostly invisible — which feels like fitting company for a chip designed to power a kind of intelligence that mostly runs out of sight.

The platform has two major pieces. The Rubin GPU handles the heavy math. The Vera CPU — built around new "Olympus" cores — handles everything around the math. Together, they form the NVL72: a rack-scale system connecting 72 Rubin GPUs into a single, shared memory domain. Production is ramping now at CoreWeave, Google Cloud, Microsoft Azure, and Oracle Cloud Infrastructure.

That's what it is. Now for why it had to exist.

How it actually works

For most of AI's recent history, GPU design had one guiding assumption: the GPU is the computer, and the CPU is just the traffic cop. Training a large model is embarrassingly parallel work — billions of numbers multiplied simultaneously, the same operation repeated across thousands of compute cores. The CPU's job was to load data and stay out of the way.

This shaped a decade of silicon design. GPU companies poured engineering into raw parallel throughput — FLOPS, memory bandwidth, interconnect speed between GPUs. The CPU mattered only at the margins.

Agentic AI broke that assumption.

Here's the analogy. Imagine a very fast chef (the GPU) who can slice a thousand vegetables in a minute. In the old world, the kitchen was simple: you handed the chef a massive pile of ingredients, they processed it all, and you took the dish. Now imagine the chef has to stop every few minutes to: check the recipe online, call a supplier, run a quick experiment, decide whether the next step is braising or roasting, and pick the next ingredient based on what just happened. The speed of slicing no longer determines the speed of cooking. The bottleneck moved to the chef's decision loop — and that loop runs on the CPU.

That is the agentic AI shift. When a model is simply answering questions, the GPU does everything. But when an AI agent runs a code sandbox, calls an external API, waits for a result, parses it, and decides what to do next — that sequential decision logic runs on the CPU between tool calls. NVIDIA's own analysis showed that agentic AI moves a meaningful fraction of the critical execution path onto the CPU. A slow CPU doesn't just cost cycles; it costs the latency every agent pays between each step it takes.

The Olympus cores in the Vera CPU are NVIDIA's response: custom CPU cores optimized for exactly this — maximum single-threaded, latency-sensitive orchestration work, designed specifically for a world where intelligence is planning and acting, not just computing.

The GPU side of Vera Rubin had its own forcing function: Mixture of Experts (MoE). Almost every frontier model being trained today uses MoE architecture — where a model has many specialized sub-networks (the "experts") and routes each token to only a few of them. In practice, only about 10–25% of the model's parameters are active on any given forward pass. This makes very large models affordable to run. But it changes the memory access pattern entirely.

Instead of loading a fixed set of weights every pass, the system has to rapidly fetch different expert sub-networks depending on which experts were chosen for each token. This puts enormous pressure on memory bandwidth and interconnect speed — you need to move the right experts into compute cores faster than the compute itself takes. The Rubin GPU uses HBM4 memory (higher bandwidth than Blackwell's HBM3e) and NVLink5 for GPU-to-GPU communication. The NVL72's 72-GPU shared memory pool is sized specifically to keep whole MoE models resident without constant paging. NVIDIA announced a world record for MoE pre-training throughput on the GB300 NVL72 this week.

The networking piece ties it together. When you scale to hundreds of thousands of GPUs — what NVIDIA calls the "gigascale AI factory" era — the interconnect between racks becomes the bottleneck. Spectrum-6 is a new InfiniBand networking architecture built to make that interconnect a multiplier, not a limiter.

Why it matters now

If you build with AI models, you are already downstream of this. The hardware determines what models can be trained, which shapes what models get trained, which shapes what you can build with. A platform purpose-built for MoE pre-training at gigascale is a platform designed to produce the next generation of very large, very capable models — at lower per-token cost than a training cluster that wasn't designed for it.

The CPU story is more immediate. As you push more of your application into agentic workflows — loops, tool calls, multi-step reasoning — the execution cost shifts from GPU inference time toward CPU orchestration latency. Knowing that the hardware roadmap is addressing this means the platforms you're building on today are heading toward shorter tool-call round-trips and faster agent step cycles. That changes what's worth building.

For teams running their own inference infrastructure: the MoE + HBM4 combination is a preview of the access pattern your serving stack needs to optimize for. Experts that can be loaded and swapped fast, not monolithic weight tensors that have to be held resident in full.

What is overhyped

The benchmark theater. Every launch cycle, record-breaking training throughput numbers get announced, and they are real — but they're peak throughput under ideal conditions, not the throughput your production workload will see. Real agentic workloads have uneven GPU utilization, CPU-bound phases, and bursty network traffic that look nothing like the benchmark setup.

The supply story also needs context. Vera Rubin NVL72 production is ramping with hyperscaler partners. For most teams building with AI today, Vera Rubin is three to five layers of abstraction away — you're calling an API that runs on whatever hardware the cloud provider chose. The platform shapes your world, but through the pricing and model capability it enables, not the spec sheet.

What to watch

Three things worth tracking:

1. MoE becomes the default. Watch for new model releases from every major lab to converge on MoE architectures. The hardware now strongly favors them. Mixture-of-Experts goes from a clever optimization to the standard approach.

2. CPU orchestration as a first-class benchmark. Today's AI benchmarks measure GPU throughput. As agentic workloads become the real production case, watch for a new class of benchmark that measures agent step latency — how fast a system can complete a full loop of reason, act, and observe.

3. Spectrum-6 adoption and what it unlocks. Gigascale AI factories running Spectrum-6 will be able to train at a scale and speed that simply wasn't possible before. The models that emerge from those factories in 12–18 months are the ones you'll be shipping applications on in 2027.

---

Tonight's episode covered the Vera Rubin launch from the hardware up. If you want to go deeper on the MoE architecture side, the world-record MoE pre-training post from NVIDIA is the clearest technical write-up we found. And if you want the networking piece, Spectrum-6's own launch post explains the AI factory interconnect story well.

See you tomorrow morning on Cold Open.

---

Sources:

• Inside NVIDIA Rubin GPU Architecture — NVIDIA Developer Blog
• Inside the Vera CPU: Olympus Cores — NVIDIA Developer Blog
• NVIDIA Vera Rubin: Performance Per Watt — NVIDIA Technical Blog
• World Record MoE Pre-Training on GB300 NVL72 — NVIDIA Developer Blog