Below the Ice — The Mesh That Thinks
What if your laptop, a friend's server, and a rented VPS could pool their memory to run a model none of them could hold alone? Mesh LLM and the iroh peer-to-peer library are making that question serious.
The evening deep-dive. One concept, told properly. Listen along: Below the Ice on The Penguin Alley.
The headline: your device is not the limit
Every serious language model you can run locally today fits in your GPU's memory — barely, with quantization tricks and some sweating. Go one size up and the math stops working. A 70B parameter model at full precision needs around 140 GB of VRAM. No consumer GPU has that. No two of them together have that, unless you wire them through NVLink, a fabric that costs more than a car and lives in a data center.
So the question has always been: what if you could use the network as the memory bus?
That is exactly what Mesh LLM proposes. And tonight we go under the headline.
What it is
Mesh LLM is an open experiment by the team behind iroh — a production-grade Rust peer-to-peer networking library — that asks whether transformer inference can be distributed across a mesh of heterogeneous machines connected over regular internet. Not NVLink. Not InfiniBand. The real internet, with its latencies and packet loss and asymmetric upload speeds.
iroh is the networking layer underneath. Think of it as the plumbing: it handles hole-punching through NAT (the reason your laptop and your friend's server can't usually talk directly), provides relay fallback when direct connections fail, and offers a clean Rust API that makes peer-to-peer feel like passing messages rather than wrangling raw sockets. It was built by the team at n0 and has been maturing quietly for two years as a building block for decentralized applications.
Mesh LLM puts the two together: transformer layer splitting on top of iroh's p2p fabric.
How it actually works
To understand distributed inference, start with what a transformer model is, physically.
A transformer is a stack of layers. Each layer takes a tensor in — a large block of floating-point numbers representing the state of your text — does some matrix math, and hands a new tensor to the next layer. In a standard local run, all of this happens inside one GPU: the weights for every layer are loaded into VRAM at startup, and inference is a cascade of matrix multiplications flowing through them in sequence.
Pipeline parallelism is the first trick for splitting that across machines. You cut the model at layer boundaries: machine A holds layers 1 through 16, machine B holds layers 17 through 32. Machine A runs its chunk on an input prompt, produces an intermediate tensor, and sends it over the wire to machine B, which runs its half and returns a result. The analogy is a relay race: each runner carries only their segment, and the baton — the tensor — passes between them.
iroh handles the baton passing. When machine A finishes its segment, it uses iroh to open a direct connection to machine B (or a relayed one if firewalls intervene) and streams the tensor bytes across. Machine B deserializes, runs its layers, and the cycle continues token by token until the response is complete.
The latency arithmetic is where it gets interesting — and honest. A single forward pass through a 7B model might take 50ms on a decent local GPU. If you split it across two machines with 20ms round-trip latency between them, that relay adds roughly 20ms per layer boundary per token. For a 32-layer model split in half, you pay that cost on every generated token. At one token per second it is painful. At ten tokens per second it starts compounding badly.
The project does not hide this. The goal is not speed. The goal is reachability — running a model that no single machine could hold at all.
Why it matters now
Three forces converging right now make this more than a weekend experiment.
The HBM wall. High-bandwidth memory — the fast VRAM that makes GPU inference possible — is expensive, scarce, and physically constrained by chip packaging limits. Frontier models keep growing. The gap between what fits on a device you own and what frontier capability requires keeps widening. Pipeline parallelism over a mesh is one architectural response to that gap that does not require a hardware upgrade.
The cost of centralized cloud compute. Running a 70B parameter inference endpoint on a cloud provider costs real money at any scale. Aggregating spare capacity across commodity machines — home servers, idle workstations, modest VPS instances — is a fundamentally different cost structure. If the coordination overhead can be held low enough, the economics shift.
The decentralization of AI compute is a live research area. Projects like Petals, Hugging Face's earlier collaborative inference work, showed the concept was real at scale. Mesh LLM is a second-generation attempt with a cleaner, production-grade networking library underneath. The fact that it builds on iroh matters: iroh is already used in real applications, so the p2p plumbing is not the experimental part.
What is genuinely overhyped
Let's be honest about the ceiling.
Network bandwidth is not memory bandwidth. Inside a GPU server with NVLink, inter-GPU bandwidth runs at 600 to 900 GB/s. A home fiber connection runs at 1 to 2 Gbps — three to four orders of magnitude slower. Sending a large intermediate tensor between layers across the internet takes real time, and that time shows up in every token. This is not a footnote. It is the defining constraint of the entire approach, and no software engineering fixes it.
Heterogeneous hardware is harder than it looks. When one node is an NVIDIA A100 and another is an AMD GPU and a third is Apple Silicon, tensor formats, precision levels, and memory layouts diverge. Getting them to cooperate on a contiguous inference pipeline requires careful serialization and deserialization at each hop — and the compatibility surface is genuinely hard to maintain as the model ecosystem evolves.
Latency-sensitive applications will not move here. If you need fast, interactive inference — a coding assistant, a chat interface, real-time voice — centralized GPU endpoints are the answer. Mesh LLM is for the case where you are willing to trade latency for access: running a model that would otherwise simply be impossible.
What to watch
1. Whether node operators show up. Petals proved in 2022 that BitTorrent-style collaborative inference was possible. Mesh LLM is a second attempt with better foundations. The real test is whether a community of node operators forms the way Folding@home once did for scientific compute — people donating GPU time because the goal feels worth it.
2. iroh's broader adoption curve. Mesh LLM is as much a demonstration of iroh as it is a distributed inference system. If iroh's library picks up other real-world applications — distributed storage, agent communication, edge coordination — the networking fabric underneath Mesh LLM matures for free, without anyone working directly on the inference problem.
3. Tensor compression research. The main bottleneck is how many gigabytes of data need to travel between nodes per generated token. Smarter quantization, better pipeline scheduling, and tensor compression techniques under active research could meaningfully shrink that number. Watch the papers on efficient communication for distributed inference — they are the unlock that could make this genuinely fast on realistic hardware.
The Below the Ice podcast goes deeper on this — slow, story-driven, audio for builders winding down. Tonight's episode covers all of this and more.
Sources: