skillPublished 2026-07-24

Below the Ice — Temperature is Not Diversity

Running the same prompt 10 times at temperature 1.0 feels like getting diverse AI opinions. A paper published today proves it isn't — and the gap matters more than you think for multi-agent systems.

— views

🎙️ Listen to tonight's episode: Below the Ice — July 24

The headline: "just run it multiple times" has a ceiling nobody talks about

If you've built anything with LLMs in the last two years, you've probably used this pattern: run the same prompt a few times, take the most common answer, and call that more reliable than a single run. It's called self-consistency, and it works. What it doesn't do — and a paper published today on arXiv explains exactly why — is give you genuine diversity. We go below that.

What it is

"Stochastic Sampling is Epistemically Shallow" (arXiv:2607.20464) makes a precise and uncomfortable claim: temperature-based variation and model-based diversity are different phenomena living in different dimensions of output space. The paper asks a clean question — when a language model gives different answers on repeated runs, does that variation tell you anything beyond how hard that particular question is for that particular model? The answer, they argue, is mostly no.

How it actually works — from first principles

Start with what temperature does. A language model produces a probability distribution over possible next tokens at each step. A high temperature flattens that distribution — the model assigns more probability to alternatives it would normally ignore. A low temperature sharpens it — the model commits more aggressively to its top picks. More temperature means more randomness. More randomness means more varied outputs.

Here's the thing: all of those varied outputs still come from the same landscape of knowledge. The hills and valleys — what the model knows, what it doesn't, which concepts it has linked to which — remain unchanged. You're taking different random steps across the same terrain.

True model diversity is different terrain altogether. Two models trained on different datasets, with different architectures, or different fine-tuning objectives will each have their own landscape. Ask them both a hard question and they may each be confidently wrong — but in different directions. That disagreement is meaningful. It tells you something structural about the question itself.

The paper frames this as a dimensionality gap. Stochastic sampling varies outputs along a low-dimensional subspace — roughly, "how far from the most likely answer are we willing to wander?" Model diversity varies outputs along a high-dimensional subspace — the full spectrum of what a different training history might have encoded differently. Self-consistency samples are correlated with each other in a way that samples from diverse models aren't. Related questions tend to flip together within one model's temperature variance; across truly diverse models, the correlation structure is looser and more revealing.

Analogy: You need three people's opinions on a building's structural weakness. You could ask one structural engineer three times, letting them think out loud and varying their mood. That's temperature. Or you could ask a structural engineer, an architect who's seen buildings fail, and a contractor who's built fifty of them. That's model diversity. The first gives you three takes on the same expertise base. The second might reveal a problem none of them would have surfaced alone.

Why it matters now

Self-consistency, introduced by Wang et al. in 2022, became one of the most-cited prompting techniques in the field. Majority-vote over multiple samples improves accuracy on reasoning tasks meaningfully — on some benchmarks by 10-20 percentage points over a single sample. So the technique is real and useful. The paper doesn't dispute that.

What it disputes is what self-consistency is actually measuring. Its claim: self-consistency estimates per-question difficulty well — "is this question hard for this model?" — but misses cross-question structure — "are there categories of questions where this model systematically fails?" That second kind of uncertainty is what you need when you're building a multi-agent system and asking whether your agents will disagree in useful ways.

Multi-agent consensus systems built on one base model (different prompts, different temperatures, different personas) likely share the same structural blind spots. The diversity looks real from the outside — they give different answers — but the answers explore the same neighborhood of the model's knowledge. A hard question that one instance fails may be confidently answered by another instance, but the chance of being wrong is not as independent as it would be with genuinely different models.

This is a practical problem for anyone building:

Agentic test frameworks — running multiple agents over the same test suite and flagging disagreements as bugs
Multi-agent reasoning pipelines — using majority vote across agent outputs to improve answer quality
Uncertainty quantification — treating spread in temperature samples as a calibrated confidence signal

All three assume that more samples means more independent information. The paper's position is that the independence breaks down under the hood.

What is overhyped

The takeaway isn't "self-consistency doesn't work." It does work, for the thing it actually is: a variance reducer. If you have one model and you want a more reliable answer to a specific question, sampling multiple times and taking the majority is a legitimate improvement over a single sample. The improvement is real.

The overhyped version is the stronger claim — that majority voting over one model's temperature samples is a good substitute for having diverse models. It isn't. It's a cheap and effective variance reducer masquerading as diversity. The practical gap between the two mostly doesn't matter on standard benchmarks with clean right answers. It starts to matter in production, on questions that are genuinely uncertain, on tasks where the model has a systematic gap rather than random noise.

What to watch

Three directions follow from this:

1. Heterogeneous model routing. Instead of sampling the same model at high temperature, route uncertain queries to a different model — different pretraining, different fine-tuning, possibly different architecture. The routing signal can be cheap (a small classifier on the question embedding). The diversity gain is real.

2. Active uncertainty probing. Rather than sampling to estimate uncertainty, probe it directly — ask the model what it doesn't know, or use structured probes that test the boundary of its knowledge rather than sampling from its interior. The AINTMA paper from today's brief shows early work on agentic systems that adaptively probe quality boundaries rather than passively sampling.

3. Calibrated ensemble baselines. If you're shipping a multi-agent system and using self-consistency as the reliability story, you now have a reason to benchmark it against a two-model ensemble. It may surprise you. The computational overhead is higher; the independence gain may be worth it.

The deeper question the paper opens is what epistemic diversity actually means for LLMs — and whether we can measure it without training a completely separate model. That's an active research direction worth watching.

Sources:

Below the Ice is The Penguin Alley's evening deep-dive: one topic per night, told properly, for builders winding down. Listen on your podcast app.