The ratio d²/M: flash traffic over standard's crime term, derived in §7. Below 1, the tiled machine wins; above 1, it hauls MORE than the crime it replaced. The wall sits at d = √M ≈ 316 — and every real head dimension lives far to its left. That is not luck: the multi-head split keeps d small, and the bench amortization is the reward. Hover or tap the curve to read any d.
hover the curve — or press ←/→
Markers, all verified against configs and vendor docs: GPT-2 heads (d = 64, ratio 0.04), Llama-class (d = 128, ratio 0.16), the Gemma family (d = 256, ratio 0.66 — flirting with the wall), the wall itself (d = √M ≈ 316), and Gemma 4's global layers (d = 512, ratio 2.6) — which crossed it in March 2026 and promptly lost FlashAttention support entirely: every FA kernel caps at head_dim 256, so those layers fall back to slower attention. The wall we derived with a pencil is roughly where the real ecosystem stops building kernels. Far right: the one-fat-head fantasy (d = 3072, ratio ≈ 94). Multi-head is FlashAttention's secret ally.