Widget 4 — Two ledgers: the courier's bill, before and after

Two ledgers — the courier's bill, before and after

Both ledgers were counted by hand in this article: standard attention hauls T_std = 4N² + 4Nd numbers (the hatched region is pure ghost traffic — S and P's four pointless trips), while the tiled machine hauls T_flash ≈ 4N²d²/M. Slide N and watch the gap. The quadratic never dies — it gets divided by the bench.

standard — T_std = 4N² (ghost, hatched) + 4Nd

FlashAttention — T_flash = 4N²d²/M + 2Nd

traffic ratio:

N (sequence length)

fixed: d = 64 M = 100,000 counts in numbers moved (multiply by 2 bytes for BF16 traffic) paper's receipt at GPT-2 scale: 40.3 GB vs 4.4 GB measured — our clean count brackets it from above.