Series

Pass — the series

4 essays on one idea, from the one that introduces it to the one that assumes the rest.
  1. above 0.01 of the stream · 13 really areone pass: 18 candidatestwo passes: 13 keys, exactly the heavy ones123456789101112131415161718128 counters · carry 1,376 bits5 of 18 spurious

    The pass that was never a parameter

    One pass is the streaming model's defining restriction, and this collection has enforced it by making a second read throw. A prohibition cannot be swept. Turn it into a dial and the first thing it says is that a second pass turns eighteen candidate heavy hitters, six of them wrong, into twelve that are exactly right — for 3,680 bits carried across the boundary and no extra space at all.

    part 1 · bounds
  2. 10³10⁴10⁵10⁶passes over the datapeak bits of stateone pass, exact: 1,048,576 bitsradix, uniformsample, uniformradix, paretosample, paretoevery point exact · 32,768 values320 bits at best

    What a second pass buys

    Exact selection of a median from thirty-two thousand values needs the whole stream in one pass — a million bits — and eleven thousand in two. By nine passes it is three hundred and twenty. The state falls as n to the power one over p, which is a law with an exponent worth fitting, and on skewed data the deterministic rule misses it by three orders of magnitude.

    part 2 · space
  3. universe 0…11 · prefixes of 6 · candidate keeps 9 bitsthe two prefixes that collideA01234567891011B01234567891011first differencesame state — the candidate stores "7" after boththen both read the same suffix -1, -2, 12, 13, 14true median of A3true median of B4and one answer for both924 prefixes · floor ⌈log₂ C(12,6)⌉ = 10 bits10 bits collide on none

    A floor one pass cannot get under

    An exact one-pass selector must reach a different memory state for every prefix it might have read, and the pigeonhole that proves it is small enough to perform — nine hundred and twenty-four prefixes through a nine-bit state, the collision produced, the suffix that separates it, and two true medians it cannot both return. Ten bits collide on none, so the bound is exact — and a second pass walks under it by a factor of ninety.

    part 3 · floors
  4. read from the start40,000 readsa ring of 4,096 keys and 4,096 stamps184,320 bitsread from the end4,096 reads792 counters, no stamps50,688 bitsitems readsame answer: 1×700, 2×341, 3×224…W = 4,096 · φ = 0.02 · stationary Zipf9.8× fewer reads, same answer

    The pass that runs the other way

    Exact heavy hitters over the last 4,096 of 40,000 arrivals cost 40,000 reads and a ring of 4,096 keys and stamps read forwards, and 4,096 reads with no stamps read backwards. Every lower bound in the sliding-window model is a bound about an access pattern, and the word doing the work never appears in the statement.

    part 4 · floors

All series