Field

One pass, and no room

The data goes past once and there is not room to keep it. What survives is a summary of a few hundred bits, and the answer it gives is wrong — the whole design is a choice about how wrong, and every structure here buys accuracy with bits at a rate that can be measured.
1,00010,0000.1bits of state heldrelative errorHyperLogLog (-0.49)LogLog (-0.44)bottom-k (-0.38)50,000 distinct keys · 14 runs per point · truth counted exactlybest: 2.00% at 10,240 bits

The answer that is allowed to be wrong

What changes when an algorithm is not allowed to keep its input, why the answer stops being exact, and how the trade between bits of state and size of error is measured rather than asserted.

one summary, k = 32769one summary, k = 25608 summaries of 32, merged536worst error over the top keys, in arrivalsconcentration 1.00 — the mean share of a heavy key held by one shard8 shards · hashed · balancedmerge 536 against matched 0

The state a merge is standing in for

Why comparing a merged summary against a single small one is the wrong comparison, what the right one costs, and how much accuracy is given up for never having the whole stream in one place.

0.1bits the register neededrelative errora = 2a = 1.5a = 1.2a = 1.1a = 1.05a = 1.02√((a−1)/2), predicted200 runs per base · n = 20,000 · exact counter needs 15 bits72.6% at 5 bits

Counting past what the register holds

The oldest sketch there is, measured — why the estimate is unbiased, what the register actually costs, and how the base sets the exchange rate between bits and error.

313roundconc 0.14536hashedconc 1.00309blockedconc 0.14how the arrivals were partitionedworst error over the top keysone summary, k = 32one summary, k = 256the merge of 8Space-Saving · stationary Zipf · 40,000 arrivals8 shards

The partition the analysis did not mention

How a stream is split across machines is not in any bound these structures carry, and it moves the merged error by a factor of six in opposite directions for two structures that are otherwise identical.

hash rowh1h2h3h4h1 → cell 30: 10,433h2 → cell 29: 10,624h3 → cell 19: 10,575h4 → cell 4: 10,895key 0 occurred 9,579 times · the minimum of the four is 10,433 · over by 854the additive bound at this width is e/w × N = 5,0974×32 counters · 4,096 bits · Zipf s = 1.1exact would take 162,288 bits

A count that is never under

How a frequency sketch works, why the minimum rather than the mean, what the additive bound actually promises, and what a published improvement gives away to get a better average.

01325leading-zero rank keptregister, 0 to 255estimate 7,107truth 7,368error -3.54%0 still empty256 registers × 5 bits = 1,280 bitspredicted ±6.50%

A count read off the leading zeros

How counting distinct things becomes a question about the most surprising hash, why one register is useless, and what stochastic averaging buys that more hash functions do not.

16326412825610244096window length W, in arrivals (D = W × the mean gap)share of the counts they disagree onevenPoissonburstydrifting1 ms clock · mean gap 10 ticks · 30,000 arrivalseven: 0% at every length

A window that is a duration

What changes when a sliding window is measured in seconds rather than arrivals — the occupancy stops being a constant, the stamp width stops depending on the rate, and every per-item bound acquires a precondition.

exact-11.2%-3.3%0.0%3.3%11.2%rmse 3.61%worst 9.71%23 of 60outside the band5,120 bits · 60 seeds · relative error of one runpredicted ±3.25%

The correction that makes it work

What the harmonic mean buys over the arithmetic one on identical state, why the raw estimator is biased at low cardinality, and what a correction that switches estimators is actually doing.

0.010.10.11Jaccard index of the two setsrelative error of the intersectionas wrong as the answerinclusion–exclusionthe minimum valuesHyperLogLog p = 12, 20,480 bits · bottom-4,096, 131,072 bitstruth from a hash map over 200,000 keys

The error of a difference

An intersection of two 100,000-key sets estimated by inclusion–exclusion over HyperLogLog sketches and by the k smallest hash values, swept over Jaccard indices from 0.82 to 0.005. Subtracting keeps a fixed error as a share of the union and so loses all its relative accuracy; sampling the union does not, and at equal state the two cross at a Jaccard index of about 0.05 — which itself moves with how many bits are spent.

+1−1keys, most frequent firstΣ s(x)·f(x) = 9,034squared: 81,613,156true F2: 36,931,352121.0% outa polynomial of degree 4 · cash-register model · counts exactone register, 32 bits

The estimate that squares the stream

The second frequency moment by the tug-of-war estimator, why one register is hopeless and unbiased at the same time, and what averaging and a median each buy.

answered 19.90%25%50%75%100%0.36420.52,170value, logarithmicfraction of the stream at or belowrank ±2%value 19.3–21.9answered 2.9% out20,000 values · log-normal, σ = 1.2 — a latency distribution · Greenwald–Khanna, ε = 0.02rank 1.02% · value 2.9%

The error that is on the rank

Greenwald–Khanna quantile summaries — the invariant, the measured size against the bound, and why a deterministic structure beats a uniform sample of the same bits on the promise it makes.

decayedwindowed060120secondscount attributed to key 0half-life 4 s · window 5.77 s · burstythe same key, two questions

The counter with no window in it

What an exponentially decayed counter answers, how it relates to a sliding window and where the relation stops, and why the family with no W in it is the one monitoring systems actually run.

3,600 — the window opens4,000 — now32161616884the window edge215 in buckets−16 for the oldest= 200true 2000.3% outsliding-window model · ε = 0.2, k = 5 · 1,764 merges406 bits against 400

The summary that has to forget

The sliding-window model and the exponential histogram — buckets of powers of two, why the entire uncertainty is one bucket, and a half-count that is wrong when the window is nearly empty.

windowed HyperLogLog4,088 bitsone stamp per live key8,775 bitsblocks of Misra-Gries18,549 bitsthe last W keys, kept131,072 bitskeysstampspayloadindexthe parts, in the order they stackwindow 4,096 · the popular keys drift131,072 bits at most

The bits that say when

The eleventh counter — bits spent on knowing when an item arrived, measured apart from bits spent on knowing what it was, across four structures that answer the same question over a sliding window.

cash register — every key49,952 bitscash register — HyperLogLog2,560 bitswindow — a stamp per live key8,505 bitswindow — HyperLogLog5,494 bits40,000 arrivals · the popular keys drift, so old keys are gone rather than rare · W = 4,0961,561 distinct in the stream · 169 in the window20× against 1.5×

A register that became a list

Counting distinct items over a sliding window — why a maximum cannot be one number when the arrivals expire, what a list of future possible maxima costs, and why approximation buys twenty times less here than it does over a stream.

0.1%1%10%100%0.50.750.90.990.999quantile asked forrank error ÷ (1 − q)Greenwald–Khanna7,080 bitshigh-biased56,064 bitst-digest5,952 bits8 streams per point · ε = 0.01, δ = 100denominator: the tail

An error measured against the answer

Relative-error quantile summaries — why an ε on the whole stream is the wrong promise at a tail percentile, what a cap that shrinks with the tail costs, and why a rank error carries no information about the values at all.

shard 1 · 120 → 119shard 2 · 121 → 120shard 3 · 120 → 120shard 4 · 121 → 120shard 5 · 119 → 118shard 6 · 124 → 123shard 7 · 121 → 120shard 8 · 122 → 121floor, in arrivalsnaive: tail ÷ kfixed pointmeasuredstationary Zipf · round · k = 321.006× the measured floor

The floor a histogram already knows

How the smallest counter a bounded-space summary settles at can be predicted from the arrival frequencies alone, why dividing the tail mass by the width is not the answer, and what the fixed point is worth.

1,00010,000248163264ε = 0.02, α = 0.58ε = 0.01, α = 0.56ε = 0.005, α = 0.54tuples keptshards mergedlog-normal, σ = 1.2 — a latency distribution · 20,000 valuesα 0.58 / 0.56 / 0.54 · worst residual 2.0%

The tuples a merge does not give back

How the state of a merged quantile summary grows with the number of shards, why compression cannot undo a merge, and what the fitted exponent depends on.

even5.1 s – 5.1 s1.00× · dispersion 0.00Poisson4.7 s – 5.5 s1.16× · dispersion 0.70bursty4.5 s – 4.5 s1.00× · dispersion 0.57drifting565 ms – 14.8 s26.14× · dispersion 849.57duration one block covered4,096 arrivals in 8 blocks · 100 Hz · 1 ms clock26.1× on the drifting stream

The window that is even in the wrong currency

What an arrival-counted window promises about time, measured — and why the unevenness a time window shows in its occupancy is the same unevenness on the other axis.

10010³10⁴10⁵10⁶⌊1/2ε⌋ = 50151025501002005001,000tuples, and tuples examinedcompression period, in updatestuples examinedpeak tuplesresident tuplesworst rank errorε = 0.01 · 20,000 arrivalspeak 10× · work 72× · answer 1.21×

The period that is not a promise

What the compression period of a quantile summary actually decides, why it is hooked to the error parameter, and what a deployment is choosing when it tunes ε.

00.2500.5000.7501010203040age of an arrival, secondsweight it carries nowexponential, half-life 4.0 sbackward polynomial, α = 2, scale 4.0 sforward, β = 2, landmark 20 s backforward, β = 2, landmark 160 s backdotted: half weightweights at the moment of reading

A decay measured from where it started

Exponential, backward polynomial and forward polynomial decay compared on the weights they give, the age at which an arrival counts half, and a Poisson stream whose rate rises fourfold: how long each takes to follow the rise as the stream ages, what restarting the landmark does, and where the three sit on the trade between noise and speed.

All essays