Field

One pass, and no room

The data goes past once and there is not room to keep it. What survives is a summary of a few hundred bits, and the answer it gives is wrong — the whole design is a choice about how wrong, and every structure here buys accuracy with bits at a rate that can be measured.
1,00010,0000.1bits of state heldrelative errorHyperLogLog (-0.49)LogLog (-0.44)bottom-k (-0.38)50,000 distinct keys · 14 runs per point · truth counted exactlybest: 2.00% at 10,240 bits

The answer that is allowed to be wrong

What changes when an algorithm is not allowed to keep its input, why the answer stops being exact, and how the trade between bits of state and size of error is measured rather than asserted.

one summary, k = 32769one summary, k = 25608 summaries of 32, merged536worst error over the top keys, in arrivalsconcentration 1.00 — the mean share of a heavy key held by one shard8 shards · hashed · balancedmerge 536 against matched 0

The state a merge is standing in for

Why comparing a merged summary against a single small one is the wrong comparison, what the right one costs, and how much accuracy is given up for never having the whole stream in one place.

0.1bits the register neededrelative errora = 2a = 1.5a = 1.2a = 1.1a = 1.05a = 1.02√((a−1)/2), predicted200 runs per base · n = 20,000 · exact counter needs 15 bits72.6% at 5 bits

Counting past what the register holds

The oldest sketch there is, measured — why the estimate is unbiased, what the register actually costs, and how the base sets the exchange rate between bits and error.

313roundconc 0.14536hashedconc 1.00309blockedconc 0.14how the arrivals were partitionedworst error over the top keysone summary, k = 32one summary, k = 256the merge of 8Space-Saving · stationary Zipf · 40,000 arrivals8 shards

The partition the analysis did not mention

How a stream is split across machines is not in any bound these structures carry, and it moves the merged error by a factor of six in opposite directions for two structures that are otherwise identical.

hash rowh1h2h3h4h1 → cell 30: 10,433h2 → cell 29: 10,624h3 → cell 19: 10,575h4 → cell 4: 10,895key 0 occurred 9,579 times · the minimum of the four is 10,433 · over by 854the additive bound at this width is e/w × N = 5,0974×32 counters · 4,096 bits · Zipf s = 1.1exact would take 162,288 bits

A count that is never under

How a frequency sketch works, why the minimum rather than the mean, what the additive bound actually promises, and what a published improvement gives away to get a better average.

01325leading-zero rank keptregister, 0 to 255estimate 7,107truth 7,368error -3.54%0 still empty256 registers × 5 bits = 1,280 bitspredicted ±6.50%

A count read off the leading zeros

How counting distinct things becomes a question about the most surprising hash, why one register is useless, and what stochastic averaging buys that more hash functions do not.

16326412825610244096window length W, in arrivals (D = W × the mean gap)share of the counts they disagree onevenPoissonburstydrifting1 ms clock · mean gap 10 ticks · 30,000 arrivalseven: 0% at every length

A window that is a duration

What changes when a sliding window is measured in seconds rather than arrivals — the occupancy stops being a constant, the stamp width stops depending on the rate, and every per-item bound acquires a precondition.

exact-11.2%-3.3%0.0%3.3%11.2%rmse 3.61%worst 9.71%23 of 60outside the band5,120 bits · 60 seeds · relative error of one runpredicted ±3.25%

The correction that makes it work

What the harmonic mean buys over the arithmetic one on identical state, why the raw estimator is biased at low cardinality, and what a correction that switches estimators is actually doing.

0.010.10.11Jaccard index of the two setsrelative error of the intersectionas wrong as the answerinclusion–exclusionthe minimum valuesHyperLogLog p = 12, 20,480 bits · bottom-4,096, 131,072 bitstruth from a hash map over 200,000 keys

The error of a difference

An intersection of two 100,000-key sets estimated by inclusion–exclusion over HyperLogLog sketches and by the k smallest hash values, swept over Jaccard indices from 0.82 to 0.005. Subtracting keeps a fixed error as a share of the union and so loses all its relative accuracy; sampling the union does not, and at equal state the two cross at a Jaccard index of about 0.05 — which itself moves with how many bits are spent.

+1−1keys, most frequent firstΣ s(x)·f(x) = 9,034squared: 81,613,156true F2: 36,931,352121.0% outa polynomial of degree 4 · cash-register model · counts exactone register, 32 bits

The estimate that squares the stream

The second frequency moment by the tug-of-war estimator, why one register is hopeless and unbiased at the same time, and what averaging and a median each buy.

01/82/83/84/85/86/87/811×1.5×2×3×5×10×share of the bits given to the HyperLogLogworst error over all overlaps ÷ the best possible1.81×switch at a crossingmeasured in advanceswitch on the textbookvariancesweight by the textbookvariances40,960 bits a set, 64 seeds, J from 0.98 to 0.001ringed: the minimax split

A budget split before the question arrives

A fixed budget of 40,960 bits a set split in eighths between a HyperLogLog and a bottom-k sample, measured against an oracle that knew the overlap: each split's worst regret over Jaccard indices from 0.98 to 0.001, three rules for choosing the estimate at query time, why the textbook variances choose wrongly, where the crossing sits for each split, how the minimax split moves with the smallest overlap a system must answer, and what changes when one set is ten times the other.

answered 19.90%25%50%75%100%0.36420.52,170value, logarithmicfraction of the stream at or belowrank ±2%value 19.3–21.9answered 2.9% out20,000 values · log-normal, σ = 1.2 — a latency distribution · Greenwald–Khanna, ε = 0.02rank 1.02% · value 2.9%

The error that is on the rank

Greenwald–Khanna quantile summaries — the invariant, the measured size against the bound, and why a deterministic structure beats a uniform sample of the same bits on the promise it makes.

decayedwindowed060120secondscount attributed to key 0half-life 4 s · window 5.77 s · burstythe same key, two questions

The counter with no window in it

What an exponentially decayed counter answers, how it relates to a sliding window and where the relation stops, and why the family with no W in it is the one monitoring systems actually run.

3,600 — the window opens4,000 — now32161616884the window edge215 in buckets−16 for the oldest= 200true 2000.3% outsliding-window model · ε = 0.2, k = 5 · 1,764 merges406 bits against 400

The summary that has to forget

The sliding-window model and the exponential histogram — buckets of powers of two, why the entire uncertainty is one bucket, and a half-count that is wrong when the window is nearly empty.

windowed HyperLogLog4,088 bitsone stamp per live key8,775 bitsblocks of Misra-Gries18,549 bitsthe last W keys, kept131,072 bitskeysstampspayloadindexthe parts, in the order they stackwindow 4,096 · the popular keys drift131,072 bits at most

The bits that say when

The eleventh counter — bits spent on knowing when an item arrived, measured apart from bits spent on knowing what it was, across four structures that answer the same question over a sliding window.

cash register — every key49,952 bitscash register — HyperLogLog2,560 bitswindow — a stamp per live key8,505 bitswindow — HyperLogLog5,494 bits40,000 arrivals · the popular keys drift, so old keys are gone rather than rare · W = 4,0961,561 distinct in the stream · 169 in the window20× against 1.5×

A register that became a list

Counting distinct items over a sliding window — why a maximum cannot be one number when the arrivals expire, what a list of future possible maxima costs, and why approximation buys twenty times less here than it does over a stream.

0.1%1%10%100%0.50.750.90.990.999quantile asked forrank error ÷ (1 − q)Greenwald–Khanna7,080 bitshigh-biased56,064 bitst-digest5,952 bits8 streams per point · ε = 0.01, δ = 100denominator: the tail

An error measured against the answer

Relative-error quantile summaries — why an ε on the whole stream is the wrong promise at a tail percentile, what a cap that shrinks with the tail costs, and why a rank error carries no information about the values at all.

shard 1 · 120 → 119shard 2 · 121 → 120shard 3 · 120 → 120shard 4 · 121 → 120shard 5 · 119 → 118shard 6 · 124 → 123shard 7 · 121 → 120shard 8 · 122 → 121floor, in arrivalsnaive: tail ÷ kfixed pointmeasuredstationary Zipf · round · k = 321.006× the measured floor

The floor a histogram already knows

How the smallest counter a bounded-space summary settles at can be predicted from the arrival frequencies alone, why dividing the tail mass by the width is not the answer, and what the fixed point is worth.

1,00010,000248163264ε = 0.02, α = 0.58ε = 0.01, α = 0.56ε = 0.005, α = 0.54tuples keptshards mergedlog-normal, σ = 1.2 — a latency distribution · 20,000 valuesα 0.58 / 0.56 / 0.54 · worst residual 2.0%

The tuples a merge does not give back

How the state of a merged quantile summary grows with the number of shards, why compression cannot undo a merge, and what the fitted exponent depends on.

even5.1 s – 5.1 s1.00× · dispersion 0.00Poisson4.7 s – 5.5 s1.16× · dispersion 0.70bursty4.5 s – 4.5 s1.00× · dispersion 0.57drifting565 ms – 14.8 s26.14× · dispersion 849.57duration one block covered4,096 arrivals in 8 blocks · 100 Hz · 1 ms clock26.1× on the drifting stream

The window that is even in the wrong currency

What an arrival-counted window promises about time, measured — and why the unevenness a time window shows in its occupancy is the same unevenness on the other axis.

10010³10⁴10⁵10⁶⌊1/2ε⌋ = 50151025501002005001,000tuples, and tuples examinedcompression period, in updatestuples examinedpeak tuplesresident tuplesworst rank errorε = 0.01 · 20,000 arrivalspeak 10× · work 72× · answer 1.21×

The period that is not a promise

What the compression period of a quantile summary actually decides, why it is hooked to the error parameter, and what a deployment is choosing when it tunes ε.

00.2500.5000.7501010203040age of an arrival, secondsweight it carries nowexponential, half-life 4.0 sbackward polynomial, α = 2, scale 4.0 sforward, β = 2, landmark 20 s backforward, β = 2, landmark 160 s backdotted: half weightweights at the moment of reading

A decay measured from where it started

Exponential, backward polynomial and forward polynomial decay compared on the weights they give, the age at which an arrival counts half, and a Poisson stream whose rate rises fourfold: how long each takes to follow the rise as the stream ages, what restarting the landmark does, and where the three sit on the trade between noise and speed.

010200100200seconds since the burstestimated rate, arrivals a secondthe burstexponentialexact polynomialthree exponentialsrestarted forwardfaint: the same streamswith the burst removedmatched at 5.0% spread · burst of 300 in one secondthe same event, four descents

What a heavier tail actually buys

Exponential, backward polynomial, a fitted sum of three exponentials and a restarted forward counter, matched at five per cent spread and then asked to detect one burst at a fixed false-alarm rate: the detection horizon at six burst sizes, the growth law each shape obeys, the exponent checked against 1/α at four values of α, and what reaching further costs to store.

0.313103000.250.50.7511.251.52Zipf exponent s of the stream — 0 is flatvariance of one copy ÷ moment²sampling, F4sampling, F3sampling, F2signs, F22,048 keys, 60,000 items, exactdashed: tug-of-war

The moments past the second

The sampling estimator for frequency moments Fₖ past the second, with its variance computed exactly over every position: the telescoping that makes it unbiased, what individual positions report, the variance against the stream's skew, the guarantee k·n^(1−1/k) and the stream that comes within a constant of it, copies needed for ten per cent, and drawn means against the exact spread.

3009003,0009,00010100arrivals in the bursthalf-detection horizon, sthe oracle's thresholdscale held, R = 64scale held, R = 16scale learns, R = 64scale learns, R = 16ten arrivals a second, sixty streamsdashed: R = 16

A detector that learns its own quiet

A self-calibrating burst detector on Poisson streams of ten arrivals a second — a fast exponential counter matched to 5% spread, a slow one at R times its half-life, and an exponentially weighted mean square of their difference as the scale — against the fast counter with an oracle threshold from sixty independent streams: the half-detection horizon at four burst sizes, false alarms on a still stream at five values of R, one stream traced through a burst, false alarms under a steady rise in the background, and detection against lag.

5%10%20%40%rise in the background rate every 1,000 sreadings that alarm0%5%20%40%60%scale heldalarm clockpositive clockrebase on the positive clockscale learnsR = 16, 24 streams a pointdashed: 5%

The clock a burst cannot outlast

Three schedules for the scale of a self-calibrating burst detector on Poisson streams of ten arrivals a second — release after the alarm has lasted L*, release after the fast-less-slow difference has stayed positive for L*, and resetting the slow counter at that point — measured against the held and learning scales: alarms on linearly rising backgrounds at two ratios of half-lives, the half-detection horizon at four burst sizes, both clocks traced on one stream, alarms against the drift's offset in spreads of the difference, and the rebasing design traced through a steep rise.

00.250.50.7511.5210010³Zipf exponent of the streamwords for 10% on F3sample every positionheavy keys first60,000 arrivals, 2,048 keysa word: a key or a count

The keys worth counting before sampling

A one-pass estimator for F3 that keeps a Count-Sketch and a list of candidate heavy keys beside the sampling estimator's copies, measured on 60,000-arrival streams over 2,048 keys at Zipf exponents from 0 to 2 and on the one-heavy-key stream at 128 to 16,384 keys: the share of F3 the top keys hold, words for 10% against the number of candidates in two ways of reading a candidate's frequency, the parts of the bill against the sketch's width, words against the exponent, and words against the size of the worst stream.

0.310.621.242.484.859.7the rise's offset in d, in spreads of dreadings that alarm0%5%25%50%75%100%held while alarmedthe positive clockcorrected for the slopesolid: R = 16; dashed: R = 64dashed rule: a still stream's 5%

The slope the slow counter already shows

The two-counter burst detector at R = 16 and 64 on Poisson backgrounds rising 5 to 40% every 1,000 seconds, with the rise's predicted offset subtracted from the counters' difference — the slow counter's change over one slow half-life, decayed with half-lives of one, four and sixteen slow half-lives, times (R − 1)H/ln 2 — against the held scale and the positive clock: alarms against the rise's offset in spreads of the difference, one stream's corrected difference, the detection horizon after bursts of 300 to 9,000 arrivals, the estimate's half-life, and how long it takes to catch up.

-2-1012305001e+3seconds from the end of the risemean difference, in spreads of dthe rise endsuncorrected, 20% risecorrected, 20% riseuncorrected, 40% risecorrected, 40% risemean of 24 streams, R = 16zero: the background's own level

The rise the correction kept subtracting

The two-counter burst detector at R = 16 on Poisson backgrounds that rise by 20% or 40% every 1,000 seconds and then hold level, with three ways of setting its threshold — a scale held while alarmed, the positive clock, and the held scale with the rise's offset subtracted — every point averaged over 24 to 60 streams: the mean corrected difference around the end of the rise, the alarms each design raises on the level stretch, the share of bursts of 20 and 30 arrivals caught at stated lags after the rise ends, and how long a burst of 300 stays visible.

501002005001,0002,0005,000Zipf exponent of the stream (0 is flat)words for 10% on F3 (log scale)00.250.50.7511.52sample every positionshare 3%, list of 64share 1%, list of 8the best fixed list60,000 arrivals, 2,048 keysa word: a key or a count

A share of a list of equals

The third frequency moment of 60,000 arrivals over 2,048 keys at Zipf exponents 0 to 2, estimated to a relative standard deviation of 10% by a Count-Sketch of depth five, a list of candidates counted from entry, and copies of the sampling estimator: words against the exponent for the plain sampler, the best fixed list, and lists of 8 to 64 whose candidates are counted only above 1%, 3% or 10% of the list's own moment; candidates kept; words over the best on each exponent for each list length; each candidate's share of its list on one pass, flat against Zipf 1; words by part; and words against the list's length at Zipf 0.5.

All essays