Concept

Timestamp — where it appears

A recorded arrival position, which is what lets a structure decide that an item has left a window, and which costs bits nothing else pays for. It costs bits nothing else pays for, and in a windowed summary those bits are frequently more than half the structure.

Named by 10 essays across 5 fields — each of them below, with the objects they name alongside it.

16326412825610244096window length W, in arrivals (D = W × the mean gap)share of the counts they disagree onevenPoissonburstydrifting1 ms clock · mean gap 10 ticks · 30,000 arrivalseven: 0% at every length

A window that is a duration

Nobody asks for the error rate over the last four thousand and ninety-six requests. They ask for the last five minutes. The two are the same question exactly when the arrivals are evenly spaced, and on a stream whose rate drifts they disagree about fifty-seven per cent of the counts.

streaming · Window
71 s60 ticks11100 ms600 ticks1410 ms6,000 ticks171 ms60,000 ticks270.001 ms60,000,000 ticksclock resolutionbits per stamp⌈log₂ 2D/r⌉no arrival rateappears in itD = 60 s · key 32 bitscomputed, not measured

The clock that cannot see the burst

A stream generator asked for a burst ten times faster than its mean rate, on a clock whose resolution was the mean gap, produced a perfectly even stream — index of dispersion 0.00, for something called bursty. Nothing had gone wrong except that the instrument could not represent what it was being asked to measure.

space · Window
decayedwindowed060120secondscount attributed to key 0half-life 4 s · window 5.77 s · burstythe same key, two questions

The counter with no window in it

A counter that fades by half every H settles, on a steady stream, at exactly the count of a window of 1.44H. That correspondence holds in the mean, on a steady stream, and nowhere else — and it is the reason a decayed counter is not an estimate of a windowed count for any window.

streaming · Decay
windowed HyperLogLog4,088 bitsone stamp per live key8,775 bitsblocks of Misra-Gries18,549 bitsthe last W keys, kept131,072 bitskeysstampspayloadindexthe parts, in the order they stackwindow 4,096 · the popular keys drift131,072 bits at most

The bits that say when

A windowed cardinality estimator holds 4,592 bits and 2,392 of them are clocks. Every summary in this collection has reported its size from the shape of its own structure, and not one of those numbers has ever been asked what the bits were for — so the resource that half of these structures spend most of their state on has been invisible while being counted.

streaming · Window
fades a lazy table computed382,976fades an eager table would compute3,830,256fade multiplications (log scale)11,968 evictions, each comparing all 32 counters at one instant32 counters · half-life 4 s · bursty10× fewer

The fading nobody computes

Twelve thousand arrivals into thirty-two decayed counters cost 382,976 fade multiplications. An implementation that aged every counter on every tick would have cost 3,830,256, and the ratio is exactly the mean gap between arrivals — not a coincidence, and the reason the family is deployable.

practice · Decay
cash register — every key49,952 bitscash register — HyperLogLog2,560 bitswindow — a stamp per live key8,505 bitswindow — HyperLogLog5,494 bits40,000 arrivals · the popular keys drift, so old keys are gone rather than rare · W = 4,0961,561 distinct in the stream · 169 in the window20× against 1.5×

A register that became a list

HyperLogLog replaces a key per distinct item with a five-bit register, and over a whole stream that is a saving of a hundred times. Ask it about the last four thousand arrivals instead and the same comparison against the same exact structure comes out at five. The estimator did not get worse. The exact answer got cheap.

streaming · Window
read from the start40,000 readsa ring of 4,096 keys and 4,096 stamps184,320 bitsread from the end4,096 reads792 counters, no stamps50,688 bitsitems readsame answer: 1×700, 2×341, 3×224…W = 4,096 · φ = 0.02 · stationary Zipf9.8× fewer reads, same answer

The pass that runs the other way

Exact heavy hitters over the last 4,096 of 40,000 arrivals cost 40,000 reads and a ring of 4,096 keys and stamps read forwards, and 4,096 reads with no stamps read backwards. Every lower bound in the sliding-window model is a bound about an access pattern, and the word doing the work never appears in the statement.

floors · Pass
1,8103,6205,4297,239010,00020,00030,00040,000arrivals so farcount of key 1 in the last 4,096Misra-Gries, no clockblocks of Misra-Griesthe truth, and the ring buffera heavy hitter that stops · sampled every 5006,493 claimed, 0 true

The count that outlives its arrivals

A Misra-Gries counter holding six thousand is not a record of six thousand arrivals. It is a number that has been added to and taken from, and nothing in the structure says when any of it happened — so when the key stops arriving the counter stays, and goes on reporting a key with nothing in the window as the heaviest thing in it.

structures · Window
even5.1 s – 5.1 s1.00× · dispersion 0.00Poisson4.7 s – 5.5 s1.16× · dispersion 0.70bursty4.5 s – 4.5 s1.00× · dispersion 0.57drifting565 ms – 14.8 s26.14× · dispersion 849.57duration one block covered4,096 arrivals in 8 blocks · 100 Hz · 1 ms clock26.1× on the drifting stream

The window that is even in the wrong currency

A window of four thousand arrivals in eight blocks retires a block every 5.1 seconds on a steady stream and anywhere between 0.57 and 14.8 seconds on a stream whose rate moves. The structure cannot tell, because it is counting arrivals, and the alert written against it is in seconds.

streaming · Window
00.2500.5000.7501010203040age of an arrival, secondsweight it carries nowexponential, half-life 4.0 sbackward polynomial, α = 2, scale 4.0 sforward, β = 2, landmark 20 s backforward, β = 2, landmark 160 s backdotted: half weightweights at the moment of reading

A decay measured from where it started

An exponentially decayed counter is one number because its weights fade by elapsed time alone. Forward decay keeps a polynomial weight in one number too, by measuring each arrival from a fixed landmark. Its memory is then a share of the time since that landmark: at β = 2 an arrival counts half at 29% of that time. Anchored at the start of a stream, it takes 5.3 seconds to register a fourfold rise twenty seconds in and 83 seconds when the rise comes at five minutes.

streaming · Decay

Named alongside it

The objects these essays reach for when they reach for this one.

State bitsSliding windowStreaming modelArrival processMeasurementTrade offExpiryHonest limitTime windowBurstinessClock resolutionExponential decay

All concepts