Concept

Zipf — where it appears

A frequency distribution in which the rth most common item appears about proportionally to one over r, which is what makes heavy hitters exist. It is what makes heavy hitters exist, and a summary tested on anything else is being tested on a stream no deployment produces.

Named by 3 essays across 2 fields — each of them below, with the objects they name alongside it.

1,8103,6205,4297,239010,00020,00030,00040,000arrivals so farcount of key 1 in the last 4,096Misra-Gries, no clockblocks of Misra-Griesthe truth, and the ring buffera heavy hitter that stops · sampled every 5006,493 claimed, 0 true

The count that outlives its arrivals

A Misra-Gries counter holding six thousand is not a record of six thousand arrivals. It is a number that has been added to and taken from, and nothing in the structure says when any of it happened — so when the key stops arriving the counter stays, and goes on reporting a key with nothing in the window as the heaviest thing in it.

structures · Window
key 16,726exactkey 23,236exactkey 32,035exactkey 41,440± 2key 3109± 937key 42117± 937key 8402± 937key 14914± 937bracket, with the truth marked · widest 937 · 3 exactcash register · stationary Zipf · 32 countersthe smallest counter is 938

The counter that takes the smallest slot

Space-Saving keeps two numbers per key and they bracket the truth from both sides. On the twenty heaviest keys of a stream its mean error is a tenth of one arrival, against a hundred and ten for Misra-Gries at the same bits — and on the keys ranked past a hundred the ordering reverses.

structures · Sketch
01251025501002505000.00010.0010.010.1110100how far S's ranking is rotated from R'strue size ÷ estimatethe uniform estimate8 counters a side8 counters, using their silenceexactZipf 2, 8 counters a sideabove the line: the estimate is too low

A summary's silence is a bound, not a count

When both sides of a join are skewed towards the same values the textbook estimate is 399 times too low and the order it picks costs 244 times the better one; two counters a side repair it. Skew them towards different values and the same estimate becomes 254 times too high — and the counters have nothing to say, because no value is in both summaries. Using their silence instead repairs the middle of the sweep and makes the far end six times worse.

applied · Transfer

Named alongside it

The objects these essays reach for when they reach for this one.

Heavy hitterMisra–GriesSummaryMeasurementState bitsTrade offCardinality estimationCorrelationCount-MINExpiryGuaranteeHonest limit

All concepts