Count-Sketch — where it appears
Named by 5 essays across 3 fields — each of them below, with the objects they name alongside it.
When the stream takes it back
Count-Min's estimate is never below the truth. That is a theorem about a stream where every update adds — and allow deletions that can take a count below zero and it comes back under on 93% of queries, with nothing in the number to say so.
A sketch that is allowed to be under
Count-Min's estimate is never below the truth, and it pays for that with an error proportional to the whole stream. Give every key a sign and take a median instead, and the same table is 2.7 times more accurate on the keys anybody asks about — and wrong in both directions.
The keys worth counting before sampling
The sampling estimator for the third frequency moment spends most of its copies on keys that barely contribute to the answer. Finding the heavy keys first with a Count-Sketch, counting them, and sampling only the rest cuts the state for ten per cent from 5,260 words to about 1,500 on a moderately skewed stream, and from 23,598 to under a thousand on the stream built to be worst for the sampler. It costs five times more than it saves on a flat stream, where there is no heavy key to find. What decides it is how the frequencies are shaped, not how long the stream is, and the frequencies must be counted, not read off the sketch, because the sketch's error on a heavy key is cubed.
A share of a list of equals
Estimating the third frequency moment by finding the heavy keys first and sampling the rest works best with one counted candidate on steep streams and sixteen at Zipf 0.5, and a system that does not know its stream's skew has to pick. Keeping a long list and counting a candidate only if it holds a stated share of the list's own moment was predicted to come within a few per cent of the best fixed list on every exponent. The share does choose: at 3% it keeps one candidate at Zipf 2, three at Zipf 1, five at Zipf 0.5. But a list the stream shortens is not a short list. Its sixty-four candidates cost 192 words on every stream, four times the whole of the best estimator on the steep ones. On a flat stream every candidate is a near-equal share of a list of equals, and no share can tell that none of them is heavy. No list length and share comes within 10% everywhere; at Zipf 0.5 alone, the chosen list beats every fixed one.
A list that gives back its places
An estimator of the third frequency moment keeps a list of candidate heavy keys beside a Count-Sketch and samples the rest, and cutting a long list at the end of the pass to the candidates that hold a share of its moment picked the right keys and paid for the whole list. Dropping a candidate during the pass, as soon as it falls below that share, was predicted to make the list's memory follow the stream: under four candidates on steep streams, within a fifth of the best fixed list, and about ten on Zipf 0.5 with most of the earlier win kept. The list ends the pass the right length — three candidates on Zipf 1 — but its memory is its peak, and the peak is set by the twenty-fifth arrival, before any key has stood out: nineteen candidates on Zipf 1, and on flatter streams the most a share θ allows, one over θ. Charged its peak, it beats the end-of-pass cut on every stream from Zipf 0.75 up, 208 words against 364 at Zipf 1, and never reaches the best fixed list, 159.
Named alongside it
The objects these essays reach for when they reach for this one.
Heavy hitterUnbiased estimatorZipf distributionHonest limitSamplingAdditive errorCount-Min sketchLinear sketchOne-sided errorState bitsStreaming algorithmThreshold