Concept

Pigeonhole — where it appears

The observation that more things than boxes forces a shared box, which turns a count of memory states into a lower bound on the bits a structure needs. It is the move behind every counting lower bound here, and the work is entirely in identifying the two sets rather than in the argument itself.

Named by 8 essays across 4 fields — each of them below, with the objects they name alongside it.

1,00010,0000.1bits of state heldrelative errorHyperLogLog (-0.49)LogLog (-0.44)bottom-k (-0.38)50,000 distinct keys · 14 runs per point · truth counted exactlybest: 2.00% at 10,240 bits

The answer that is allowed to be wrong

Every algorithm on this site so far was checked for correctness before it was measured. A summary of a stream cannot be — the data goes past once and does not fit — so the error becomes a resource, bought with bits, at an exchange rate that is a measurement.

streaming · Sketch
pattern of 24, 3 errors allowedpositions in the text20,000candidates proposed140candidates verified140occurrences7pattern 24 · 3 errors · 28,700 cells against 480,000selectivity 5.0%

The filter that feeds the table

A self-index answers exact queries and nothing else. Approximate matching needs a table with twenty thousand columns in it. The pigeonhole joins them — cut the pattern into k+1 pieces and at least one occurs exactly, and the index that cannot answer the question decides where to ask it.

text · Distance
011228101214161820universe size ubits of statea u-bit bitmaplog₂ C(u, u/2)⌈log₂(u+1)⌉, a counterat u = 12: 924 subsets, 8 bitstwo collide → answers 6 and 7floor computed exactly · collision found by exhaustion at u = 12floor 9.85 bits

The floor under a summary

An exact one-pass distinct-counter over a universe of u keys needs at least log2 of u-choose-u-over-2 bits of state — the same counting argument as the sorting floor, applied to memory states instead of outcomes. At u = 12 that is 9.85 bits, and an eight-bit candidate is shown to collide by running all 924 subsets.

floors · Cardinality
1,00010³window length W, in arrivalsbits of state heldW bits, exactε = 0.2ε = 0.1ε = 0.05sliding-window model · 40,000 arrivals · state from the shape of the structure2,808 bits at W = 8,000

The floor under a window

An exact count of the ones in the last W arrivals needs W bits, and the argument is a pigeonhole that can be performed rather than quoted — 1,024 windows, an eight-bit state, the colliding pair produced, and the two answers it cannot tell apart.

floors · Window
universe 0…11 · prefixes of 6 · candidate keeps 9 bitsthe two prefixes that collideA01234567891011B01234567891011first differencesame state — the candidate stores "7" after boththen both read the same suffix -1, -2, 12, 13, 14true median of A3true median of B4and one answer for both924 prefixes · floor ⌈log₂ C(12,6)⌉ = 10 bits10 bits collide on none

A floor one pass cannot get under

An exact one-pass selector must reach a different memory state for every prefix it might have read, and the pigeonhole that proves it is small enough to perform — nine hundred and twenty-four prefixes through a nine-bit state, the collision produced, the suffix that separates it, and two true medians it cannot both return. Ten bits collide on none, so the bound is exact — and a second pass walks under it by a factor of ninety.

floors · Pass
k = 0k = 1k = 2k = 3k = 4k = 5k = 6q = 223211917151311q = 3221916131074q = 4211713951-3q = 520151050-5-10q = 6191371-5-11-17q = 81791-7-15-23-31a shaded cell is a threshold of zero or less: every window proposedt = m + 1 − q(k+1) · m = 2411 collapsed cells

The q-grams an error cannot destroy

A pattern of twenty-four characters holds twenty-one four-grams. Two errors can destroy at most eight of them, so any occurrence with two errors still shares thirteen — and a filter that keeps only the windows sharing thirteen proposes 104 of 3,977 and computes 16,744 table cells instead of 96,000.

text · Distance
right to left8302.74xleft to right8772.89xone per piece4471.48xthe scheme3031.00xinterval extensionsall four found the same 6 positions15 characters · k = 22.74x apart

The search that starts in the middle

The same pattern, the same six occurrences, the same index — and 830 interval extensions, or 303, according to which end the search begins at. A pattern cut into three pieces has an error-free one, and only a search with two ends can start there.

text · Distance
errors, by piecetaken by(0, 0, 0)1(0, 0, 1)1(0, 0, 2)1(0, 1, 0)1(0, 1, 1)1(0, 2, 0)3(1, 0, 0)2(1, 0, 1)2(1, 1, 0)3(2, 0, 0)30 of 10 covered more than once · the numbers are the searchesk = 2 · 3 pieces10 distributions

A schedule nobody writes down

A search scheme is valid when its searches between them cover every way the errors can fall — ten cases at two errors, enumerable in a line. A scheme missing one of them finds seven occurrences of eight, all real, at the right error counts, and reports nothing wrong.

wrong · Distance

Named alongside it

The objects these essays reach for when they reach for this one.

MeasurementApproximate matchingState bitsCounting argumentLower boundStreaming algorithmTrade offVerificationBidirectional indexCardinalityEdit distanceError budget

All concepts