Concept

Approximate membership — where it appears

Answering whether an item is in a set with a bounded false-positive rate and no false negatives, in far fewer bits than the set itself. It costs far fewer bits than the set itself, and the saving is bought entirely with false positives whose rate is a parameter.

Named by 6 essays across 2 fields — each of them below, with the objects they name alongside it.

46810120.010.1bits per element (m / n)false-positive ratemeasured(1 − e^(−kn/m))^kfrom the bits set60,000 absent-key queries per point, seed 80800 false negatives at every size

A filter that is allowed to be wrong

A Bloom filter holding four thousand keys in five thousand bytes answers membership in four memory probes and gets 1.14% of its negative answers wrong. It never gets a positive one wrong. That asymmetry is the whole design, and the rate it makes errors at is a third quantity beside the operation count and the space.

randomness · Randomness
bits per elementε = 0.13.3 → 4.8 (+1.5)ε = 0.035.1 → 7.3 (+2.2)ε = 0.016.6 → 9.6 (+2.9)ε = 0.0038.4 → 12.1 (+3.7)ε = 0.00110.0 → 14.4 (+4.4)ε = 1e-413.3 → 19.2 (+5.9)floor log₂(1/ε) filled; Bloom's log₂(1/ε)/ln 2 outlined44.3% above the floor at every rate

A floor on the bits

Answering membership for n keys with a false-positive rate of 1% and no false negatives requires at least 6.64 bits per key, whatever the structure. A Bloom filter uses 9.59. The gap is 44.27% at that rate and at every other rate, and it is the first bound on this site that a real structure comes close to.

floors · Floor
bits used · answers that were wrongBloom, bits cleared16,384 bits638 said no wrongly · 14 said yes wronglycounting, 4 bits a cell65,536 bits0 said no wrongly · 42 said yes wronglyfingerprints in two slots32,768 bits0 said no wrongly · 153 said yes wronglyand the condition on the caller1,000 deletions of keys never inserted lost 15 that were2,000 keys, 1,000 deleted, 20,000 absent keys querieda false negative is a different kind of wrong from a false positive

The evidence a filter cannot remove

A Bloom filter never says no about a key it holds, and that is its whole guarantee. Clear the bits of a thousand deleted keys and it starts saying no about 638 of the thousand it still holds. A counter in every cell repairs it at four times the space; a fingerprint repairs it at twice, and acquires a condition on the caller that neither of the others has.

randomness · Randomness
queries answered yesabsent key, the AND0.190%absent key, built on the intersection0.025%in one set only, the AND1.800%in one set only, built on it0.000%bits set: A 6,351, B 6,294, AND 3,232, direct 1,859no common key is ever denied

The intersection two filters cannot report

Two Bloom filters over sets that share five hundred keys, ANDed bit by bit. The result never denies a shared key, and it looks like a filter of the intersection. It is not one — a key in only one of the sets passes it 1.8% of the time where a real filter of the intersection passes none, and reading the intersection's size off its bits gives 900.

randomness · Randomness
0.01%0.1%1%10%100%4,00012,00020,00028,00036,00044,00052,00060,000keys insertedabsent keys answered yesfingerprints, none reserved1 bit longer a doubling2 bits longer a doublingfingerprints, 5 reservedforecast 2,000, target 1.0%; dotted: the forecastdashed: the target rate

The bits given to the wrong keys

A fingerprint table that gives later arrivals longer fingerprints holds 3.6% where a table that reserves nothing holds 21.1%, and it never runs out of reserve because it has none. It also dies at exactly the same size as the table that reserved nothing — 32 times its forecast, on the same key — because every generation shares one quotient, and the generation with the shortest fingerprint is the one that arrived first.

randomness · Randomness
0%1.1%2.1%3.2%4.2%the whole filterblocks of 64blocks of 5120123456distinct 512-bit lines a lookup readsabsent keys answered yes16,384 bits, 2,048 keys, k = 6one line is what a block buys

Positions confined to one line

A Bloom filter lookup reads 5.55 cache lines because its six positions are scattered across the whole filter. Confining them to a 512-bit block makes it exactly one, and costs 7% more false positives at eight bits a key. At sixteen bits a key the same block costs 91%, and the two-value trick that is free across a whole filter costs another 135% inside one — because a block is a small filter, and small filters are where the penalty lives.

randomness · Randomness

Named alongside it

The objects these essays reach for when they reach for this one.

Bloom filterFalse-positive rateHash functionOne-sided errorCacheDesign parameterFingerprintSpace time tradeAmortisationBalls in binsBlocked filterCardinality

All concepts