Generator

False-positive rate against bits per element, k = 4, n = 4,000

Rendered here at the parameters it defaults to, with every essay that calls it — which is the same list as the blast radius of changing it.
False-positive rate against bits per element, k = 4, n = 4,000Filled circles are measured: 60,000 queries for keys that were never inserted, counted. The solid line is (1 − e^(−kn/m))^k, the formula every sizing guide prints. The dashed line is the same expression computed from the bits the filter actually has set. At 4 bits per element the measured rate is 16.2% and at 12 it is 0.620%. Not one inserted key tested absent at any size: the errors a filter makes are all in one direction. The keys are drawn at random, which is what the analysis assumes; the two lines then agree with the points to within 2.8%.46810120.010.1bits per element (m / n)false-positive ratemeasured(1 − e^(−kn/m))^kfrom the bits set60,000 absent-key queries per point, seed 80800 false negatives at every size

False-positive rate against bits per element, k = 4, n = 4,000

Filled circles are measured: 60,000 queries for keys that were never inserted, counted. The solid line is (1 − e^(−kn/m))^k, the formula every sizing guide prints. The dashed line is the same expression computed from the bits the filter actually has set. At 4 bits per element the measured rate is 16.2% and at 12 it is 0.620%. Not one inserted key tested absent at any size: the errors a filter makes are all in one direction. The keys are drawn at random, which is what the analysis assumes; the two lines then agree with the points to within 2.8%.

Drawn at 690 × 400, wide on the page. Everything above is what bloom-rate returns with no arguments; the caption is the generator's own, computed from the numbers in the drawing rather than written beside it.

25 essays call bloom-rate. The drawing above is what it returns with no arguments at all; every call below passes it something, because a placement that passes nothing draws whichever member of the family the generator happens to default to rather than the one its essay argues about — which is what optcheck and figfill exist to catch.

Where it is called

Changing this generator changes every one of these figures.

hash rowh1h2h3h4h1 → cell 30: 10,433h2 → cell 29: 10,624h3 → cell 19: 10,575h4 → cell 4: 10,895key 0 occurred 9,579 times · the minimum of the four is 10,433 · over by 854the additive bound at this width is e/w × N = 5,0974×32 counters · 4,096 bits · Zipf s = 1.1exact would take 162,288 bits One pass, and no room

A count that is never under

46810120.010.1bits per element (m / n)false-positive ratemeasured(1 − e^(−kn/m))^kfrom the bits set60,000 absent-key queries per point, seed 80800 false negatives at every size When the algorithm flips a coin

A filter that is allowed to be wrong

46810120.010.1bits per element (m / n)false-positive ratemeasured(1 − e^(−kn/m))^kfrom the bits set60,000 absent-key queries per point, seed 80800 false negatives at every size What is taught wrongly

The formula everybody sizes filters with

bits used · answers that were wrongBloom, bits cleared16,384 bits638 said no wrongly · 14 said yes wronglycounting, 4 bits a cell65,536 bits0 said no wrongly · 42 said yes wronglyfingerprints in two slots32,768 bits0 said no wrongly · 153 said yes wronglyand the condition on the caller1,000 deletions of keys never inserted lost 15 that were2,000 keys, 1,000 deleted, 20,000 absent keys querieda false negative is a different kind of wrong from a false positive When the algorithm flips a coin

The evidence a filter cannot remove

queries answered yesabsent key, the AND0.190%absent key, built on the intersection0.025%in one set only, the AND1.800%in one set only, built on it0.000%bits set: A 6,351, B 6,294, AND 3,232, direct 1,859no common key is ever denied When the algorithm flips a coin

The intersection two filters cannot report

HyperLogLog1024/1024 identicalCount-Min256/256 identicalbottom-k128/128 identicalMisra-Gries20/36 identicalfraction of the state that merged to the identical value→ 1,974→ 10,291→ 2,322→ 8,766two streams of 30,000 · 2,007 distinct keys in the union3 of 4 merge exactly Structures

The summaries that add

0.01%0.1%1%10%100%1,0002,0003,0004,0005,0006,0007,0008,000keys insertedabsent keys answered yesone filterm = 19,171, k = 7; dotted: the design sizedashed: the design rate When the algorithm flips a coin

A filter past its design size

0.01%0.1%1%10%100%1,0003,0005,0007,0009,00011,00013,00015,000keys insertedabsent keys answered yesone Bloom filtera stack of Bloom filtersfingerprints, none reservedfingerprints, 3 reservedforecast 2,000, target 1.0%; dotted: the forecastdashed: the target rate When the algorithm flips a coin

A filter that grows by moving a bit

3579111311.523510positions per key, krate ÷ the independent rateh₁ + i·h₂h₁ + i·h₂, step oddh₁ + i·h₂ + (i³ − i)/6optimal load m·ln 2 / k; one set of hash functions per schemedashed: the account When the algorithm flips a coin

Two hash values and the keys they copy

0.01%0.1%1%10%100%4,00012,00020,00028,00036,00044,00052,00060,000keys insertedabsent keys answered yesfingerprints, none reserved1 bit longer a doubling2 bits longer a doublingfingerprints, 5 reservedforecast 2,000, target 1.0%; dotted: the forecastdashed: the target rate When the algorithm flips a coin

The bits given to the wrong keys

0%1.1%2.1%3.2%4.2%the whole filterblocks of 64blocks of 5120123456distinct 512-bit lines a lookup readsabsent keys answered yes16,384 bits, 2,048 keys, k = 6one line is what a block buys When the algorithm flips a coin

Positions confined to one line

8121620240.000010.00010.0010.01bits a keyfalse-positive ratethe whole filterone block of 512 bitsthe emptier of two 512-bit blocksone block of 1,024 bits2,048 keys · 24 filters a pointdashed: no blocks When the algorithm flips a coin

Two blocks and the chances they add

8121620240.00010.0010.01bits a keyfalse-positive rateone blockthe emptier of two blocksone block, chosen by athreshold2,048 keys · 24 filters a pointblocks of 512 bits · same total budgetbest from 12 bits a key When the algorithm flips a coin

A block the lookup can work out

11010010³0.00010.0010.010.1keys held, as a multiple of the forecast of 2,000share of absent keys answered yesthe 1% targetten bits reservedrebuilt from the storerebuilt only when it cannotgrowa stack of Bloom filtersforecast 2,000 at 1%, grown to 2,000,00020,000 absent keys a point When the algorithm flips a coin

The day a filter cannot grow

10,000100,000keys in the filtershare of absent keys answered yes0.04%0.05%0.06%0.08%0.10%the emptier of twoone blockone block, Poisson averageone block, by threshold16 bits a key, 512-bit blocks2,000,000 absent queries a point When the algorithm flips a coin

A penalty set by the block

0.00%0.05%0.10%0.15%256 bits · one block0.148%256 bits · a threshold on the first0.089%256 bits · thresholds on both0.095%512 bits · one block0.089%512 bits · a threshold on the first0.062%512 bits · thresholds on both0.066%1024 bits · one block0.066%1024 bits · a threshold on the first0.058%1024 bits · thresholds on both0.052%with no spreadwhat the spread adds16 bits a key, 8,192 keysno spread: every block at its mean load When the algorithm flips a coin

The bits a second threshold costs

124816adjacent blocks sharing one thresholdfalse-positive rate0.06%0.08%0.10%0.12%0.14%256-bit blocks512-bit blocks256: one block, no threshold256: one block twice as wide8,192 keys, 16 bits a keyfour-bit thresholds When the algorithm flips a coin

A threshold knows only its own block

0.0%0.1%0.2%0.3%0.4%0.5%0.6%false-positive rate64-bit blocks1927 blocks128-bit blocks992 blocks256-bit blocks504 blocks512-bit blocks254 blocksreduced by a moduloreduced by its top bitseach dot: one seedupper row of a pair: modulo; lower: top bits When the algorithm flips a coin

Two hashes that shared their low bits

22.5033.504block width, bitstable bits a block64128256512four bits eachcoded, a line each, offsetscoded, a line eachcoded, with an indexthe entropyabout two thousand blocks at each widthdashed: the closing distribution's entropy When the algorithm flips a coin

A code that has to fit its worst line

2.5033.504block width, bitstable bits a block64128256512four bits eacha bit and four, a line eacha bit and four, end to endcoded, a line eachthree bits eacha bit and three, a line eachmean of four seeds, about two thousand blockslabels at 64-bit blocks When the algorithm flips a coin

A bit for whether it closed

33.203.403.60blocks in the filter (64-bit blocks, 16 bits a key)table bits a block1,93615,497123,885the mean needsized for the worst linemean + 0 sd, overflowmean + 1 sd, overflowmean + 2 sd, overflowthe line-packed split, four-bit valueswhole lines; two seeds When the algorithm flips a coin

The worst of more lines

3.203.403.6022.012.02lines read a lookuptable bits a blockz 0z 0.5z 1z 1.5z 2offset for moved valuesoffset within 16 linesoffset for its overflow124,488 blocks, two seedslower and left is better When the algorithm flips a coin

An offset that only has to reach its neighbours

3.203.303.403.5022.012.02lines read a lookuptable bits a blockungrouped, offset fittedpooled, reserve for spillspooled, reserve as proposed124,488 blocks, groups of 64 lineswhole lines; two seeds When the algorithm flips a coin

A reserve sized for the spill

3.203.303.403.5022.012.02lines read a lookuptable bits a blockungrouped, offset fittedreserve sized for spillsspills carried forward124,488 blocks, no reservewhole lines; two seeds When the algorithm flips a coin

The room the lines lend

3.203.303.403.5022.022.04lines a lookupbits a blockungrouped linespill reservecapped at 1capped at 2capped at 4524,288 keys, two seedsdown and left is better When the algorithm flips a coin

A walk that stops at two lines

The library, page 1 of 5 — where bloom-rate sits