Generator

Peak state against passes, for two splitting rules on two distributions

Rendered here at the parameters it defaults to, with every essay that calls it — which is the same list as the blast radius of changing it.
Peak state against passes, for two splitting rules on two distributionsExact selection of the 0.5 quantile of 32,768 values, in p passes, with the state measured at its peak during the run. Every point is an exact answer — a run that returned anything else stops the build — so the vertical axis is the price of exactness rather than a price paid for being approximately right. The line at the top is what one pass costs: 1,048,576 bits, because an exact one-pass selector has to keep the stream, and the floor under that is a counting argument this collection performs rather than quotes. The equal-width rule needs no knowledge of the data and gets 37 times smaller as passes are added — on data spread evenly. On a heavy tail it collapses, because an equal-width cut of a range containing almost no data narrows the range and not the count; the sampled rule, which spends half its passes finding out where the data is, is barely affected. Both axes are logarithmic.10³10⁴10⁵10⁶passes over the datapeak bits of stateone pass, exact: 1,048,576 bitsradix, uniformsample, uniformradix, paretosample, paretoevery point exact · 32,768 values320 bits at best

Peak state against passes, for two splitting rules on two distributions

Exact selection of the 0.5 quantile of 32,768 values, in p passes, with the state measured at its peak during the run. Every point is an exact answer — a run that returned anything else stops the build — so the vertical axis is the price of exactness rather than a price paid for being approximately right. The line at the top is what one pass costs: 1,048,576 bits, because an exact one-pass selector has to keep the stream, and the floor under that is a counting argument this collection performs rather than quotes. The equal-width rule needs no knowledge of the data and gets 37 times smaller as passes are added — on data spread evenly. On a heavy tail it collapses, because an equal-width cut of a range containing almost no data narrows the range and not the count; the sampled rule, which spends half its passes finding out where the data is, is barely affected. Both axes are logarithmic.

Drawn at 700 × 440, wide on the page. Everything above is what pass-state returns with no arguments; the caption is the generator's own, computed from the numbers in the drawing rather than written beside it.

8 essays call pass-state. The drawing above is what it returns with no arguments at all; every call below passes it something, because a placement that passes nothing draws whichever member of the family the generator happens to default to rather than the one its essay argues about — which is what optcheck and figfill exist to catch.

Where it is called

Changing this generator changes every one of these figures.

above 0.01 of the stream · 13 really areone pass: 18 candidatestwo passes: 13 keys, exactly the heavy ones123456789101112131415161718128 counters · carry 1,376 bits5 of 18 spurious What a bound is

The pass that was never a parameter

10³10⁴10⁵10⁶passes over the datapeak bits of stateone pass, exact: 1,048,576 bitsradix, uniformsample, uniformradix, paretosample, paretoevery point exact · 32,768 values320 bits at best The other axis

What a second pass buys

01234,09616K64K256K1.00M4.00Mmemory in 32-bit words, both buffers included (logarithmic)blocks moved ÷ blocks in the file3 passes2 passes1 passreread the file every passwrite the interval out everypass4,194,304 keys, block 1,024 wordstwo passes from 4,096 words What a bound is

What a pass costs when it is a file

universe 0…11 · prefixes of 6 · candidate keeps 9 bitsthe two prefixes that collideA01234567891011B01234567891011first differencesame state — the candidate stores "7" after boththen both read the same suffix -1, -2, 12, 13, 14true median of A3true median of B4and one answer for both924 prefixes · floor ⌈log₂ C(12,6)⌉ = 10 bits10 bits collide on none The floors

A floor one pass cannot get under

020405101520queries answeredfile-equivalents read and written, in allevery query from nothingthe counts keptparts of equal widthparts the counts balanceuniform keys, 2²² keys, 8K wordsevery answer checked against a sort What a bound is

A file written once for the queries after

50%30%10%3%1%0.050.10.20.512share of queries outside the top 2% of rankscost over parts balanced by keysbalanced by past queriessize × queries equal(∫√q)², predictedseven parts, fifty past queriesdotted: parts balanced by keys What a bound is

Parts cut where the questions are

0.020.050.10.20.530050010³10³drift, percentiles every 100 queriestransfers a querya band that never moves, 356best fixed interval, chosenafterevery 1,000 queriesthe law, half-life 250the law, half-life 1,000four streams of 20,000 queries a speedrewrites included What a bound is

A recut the law decides

0%25%50%75%100%05,00010,00015,00020,000queriesrank in the filethe band's centrethe six cutsone stream, drift 0.1a cut is drawn at each recut What a bound is

Every cut the law moves

The library, page 3 of 5 — where pass-state sits