The thread: Distributions, not averages — page 2
A detector that learns its own quiet
A burst detector built from a fast decayed counter needs a threshold, and the one it was measured with came from sixty independent runs of the background — an oracle no deployed system has. Two counters and a running mean square of their difference can supply it from the stream itself, and on a stream that does not move they hit a 5% false-alarm rate at every setting tried. What they lose is the burst: a scale that learns every reading learns the burst as noise and forgets it in 20 to 40 seconds whatever its size. Holding the scale while alarmed brings the oracle's horizon back for small bursts, turns a slow drift into an alarm that will not stop, and still cannot see past the lag at which the slow counter holds more of the burst than the fast one.
When the algorithm is a tableThe counts only short words carry
A spelling checker capped at two edits still gives a false suggestion to many words its vocabulary lacks, and the one evidence about a writer's intention it already holds is how often each vocabulary word is used. Scored by its nearest words' counts, weighed a tenth for each edit, an answer was predicted to separate a slip from an unfamiliar word on short queries. On the streams the checker was built against, whose typos are of words drawn evenly from the list, the false suggestions go to commoner words than the corrections do, and the test gives up sixteen corrections for each false suggestion it removes. Drawn as a writer mistypes, in proportion to use, the counts separate the two at last — and still trade worse than a plain cap of one edit, except on words of four letters or fewer, where a threshold removes a ninth of the false suggestions for one correction in ninety-six.
One pass, and no roomThe clock a burst cannot outlast
A burst detector that holds its scale while alarmed keeps a burst visible to a ceiling set by two half-lives, and cannot stop alarming on a slow drift. The repair proposed for it was a schedule: learn again once an excursion has outlasted anything a burst could produce. Timed by how long the alarm has lasted, the schedule changes almost nothing, because on a gentle rise the alarm breaks every few seconds and the clock never reaches the ceiling. Timed by how long the difference between the counters has stayed positive, it keeps every burst to within about a second of the ceiling and falls below the target false-alarm rate on steep drifts — and one quantity, the drift's offset measured in spreads of the difference, says where it still fails.
CountingThe spread a merge sort does not have
On 65,536 random keys quicksort's comparison count has a standard deviation of 42,470 and merge sort's has one of 141.9 — three hundred times smaller, on means only a quarter apart. Merge sort's count is a fixed bill minus 65,535 independent leftovers, one per merge, each worth about a coin toss, so its spread grows as the square root of n where quicksort's grows as n. From 256 keys up a single run of merge sort pins its mean to one per cent, and a third of quicksort's whole spread is decided by its first pivot.
When it does not fitA planner that knows how sure it is
A planner that fits its insurance divisor from a handful of executed queries under-insures, because a spread fitted to a small sample is biased low. The textbook repair is to use the upper end of a confidence interval on the spread instead of its centre. At a read ratio of four it makes the planner worse than trusting its estimate for longer — until sixty-four queries rather than sixteen — and it buys the worst case immediately, from 22.4 times the better plan to 6.2 after two queries. The bound is not a correction towards the truth. It is a dial between two different planners.
One pass, and no roomThe keys worth counting before sampling
The sampling estimator for the third frequency moment spends most of its copies on keys that barely contribute to the answer. Finding the heavy keys first with a Count-Sketch, counting them, and sampling only the rest cuts the state for ten per cent from 5,260 words to about 1,500 on a moderately skewed stream, and from 23,598 to under a thousand on the stream built to be worst for the sampler. It costs five times more than it saves on a flat stream, where there is no heavy key to find. What decides it is how the frequencies are shaped, not how long the stream is, and the frequencies must be counted, not read off the sketch, because the sketch's error on a heavy key is cubed.
One pass, and no roomThe slope the slow counter already shows
A burst detector built from a fast and a slow decayed counter reads a steady rise in its background as an offset in their difference, β(R − 1)H/ln 2, and alarms on it. Every repair so far waited the offset out. This one subtracts it: the slow counter's own change over one slow half-life is a reading of the slope, and the offset follows from it. At R = 16 the corrected detector alarms on 5.5% to 7.8% of readings on rises that make the held scale alarm on 11% to 63%, and it costs a burst at most 3.7 seconds of detection. The estimate has to be quick: four times slower, it has recovered only 71% of the offset when alarms start to count.
One pass, and no roomThe rise the correction kept subtracting
A burst detector that subtracts a rising background's offset, read off its own slow counter, goes on subtracting it after the rise has stopped. The prediction was a deficit for about one slow half-life that would cost bursts of 300 arrivals. Measured, the deficit is deeper and longer: down to 1.46 spreads below zero, below half a spread for seventeen minutes after a 40% rise. Bursts of 300 are never missed. What the deficit costs is small bursts — 30 arrivals caught 27% of the time where the rise saw 77% — and a fifth of a large burst's visible time.
When the algorithm flips a coinA penalty set by the block
A blocked Bloom filter pays a penalty for its uneven block loads, and a threshold that lets a lookup work out its block repays about a third of it — measured on 2,048 keys, with a prediction that larger filters would widen the gain. From 2,048 keys to 131,072 nothing moves: one block holds 0.086% to 0.090% against a computed 0.086%, and the threshold design 0.060% to 0.065%. The busiest block grows and the rate does not follow it. What sets both the penalty and the repair is the keys a block holds. A threshold turns out to be worth one doubling of the block's width, and its advantage shrinks from 40% at sixteen keys a block to between 5% and 13% at 128.
The data that is not a numberWhat repetition is worth once the logarithm is gone
On prose, nearly three quarters of the fall in phrases per character with size is arithmetic that any text pays. On a collection built of copies it is a fifth, and what is left is a factor of three that is genuinely the arrangement.
When the algorithm flips a coinThe bits a second threshold costs
A blocked Bloom filter whose first block carries a threshold was proposed a second threshold on each key's second block, with a third block for keys both refuse. It does what it was meant to: at 512-bit blocks it removes nearly a third of the load spread the first threshold left. And the rate does not fall — 0.066% against 0.062% — because the second table, paid for in blocks, raises the rate the filter would have with no spread at all by more than the narrower spread saves. The threshold was stored in more bits than it needs: at four to six bits, one threshold beats the eight-bit design by about a tenth on one run, and a second threshold of the best width ties it at 512-bit blocks.
The data that is not a numberThe array is the length distribution
The document array holds each document once per character it contributed, so its symbol distribution is the collection's length distribution exactly. On equal-length documents its entropy is log d and no coding saves anything.
When the algorithm flips a coinA threshold knows only its own block
A blocked Bloom filter that gives each block a threshold on a hash of the key lets a lookup work out which of two blocks a key went to, and it spends a small table to do it. Sharing one threshold among a group of blocks would shrink the table, and was proposed as the next saving. It is a loss at every group size. With one four-bit threshold a block, 512-bit blocks reach a false-positive rate of 0.061%; shared by a pair, 0.071%; shared by sixteen, 0.100%, worse than no threshold at all. Neighbouring blocks fill at unrelated moments, so a shared threshold closes when the first of its group does, and every key it turns away lands in a second block chosen at random.
When the algorithm flips a coinTwo hashes that shared their low bits
Two pages on blocked Bloom filters with thresholds each recorded an instability they could not explain: a false-positive rate that collapsed on one seed at 256-bit blocks, and a pair of four-bit thresholds measuring 0.169% where its neighbours measured 0.08%. Both came from how a hash was brought into range. Every filter took its block and its ordering value as a modulo of a multiply-shift hash, and the low bits of such a hash depend only on the low bits of the key, so the two values were dependent whenever the block count had a factor of two to spare. At 992 blocks the threshold filter measures anything from 0.185% to 0.645%; read from the hash's top bits, 0.126% to 0.133%.
What is taught wronglyThe cap that binds on one text and not another
A periodic text looks like the one made of chains and pays 1.11 times the phrases for a cap of four. A text that repeats itself pays 2.96. The guess is backwards, and the reason is that depth measures nesting rather than repetition.
When the algorithm flips a coinThe worst of more lines
A blocked Bloom filter's threshold table, stored as a bit for whether each block closed and a value for each block that did, packed a fixed number of blocks to a cache line, costs 3.64 bits a block at 64-bit blocks because every line must hold the most closed blocks any line draws. Sizing the line for the mean plus one standard deviation, and moving a line's excess values to a shared overflow, was predicted to cost about 3.25 bits and send under one closed lookup in a hundred to a third line. The lookups behave as predicted: 0.6% read the overflow, 2.003 lines a lookup. The bits land at 3.38, because every line must carry the offset where its moved values begin. What the measurement found instead is that the worst line is a property of how many lines there are: 3.38 bits at two thousand blocks and 3.62 at a hundred and twenty thousand, while the line sized for the usual line costs the same at every size.
What a bound isFlat in the budget, and not
One saving is eleven times at every error budget, because it is a property of the alphabet. The other moves between ninety-eight and a hundred and five, because it follows the share of extensions that find nothing. Two savings, two shapes, and neither line crosses the other.
What is taught wronglyOne copy per document is one occurrence per document
The output-sensitive listing apparatus wins when a pattern occurs far more often than it occurs in documents. A collection of versions was supposed to be that case, and it is the one collection where the two numbers are equal by construction.
What a bound isA file written once for the queries after
An exact selection against a file of four million keys counts the whole file into a thousand or more pieces on its first pass, keeps the one piece holding its rank, and throws the rest away. Kept, those counts save every later query one full read of the file, from the second query on. Written out as parts, the file makes each later query read a fraction of it — but a pass that counts 3,582 pieces can write only seven, because each part needs a buffer. On a heavy tail, parts of equal width never pay; parts cut where the counts say the keys are pay by the second query and bring twenty-three selections from 69 reads of the file to 9.
What a bound isParts cut where the questions are
A file of four million keys, written once as seven parts so that each later selection reads only its own part, can be cut where the keys are or where the questions are. When nine queries in ten ask about the top 2% of ranks, parts balanced by keys cost 1,170 block transfers a query. Parts that each serve an equal share of past queries cost about as much, and a little calculus says they must, on average. Parts cut so that each one's size times its share of queries is equal cost 290, which is the optimum's square-root law, measured to within a third at every concentration tried. The saving lasts exactly as long as the questions stay where they were counted: moved a percentile and a half, the fitted parts cost more than the plain ones.
The index that replaces the textThe collections the saving was quoted for
A document array compressed to the entropy of its own symbols saves exactly what the collection's length distribution allows, and the measurements that showed it used four constructed collections. The estimate they ended on — a directory of source files is roughly log-normal, so its array falls about a bit short of the plain one, around 17% — can be checked on a real directory. A project's 222 source files fall short by 0.66 bits and save 10.5%, a fifth of which is only the rounding of ⌈log₂ d⌉. Across six real collections the skew of the lengths is worth between nothing and 0.76 bits, and the log-normal formula overstates it wherever short documents are common.
What a bound isA recut the law decides
Parts of a file cut by the square-root law where selection queries fall lost their advantage once the queries moved a percentile and a half. A partition that recuts itself when the law says the saving pays for the rewrite was predicted to stay within 20% of the best fixed recut interval at every drift speed, and to fail on traffic that jumps. With recent queries decayed at a half-life of 250, it stays within 7% up to a drift of 0.1 percentiles every hundred queries and within 22% at five times that. On jumping traffic it beats every interval fixed in advance, rewriting 13 times a stream where the best interval rewrites 80. The law does not remove a setting: it trades the interval, which has to move fourfold with the speed, for a half-life that does not.
The index that replaces the textThe shape arrived early and the bytes late
A document array coded by its documents' lengths was predicted to survive a growing collection: fitted once at a quarter of a project's files, it would lose under a tenth of a bit a symbol by the end, because the spread of the sizes is set early. The spread was set early — 0.88 at 56 files, 0.90 at 222. The fitted code loses 0.50 to 2.08 bits all the same, because the files added after the fit hold 43% of the final bytes and every one of their symbols is named through an escape. Rebuilding only the escape's subtree brings it to 0.18.
What is taught wronglyThe folklore is about a matcher
Four million steps against three hundred and twenty-nine, on a twenty-character expression matched against twenty characters. One of the two machines doubles with every character of the input and the other does not, and only one of them is what a regular expression is.
What a bound isEvery cut the law moves
A file cut into seven parts by the square-root law, and recut when the law says the saving pays, was charged a full rewrite for every recut. Charging only the parts whose boundaries change was predicted to cut a recut's cost to a tenth on slow drift. It cuts it by 2%. When the band of hot queries moves, the law moves 98% of its cuts, a median of nineteen thousand ranks each, because the parts on either side of the band have to share out what the band left. Holding small moves back keeps a quarter of the file and costs more a query; recutting only the parts around the band costs up to three times as much.
The index that replaces the textOne doubling late
A document array coded by its documents' lengths and refitted each time a growing directory's files double needs an escape for the files added in between, and the escape's right weight is the share of the characters still to come. Weighting it by the share that arrived since the previous refit was predicted to cut the mean loss below a quarter of a bit a symbol. It cuts nothing: 0.518 bits a symbol, against 0.389 for an escape that always takes half the coding space. The growth seen is the growth to come one doubling late — 71%, 50% and 27% where the doublings that followed brought 50%, 27% and 8% — because this directory's growth slowed at every doubling, and before the first refit there is no growth to read at all. A Huffman code also rounds every weight to a whole number of bits, so being right within a factor of two buys nothing. Started from half, the growth seen averages 0.356, the best a store can know, and the weight nobody can know reaches 0.183.