Concept

Prediction — where it appears

A statement about a measurement not yet made, written down before it is made so that it can turn out false. It is what makes a continuation checkable: the page that follows reports what the measurement gave rather than whether it agreed.

Named by 3 essays across 3 fields — each of them below, with the objects they name alongside it.

H0H1H2H3H4model order — symbols of contextbits per symbol0.02.14.2Uniform over 8 symbolsOrder-1 Markov chainWords from a fixed vocabulary32,768 symbols · at H4 the deepest source has 709 contexts, 0 seen oncemodels: orders 0, 1, 2, 3, 42.06 bits found by one symbol of context

The model is the compressor

One stream of 32,768 symbols has an entropy of 3.886 bits per symbol, and 2.243, and 1.186, and 0.991, and 0.909. All five numbers are correct, all five are floors, and nothing about the data changed between them. The only thing that changed is how many preceding symbols the model was allowed to look at — which makes the entropy of a file a property of a decision rather than a property of a file.

text · Bits
words drawn independentlymean +0.0038 ± 0.011ahead on 10 of 16next word set by the last lettermean +0.0064 ± 0.011ahead on 12 of 16no difference−0.0176+0.0000+0.0251bits a symbol the short context savesthe stream reported beforek = 4 · 16 streams a source · bands: ±1 standard deviationone draw, not a result

A result the size of its own noise

Sorting rotations by their first four characters beat the full Burrows–Wheeler transform by 0.022 bits a symbol. On sixteen streams of the same source the mean difference is 0.004 with a standard deviation of 0.011, and the short context is ahead on ten of them. The 1.2% was one draw. What survives is sharper than the result it replaces: from four characters upwards, every run break either rule makes is at a word boundary, and nothing above four characters decides anything at all.

floors · Bits
1001000entries in the cacheshare of the repeats it catches0%25%50%75%100%256 entries, one a pair asked in a passSelection sortBubble sortdashed: collisions alonerule at 100%: the table ofevery pair256 random elements · direct-mapped, one pair a slotoccupancy, not distance

The record that forgets on purpose

A table of every pair selection sort might ask is 131,072 bits at 256 elements and catches every repeated question. A direct-mapped cache of 1,024 pairs is 16,384 bits and catches 91% of them, at a break-even of 3.53 word operations a comparison against the table's 3.12. What decides the cache is not how long the sort waits before asking again — 98% of its repeats come within one pass — but how many other questions it asks meanwhile, and a model with nothing in it but collisions predicts every measured share to within 2.6%.

counting · Count

Named alongside it

The objects these essays reach for when they reach for this one.

Context modelEntropyHonest limitMeasurement designModel orderAdaptive codingBits per symbolBurrows-wheeler transformCacheComparison countConditional entropyCost model

All concepts