Document collection — where it appears
Named by 24 essays across 6 fields — each of them below, with the objects they name alongside it.
A corpus that was not generated
Every collection in five strands has been copies of a generated text with a fraction of its characters replaced — three numbers, one dial. Here is one that was not — twelve essays, eight source modules, ten revisions of one file — measured beside the model of it.
Documents that are not the same length
A separator per document costs a whole bit per character — on a generated collection whose alphabet is 21 symbols, where adding a few crosses 32. Real prose has 87 symbols and sits 41 short of the next power of two, so the same separators cost 0.04%.
A million characters of the same thing
Every measurement this collection has published about real text was taken on twenty-four thousand characters, because the phrase count was quadratic. It is linear now, so here is the same corpus at forty times the size — and what forty times does to its own numbers.
Two thousand documents of two hundred characters
Every collection this field has measured has been a dozen documents of ten thousand characters. A real collection is usually the other shape, and the other shape moves every term in a document index — one of them by a factor of seventy.
The crossing that never arrives
Output-sensitive document listing exists because a pattern can occur four thousand times in eight documents. On a real collection of two thousand short documents it occurs 1.04 times per document, and the whole apparatus buys nothing at all.
The cell nobody filled
Every structure here was measured either on one long repetitive text or on prose cut into short documents. The collection that is both is a two-by-two with one empty corner, and what is in it is not the product of its margins.
The array is the length distribution
The document array holds each document once per character it contributed, so its symbol distribution is the collection's length distribution exactly. On equal-length documents its entropy is log d and no coding saves anything.
The array the walk never reads
Document listing compares a chain entry against the start of a range. The comparison is true exactly when the document has not been reported yet — which the walk already knows, because it just wrote it down.
The index that does not notice
Three compressed indexes over the same characters. One is flat at six and a half bits a character however many copies the collection holds; the other two fall by factors of five and six. At one copy the two that fall are the largest of the three.
What the chain cost
The chain of previous occurrences is one row number per row — exactly as wide as the suffix array it sits beside, and the largest single part of a document-listing apparatus. It is now absent, and what replaces it is one bit per document.
A boundary that costs nothing
A separator occurs nowhere else in the collection, so it must break a phrase that would have spanned it. Cutting a repetitive text into a hundred and twenty-eight documents raises its run count by two per cent, and cutting prose raises it by ten.
The price of a boundary is what precedes it
A separator sorts before everything, so its rows sit at the top of the suffix array and hold the documents' last characters. What a document boundary costs the transform is the entropy of the character in front of it, and nothing else.
The apparatus that is smaller than its index
Answering "which documents hold this" at a price proportional to the answer used to cost 2.70 times the index it sits beside. Two changes later it costs 0.84, and the largest thing left is an array that says which document each row belongs to.
Two currencies for one separator
Giving every document its own boundary marker costs a fifth of the packed text and three per cent of the run count. Both numbers are right, they are about the same change, and which one a collection pays depends on a structure nobody named.
One copy per document is one occurrence per document
The output-sensitive listing apparatus wins when a pattern occurs far more often than it occurs in documents. A collection of versions was supposed to be that case, and it is the one collection where the two numbers are equal by construction.
The last array in the apparatus
The document-listing apparatus began as three arrays beside a suffix array. Two of them turned out to be machinery for reading the third, and both have gone. What is left is the document array, and it is the only one of the three that was information.
A collection is a construction
The same characters, arranged as one copy per document or cut across the copies, give a different run count, a different boundary cost and a different answer about which structure to build. Which one a benchmark used is usually not recorded.
A saving quoted without its collection
A check asking whether a compressed document array is smaller than the plain one passes on the rounding whenever the document count is not a power of two. It would report a saving of nothing as sixteen per cent, on a collection that has no redundancy at all.
The collections the saving was quoted for
A document array compressed to the entropy of its own symbols saves exactly what the collection's length distribution allows, and the measurements that showed it used four constructed collections. The estimate they ended on — a directory of source files is roughly log-normal, so its array falls about a bit short of the plain one, around 17% — can be checked on a real directory. A project's 222 source files fall short by 0.66 bits and save 10.5%, a fifth of which is only the rounding of ⌈log₂ d⌉. Across six real collections the skew of the lengths is worth between nothing and 0.76 bits, and the log-normal formula overstates it wherever short documents are common.
The shape arrived early and the bytes late
A document array coded by its documents' lengths was predicted to survive a growing collection: fitted once at a quarter of a project's files, it would lose under a tenth of a bit a symbol by the end, because the spread of the sizes is set early. The spread was set early — 0.88 at 56 files, 0.90 at 222. The fitted code loses 0.50 to 2.08 bits all the same, because the files added after the fit hold 43% of the final bytes and every one of their symbols is named through an escape. Rebuilding only the escape's subtree brings it to 0.18.
One doubling late
A document array coded by its documents' lengths and refitted each time a growing directory's files double needs an escape for the files added in between, and the escape's right weight is the share of the characters still to come. Weighting it by the share that arrived since the previous refit was predicted to cut the mean loss below a quarter of a bit a symbol. It cuts nothing: 0.518 bits a symbol, against 0.389 for an escape that always takes half the coding space. The growth seen is the growth to come one doubling late — 71%, 50% and 27% where the doublings that followed brought 50%, 27% and 8% — because this directory's growth slowed at every doubling, and before the first refit there is no growth to read at all. A Huffman code also rounds every weight to a whole number of bits, so being right within a factor of two buys nothing. Started from half, the growth seen averages 0.356, the best a store can know, and the weight nobody can know reaches 0.183.
Refitting when the escape is wrong
A growing directory's document array, coded by a Huffman code over its files' lengths and refitted each time the files double, loses at best 0.356 bits a symbol for 5.65 final arrays of rewriting, and even an escape weighted by the growth still to come loses 0.183. A clock read off the structure — refit when the share of characters named through the escape passes twice the share its code word stands for — was predicted to refit about six times and halve the loss. It refits 25 times, eleven in the first twelve versions, and rewrites 10.6 arrays for a loss of 0.042. Held back until the characters have grown by a quarter since the last refit, it refits 11 times and loses 0.098 bits a symbol for 4.52 arrays: under every doubling schedule on both counts, including the one that reads the future. It sees anything only if the escape starts small: weighted as half the coding space, the escape's code word is one bit, nothing behind it can ever pass twice a half, and the clock never fires.
A price that cannot see the array
A growing directory's document array, refitted when the share of characters behind its escape passes twice what the escape's code word assumes and held until the characters have grown by a quarter, loses 0.098 bits a symbol for 4.52 final arrays of reading. A hold set by price instead — refit when the bits wasted since the last refit reach a constant times the characters a refit reads — was predicted to do at least as well, refitting early where refits are cheap. It does worse: about 0.119 at the same reading. The array's size multiplies both sides of the rule, so the price never sees it; what the rule watches is the loss summed over versions, and that refits late in the small, fast-growing early history where the fixed hold refitted often. The price also counts only the refits' reading, while waiting is paid in rebuilds of the escape, so the total reading stops falling at about 4.3 arrays however high the price goes.
The quarter was a lucky number
A growing directory's document array was refitted best by a clock read off its escape and held until the characters had grown by a quarter: 0.098 bits a symbol for 4.52 final arrays read. A price that weighed the refit against the loss alone did worse, because the array's size cancelled and the price left out the reading that waiting costs. Priced with that reading — refit when the escape rebuilds since the last refit, plus the waste a refit would remove at an exchange rate, pass the characters a refit reads — the rule beats the hold wherever both read more than about five arrays: 0.080 for 5.04 against the hold's best 0.088, 0.052 for 6.38 against 0.066. It loses only at the quarter's own point, and the quarter turns out to be a lucky setting: held to 1.2 or to 1.33 the clock is worse on both counts, because two bursts of growth arrive two versions at a time and only a quarter refits after both halves.
Named alongside it
The objects these essays reach for when they reach for this one.
Index sizeDocument arrayEntropyCorpusDocument listingRepetitionMeasurement designRange minimumSeparatorBurrows-wheeler runsControlOutput-sensitive