Document array — where it appears
Named by 23 essays across 8 fields — each of them below, with the objects they name alongside it.
The cost that is the size of the answer
Ten range-minimum queries answer the listing at every point of a sweep where the occurrences run from 30 to 790. They cost 256 to 288 node visits — and the scan they replace costs 30 to 790, so the output-sensitive method loses until about thirty occurrences per document.
Work that falls as the answer grows
Output-sensitive usually means the cost rises with the answer instead of with the input. A descent over a document array costs five operations per document at an answer of seven and two at an answer of thirty-two, because the paths to many leaves share their tops.
A list of documents is not a list of occurrences
A pattern occurring 790 times in eight documents has an answer of size eight. Reading every occurrence to find out costs 790 array reads; the question a collection has that a text does not is the one its index does not answer.
One separator, or one for each
A shared separator costs one alphabet symbol and is free. Fifteen distinct ones take the alphabet from twenty-two to thirty-five, which crosses a power of two, so every character of every document costs a sixth bit — 1.2 times the packed collection, to tell the boundaries apart.
Two thousand documents of two hundred characters
Every collection this field has measured has been a dozen documents of ten thousand characters. A real collection is usually the other shape, and the other shape moves every term in a document index — one of them by a factor of seventy.
The cell nobody filled
Every structure here was measured either on one long repetitive text or on prose cut into short documents. The collection that is both is a two-by-two with one empty corner, and what is in it is not the product of its margins.
The array is the length distribution
The document array holds each document once per character it contributed, so its symbol distribution is the collection's length distribution exactly. On equal-length documents its entropy is log d and no coding saves anything.
The array the walk never reads
Document listing compares a chain entry against the start of a range. The comparison is true exactly when the document has not been reported yet — which the walk already knows, because it just wrote it down.
A code word is at least one bit
A wavelet tree of plain vectors reaches the entropy by its shape, and a Huffman code word cannot be shorter than one bit. On a collection whose document array has an entropy of 1.69 the tree costs 1.98, and the gap is a floor rather than an inefficiency.
A document already in the answer
The same walk, with the chain's test replaced by a lookup in the answer so far. It reports the same documents at the same cost, and two exchanged lines make it lose nine of seventeen without failing.
What the chain cost
The chain of previous occurrences is one row number per row — exactly as wide as the suffix array it sits beside, and the largest single part of a document-listing apparatus. It is now absent, and what replaces it is one bit per document.
The tree answers the question
The distinct documents in a range of rows are the distinct symbols of the document array in that range. A wavelet tree enumerates those in one descent, so the range minimum, the chain, the bitmap and the recursion all go at once.
The apparatus that is smaller than its index
Answering "which documents hold this" at a price proportional to the answer used to cost 2.70 times the index it sits beside. Two changes later it costs 0.84, and the largest thing left is an array that says which document each row belongs to.
The last array in the apparatus
The document-listing apparatus began as three arrays beside a suffix array. Two of them turned out to be machinery for reading the third, and both have gone. What is left is the document array, and it is the only one of the three that was information.
The apparatus, three times smaller again
Eight hundred and thirty-nine thousand bits became two hundred and eighty-five thousand, and now a hundred and twenty thousand. The listing apparatus is fourteen per cent of what it was and holds one array, which is the only part of it that was ever information.
The smaller tree hands it back unsorted
A frequency-shaped tree is sixteen per cent smaller and returns its documents in code order. A balanced one is larger and returns them sorted. The trade is d log d comparisons against a saving, which is not close — until the answer is truncated.
A saving quoted without its collection
A check asking whether a compressed document array is smaller than the plain one passes on the rounding whenever the document count is not a power of two. It would report a saving of nothing as sixteen per cent, on a collection that has no redundancy at all.
The collections the saving was quoted for
A document array compressed to the entropy of its own symbols saves exactly what the collection's length distribution allows, and the measurements that showed it used four constructed collections. The estimate they ended on — a directory of source files is roughly log-normal, so its array falls about a bit short of the plain one, around 17% — can be checked on a real directory. A project's 222 source files fall short by 0.66 bits and save 10.5%, a fifth of which is only the rounding of ⌈log₂ d⌉. Across six real collections the skew of the lengths is worth between nothing and 0.76 bits, and the log-normal formula overstates it wherever short documents are common.
The shape arrived early and the bytes late
A document array coded by its documents' lengths was predicted to survive a growing collection: fitted once at a quarter of a project's files, it would lose under a tenth of a bit a symbol by the end, because the spread of the sizes is set early. The spread was set early — 0.88 at 56 files, 0.90 at 222. The fitted code loses 0.50 to 2.08 bits all the same, because the files added after the fit hold 43% of the final bytes and every one of their symbols is named through an escape. Rebuilding only the escape's subtree brings it to 0.18.
One doubling late
A document array coded by its documents' lengths and refitted each time a growing directory's files double needs an escape for the files added in between, and the escape's right weight is the share of the characters still to come. Weighting it by the share that arrived since the previous refit was predicted to cut the mean loss below a quarter of a bit a symbol. It cuts nothing: 0.518 bits a symbol, against 0.389 for an escape that always takes half the coding space. The growth seen is the growth to come one doubling late — 71%, 50% and 27% where the doublings that followed brought 50%, 27% and 8% — because this directory's growth slowed at every doubling, and before the first refit there is no growth to read at all. A Huffman code also rounds every weight to a whole number of bits, so being right within a factor of two buys nothing. Started from half, the growth seen averages 0.356, the best a store can know, and the weight nobody can know reaches 0.183.
Refitting when the escape is wrong
A growing directory's document array, coded by a Huffman code over its files' lengths and refitted each time the files double, loses at best 0.356 bits a symbol for 5.65 final arrays of rewriting, and even an escape weighted by the growth still to come loses 0.183. A clock read off the structure — refit when the share of characters named through the escape passes twice the share its code word stands for — was predicted to refit about six times and halve the loss. It refits 25 times, eleven in the first twelve versions, and rewrites 10.6 arrays for a loss of 0.042. Held back until the characters have grown by a quarter since the last refit, it refits 11 times and loses 0.098 bits a symbol for 4.52 arrays: under every doubling schedule on both counts, including the one that reads the future. It sees anything only if the escape starts small: weighted as half the coding space, the escape's code word is one bit, nothing behind it can ever pass twice a half, and the clock never fires.
A price that cannot see the array
A growing directory's document array, refitted when the share of characters behind its escape passes twice what the escape's code word assumes and held until the characters have grown by a quarter, loses 0.098 bits a symbol for 4.52 final arrays of reading. A hold set by price instead — refit when the bits wasted since the last refit reach a constant times the characters a refit reads — was predicted to do at least as well, refitting early where refits are cheap. It does worse: about 0.119 at the same reading. The array's size multiplies both sides of the rule, so the price never sees it; what the rule watches is the loss summed over versions, and that refits late in the small, fast-growing early history where the fixed hold refitted often. The price also counts only the refits' reading, while waiting is paid in rebuilds of the escape, so the total reading stops falling at about 4.3 arrays however high the price goes.
The quarter was a lucky number
A growing directory's document array was refitted best by a clock read off its escape and held until the characters had grown by a quarter: 0.098 bits a symbol for 4.52 final arrays read. A price that weighed the refit against the loss alone did worse, because the array's size cancelled and the price left out the reading that waiting costs. Priced with that reading — refit when the escape rebuilds since the last refit, plus the waste a refit would remove at an exchange rate, pass the characters a refit reads — the rule beats the hold wherever both read more than about five arrays: 0.080 for 5.04 against the hold's best 0.088, 0.052 for 6.38 against 0.066. It loses only at the quarter's own point, and the quarter turns out to be a lucky setting: held to 1.2 or to 1.33 the clock is worse on both counts, because two bursts of growth arrive two versions at a time and only a quarter refits after both halves.
Named alongside it
The objects these essays reach for when they reach for this one.
Index sizeDocument collectionDocument listingRange minimumWavelet treeEntropyOutput-sensitiveSuffix arrayPrevious occurrence chainAmortisationHuffman codingPrediction