Bidirectional index — where it appears
Named by 26 essays across 7 fields — each of them below, with the objects they name alongside it.
An interval that grows at both ends
A backward search step is two ranks. A bidirectional step is two ranks and the width of the interval for every symbol that sorts before the one being added — 16.5 ranks on four symbols and 138.6 on twenty-six, which is a cost no account of the structure mentions.
The search that starts in the middle
The same pattern, the same six occurrences, the same index — and 830 interval extensions, or 303, according to which end the search begins at. A pattern cut into three pieces has an error-free one, and only a search with two ends can start there.
The structure that was supposed to halve
A bidirectional index holds the transform of the text and the transform of its reversal — 35,335 bits each, 70,670 together, exactly twice one index. The deferral that named it hoped it would stop the index doubling. It does not remove the doubling; it reuses it.
The half that is never asked where
A bidirectional index is two indexes and one interval. One of them is asked for ranks several hundred times a search and for a position never — and the parts that answer "where" are two of the five it is made of.
The part every saving left alone
A bidirectional index was made smaller three times, one part at a time, and every change left the largest part untouched: the wavelet trees, whose levels are two thirds of the structure. Coding every level in blocks of 63 bits — how many ones a block holds and which arrangement it is — takes that protein index from 210,934 bits to 167,707, a larger step than the two before it together. On English words it takes the same structure from 210,872 bits to 76,054, and an ordered tree, coded, ends up smaller than an unordered Huffman tree, plain. Every rank then reads 1.4 times as many entries, and a smaller superblock buys the reads back for part of the bits.
An index that cannot locate
Take an FM-index, remove the sampled positions and the bit vector marking them, and refuse every request for a position. What is left still counts, still extends intervals, still runs a whole search — and is a third smaller.
Six bits a class
Coding a bidirectional index's wavelet-tree levels in blocks left a part that did not shrink: six bits a block for the block's class, its count of ones, which on English words became a quarter of the levels. Coding the classes by their runs was predicted to halve them. It takes a third off, 0.480 bits a character to 0.323, because two thirds of the blocks are uniform but they come in stretches of about two. One entropy code over the classes does about as well, and on the repetitive collection better. Runs also cut the reads a rank makes by 15%. On random letters runs make the classes larger, and the pointer a variable-length stream needs at every superblock eats most of what an entropy code saves.
A sixth of what, exactly
The saving from a counting-only reverse half is 30.5% at one sampled position in four and 14.0% at one in a hundred and twenty-eight. A number quoted without its sampling rate is a number about a setting somebody chose.
A flag for the blocks that change once
A coded wavelet-tree level stores each mixed 63-bit block as the index of its arrangement, as if every arrangement were equally likely. Coding it instead as the gaps between its value changes was predicted to take a third off the offsets on English words. Alone it adds 4%: a third of the mixed blocks change value once, as predicted, but most of the rest change twenty times or more. A one-bit flag that lets each block take the shorter code takes 21% off, and the levels fall from 1.72 bits a character to 1.45. On the repetitive collection the offsets halve.
The saving, spent
A bidirectional index whose reverse half cannot locate is a sixth smaller. Give that sixth back to the half that does locate, and the same total size answers a locate five times faster.
The optimum the flag took away
A bidirectional index's wavelet-tree levels were coded in blocks of 63 bits because a sweep found 63 the best length: shorter blocks pay for more classes, longer ones for offsets that approach the block's own length. Then the classes were coded as runs and each offset given a flag to choose a code of its value changes instead. Lengthening the blocks to 127 bits was predicted to save on English words and the repetitive collection and to change nothing on random letters. It saves on all four texts, and there is no best length short of 1,023 bits on three of them: English words fall from 1.431 bits a character at 63 to 1.304 at 1,023. Under the codings the optimum came from, the same levels cost 3.728 at 1,023. The flag turned a long block's offset into the cost of its change points, which no block length moves, and left the length to be set by the one thing still growing with it: the work of a rank.
Every child at once
A bidirectional extension counts the occurrences of every symbol smaller than the one being added, and this collection did it with one rank per symbol — 139.7 operations an extension on a twenty-six-letter alphabet. One walk down the tree gives the same number.
A flag that weighs the work as well as the bits
A wavelet-tree level's mixed blocks are each coded by their arrangement's index or by the gaps between their value changes, a flag choosing the shorter, and longer blocks then save bits on every text at the price of more work a rank. Charging the flag for decoding work as well as bits was predicted to reach the same curve as shortening the block, and to beat it by no more than a hundredth of a bit a character. On English words, DNA and protein it reaches nothing below that curve. On the repetitive collection it reaches 0.156 bits a character below it: 1,023-bit blocks with a price on work cost 0.458 bits a character for less work a rank than 63-bit blocks do at 0.607. The difference is in which blocks the flag codes by their arrangement — on English words blocks that change value ninety-six times, on the repetitive collection blocks holding one short run of ones in two hundred and fifty-five bits.
A price the levels share
A bidirectional index's wavelet-tree levels code each mixed block by its arrangement or by its gaps, and a flag that charges each code for the work a rank costs inside it reached below the curve the block lengths draw on one text of four. Setting that price separately for each level of the tree was predicted to find a few hundredths of a bit on English words, where the deep levels hold runs. It finds nothing on any text: under a ten-thousandth of a bit beyond one shared price. The price a block wants is set by the text, not by the depth of the node it sits in, and every level holds a fifth of the index.
A schedule nobody writes down
A search scheme is valid when its searches between them cover every way the errors can fall — ten cases at two errors, enumerable in a line. A scheme missing one of them finds seven occurrences of eight, all real, at the right error counts, and reports nothing wrong.
The blocks the searches ask for
A flag that codes each block of an index's wavelet-tree levels as an arrangement or a gap code, charged for the work a rank does inside the block, was priced as if ranks landed on every position equally, and a price for each level found nothing that one price did not. Pattern searches do not ask evenly: on English words the busiest tenth of the blocks takes 22% of their ranks, and 35% when common patterns are asked more. Priced by the work of the ranks that actually land in each block, chosen on one sample of searches and charged on another, the flag reaches 0.11 to 0.16 bits a character below the old pricing at the same work on English, and 0.03 on the repetitive collection.
A factor of fourteen, for four per cent
The compound walk removes a factor of σ from every interval extension. It requires a tree whose leaves are in order, which costs between nothing and five per cent of the wavelet tree. Both numbers on one plate, and the trade is not close.
A layout the traffic does not move
A bidirectional index whose bit vectors code each block as an arrangement or a gap code saved bits at equal work once the choice was priced where a workload's ranks land. Traffic changes, so a layout fitted once was predicted to lose most of that gain when the searches moved from one half of a text's commonest patterns to the other's, and an index that recoded its blocks every thousand searches from their own counts was predicted to keep it. Neither happened, because nothing needed to be kept. Priced on the first half's patterns and never changed, the layout does 8.24 units of in-block work a rank on the second half's, 1% above a layout refitted to them for free; recoding every thousand searches rewrites 48 of 385 blocks, 0.71 bits a character, and ends no better. The busy blocks are where the text's common substrings sort, and both halves share them: 27 of the 39 busiest blocks under one half's patterns are among the 39 busiest under the other's.
Flat in the budget, and not
One saving is eleven times at every error budget, because it is a property of the alphabet. The other moves between ninety-eight and a hundred and five, because it follows the share of extensions that find nothing. Two savings, two shapes, and neither line crosses the other.
Three savings on one structure
A factor of eleven on an extension, a factor of seven on a branching search, and a sixth of the bits. Applied to one bidirectional index they do not give a factor of seventy-seven, and the reason is that two of the three are the same saving.
One set, three orders
The symbols an interval holds do not depend on the tree's shape. The order they come out in does, and a search that accumulates a running count as it reads them computes a plausible number that is wrong on eighty per cent of queries.
The ladder, and the rung that spends
Two hundred and fifty thousand bits, then two hundred and twenty-six, then two hundred and eleven. The fourth rung takes the whole saving and buys a four-times denser sampling with it, landing at ninety-six per cent of where it started and locating several times faster.
The floor was the marks
A saving reported as about a sixth of a bidirectional index, falling to an eighth and levelling off. Represent one array properly and it falls to a fiftieth instead — most of what was being dropped was a badly encoded bit vector.
Two factors that do not multiply
Eleven times and seventy-eight times against the same baseline, so an index with both should be eight hundred and sixty. It is seventy-eight, and the shortfall is eleven — the first factor, exactly, because the second operation already contains it.
The saving that is a loss
An operation that is seventy-eight times cheaper on a branching search costs twice as much on an exact one. It reports every symbol present in order to hand back the one that was asked for, and a search that knows its character needs none of the rest.
Bits and steps on one frame
The size falls from ninety-nine per cent to eighty-six as the sampling thins, and the walk to a sampled position rises from two and a half steps to sixty-four. Neither line is the answer; the answer is a point on the pair.
Named alongside it
The objects these essays reach for when they reach for this one.
Index sizeWavelet treeRankSpace accountingCompressed bit vectorRank queryBackward searchFM-indexPredictionSearch schemeTradeInterval