Field

The index that replaces the text

Every index measured here before this one was weighed at zero. A suffix array is four times the size of what it indexes and cannot answer without it; a compressed self-index is a third of it and hands the text back on request. The unit is the bit, the primitive is a rank on a bit vector rather than a comparison of two characters, and the text is taken away before any query is allowed to run.
suffix array + text311,29619.00 b/chcounter array per symbol5,407,710330.06 b/chFM-index, plain100,9476.16 b/chFM-index, compressed36,8042.25 b/chthe packed textone bar shaded darker needs the text · English-likesigma 21, sample 64

An index larger than what it indexes

What an index costs in bits, computed from the structure rather than from a serialisation — why a suffix array grows faster than its text, why it cannot be separated from it, and what the check that enforces the second one is.

0$abracadabra1a$abracadabr2abra$abracad3abracadabra$4acadabra$abr5adabra$abrac6bra$abracada7bracadabra$a8cadabra$abra9dabra$abraca10ra$abracadab11racadabra$aball 12 rotations of "abracadabra$", sorted0 character comparisons

A search that runs backwards

Backward search over a Burrows–Wheeler transform — the interval that narrows, the LF mapping it is built on, why the count arrives before any occurrence does, and why the cost does not depend on the length of the text.

a text that repeats itselfH0 4.01 · H3 0.236.291.48an order-1 sourceH0 3.00 · H3 0.895.151.67English-likeH0 3.89 · H3 0.976.162.25four symbols, uniformH0 2.00 · H3 1.994.102.59eight symbols, uniformH0 3.00 · H3 2.835.143.65bits per character of textupper bar: plain bit vectors · lower bar: compressed16,384 characters each · sample rate 64sigma 21, 8, 21, 4, 8

The index that is smaller than the text

Why a compressed self-index is compressed — class-and-offset bit vectors, the block length that turns out to have an interior optimum, and the lower-order term that is three-fifths of the structure.

1101001,00010010³position in the textLF stepsspan + sample = 64row sampling onlywith the secondsampling8,192 characters · sample one in 32 · span 32second sampling 3,598 bits

The sampling that goes the other way

Position-to-row sampling in a self-index — why extraction is asymmetric without it, what the second sampling costs, and why it is a second structure rather than an optimisation of the first.

1,00010,00010⁴10⁵characters in the collectionbitsFM-index, plainFM-index, compressedrun-length indexsample one in 64 · divergence 0 · r = 22411,900 bits at 32 copies

The index that stores the runs

The run-length FM-index — what it stores, how a rank becomes three counted operations, and where it crosses the entropy-bounded structure it replaces.

sampled positions6,99041%phi predecessor3,95223%run starts2,09712%run lengths per symbol1,97012%run heads1,5389%C table5003%English-like · n = 16,385 · r = 233the sampling is the two shaded rowsone unit = one bit17,047 bits · 1.04 bits/char

The sampling that follows the runs

Why a compressed index's last term proportional to the text is its sampling, and what replacing it with two suffix-array values per run actually costs.

patterns of 225%60 selectspatterns of 441%94 selectspatterns of 863%118 selectspatterns of 1682%117 selectsEnglish-like · 8 copies of 512share of steps needing no lookup40 patterns a row

The occurrence carried through the search

How an r-index knows one occurrence by the time its search finishes, what maintaining that costs, and why the samples it needs are at the ends of runs.

1,00010,00010⁴bits12481632characters in the collection · copies aboveFM-index, compressedr-indexphrase indexEnglish-like · divergence 0z 156 · r 233

An index with z in its size

What a self-index proportional to the phrase count stores, how it answers a query without a transform, and where it beats the two structures beside it.

10010³10⁴errors allowed, kacts0123index walkthe whole table3,000 characters · m = 20 · 4 symbolsno crossing in range

The search that spends a budget

What approximate matching looks like when it never leaves the index, what it spends, and why it is not a filter.

1,00010,00010⁴bits12481632characters in the collection · copies aboveFM-index, compressedr-indexphrase indexEnglish-like · divergence 0z 156 · r 233

The collection decides which index is small

Three self-indexes sized by three measures of repetition, with their query and extraction costs beside them, and how to choose between them from two numbers.

rank among the boundaries, sorted by the text before themsorted by the text after0 in the rectanglepattern "ss is un"split after 40 end with the left half0 begin with the rightEnglish-like · 2 copies of 512z = 156 · 0 crossing

The candidates a filter cannot avoid

What the intersection at the centre of an LZ77 index actually costs when it is done by filtering a range, and what shape the cost has as a collection grows.

11010⁴10⁵worst copy chain, phrases followedbits124816noneno capcap on each pointEnglish-like · 16 copies of 512z 156 to 7,351

What a ceiling costs in phrases

The measured trade between a bound on the copy depth and the size of the parse, across a dial, a collection and four kinds of text.

2·0·1·20101206·7·3·25·012012201012102rowsdocumentfirstpattern " was t" · 30 occurrences · 7 documentsA filled row is the first of its document. Reporting one costs a range minimum;reading every row costs one visit per occurrence.English-like · 8 documents30 rows · 7 documents

A list of documents is not a list of occurrences

The document listing problem, why it is not the occurrence problem with a post-processing step, and what answering it naively costs.

1234567characters addedoccurrencesgrown leftwardsgrown rightwards"tgatatt"4,000 characters · four symbols1 occurrences

An interval that grows at both ends

The index that can add a character at either end of the pattern, what keeps its two intervals in step, and what an extension actually costs.

plain51,600block-coded47,6687.6%run-coded60,805-17.8%plain, uniform51,600block-coded, uniform50,2422.6%run-coded, uniform67,121-30.1%bits · the lower three are a permutation with no structure3,612 points7.6% against 2.6%

A block, a class and an offset

The class-and-offset bit vector applied to a grid, what it saves, what a control says about how much of that is structure, and the select support it did not have.

run together867 bits a characterone separator877 bits a charactera separator each897 bits a charactergenerated, run together215 bits a characterdistinct symbolstwelve essays0.0% premium

Documents that are not the same length

The document strand's findings taken to real documents, which of them transfer exactly, and the headline that turns out to be a property of the generator's alphabet.

① prose12,33512① code13,1508① revisions7,77410② essays12,33582② captions2302,214② history12,96314documentscharacters in a document① the first freeze · ② the secondmedian marked

Two thousand documents of two hundred characters

A real collection of 2,214 short documents measured beside one of 82 long ones, with the document array, the separators and the join artefacts all recomputed on the shape document retrieval is actually about.

forward · wavelet tree18,377forward · rank directories5,037forward · C table100forward · sample marks8,193forward · sampled positions3,598reverse · wavelet tree18,377reverse · rank directories5,037reverse · C table100reverse · sample marks8,193droppedreverse · sampled positions3,598dropped8,192 characters · sampling every 3216.7% of both halves

The half that is never asked where

What the reverse half of a bidirectional index is actually used for, why its rows cannot be read as positions anyway, and which parts of an FM-index answer which question.

level 0 · left335 keptlevel 1 · right+207 smallerlevel 2 · left61 keptlevel 3 · right+33 smallerlevel 4 · left8 keptpositions still in play, and the half the code leaves behindsmaller symbols before position 400: 240rank of "m": 8 — from the same 5 operationsσ 21 · code 010105 ranks, not 105

Every child at once

The compound wavelet-tree operation that returns a rank and the count of smaller symbols from a single descent, measured against the loop it replaces across four alphabets.

range consideredminimum atC[at] < lonew document[476, 489)483 · document 4yesyes[476, 483)477 · document 1yesyes[476, 477)476 · document 0yesyes[478, 483)482 · document 2yesyes[478, 482)478 · document 0nono[484, 489)486 · document 3yesyes[484, 486)484 · document 0nono[487, 489)487 · document 2nono8 steps, 5 documents, and the two columns never differrows 476–489 · 8 documents8 steps

The array the walk never reads

What the test in Muthukrishnan's document listing is actually asking, why the answer so far can answer it, and what the array it appears to be about is doing in the structure.

051015204060documents, which here is also the copy countbits a characterFM-indexrun-length indexphrase index8,192 characters throughout2665 runs down to 427

The index that does not notice

The FM-index, the run-length index and the phrase index measured against the copy count of one collection, and the crossing that decides which one a corpus wants.

every occurrence5610 documentsrange minimum and chain32510 documentsrange minimum, no chain9610 documentsone descent over D3810 documentsthe document arraythe document array, the chain and a range minimumthe document array, a range minimum and one bit per documentthe document array, in a wavelet tree56 occurrences in 10 documentsoperations inside the structure

The tree answers the question

Document listing by an interval-symbol descent over the compressed document array, and the third structure this apparatus removed by noticing something already present answered its question.

acdefghijklmnopqrstuvwyzroot24 of 26 symbols present54 ranks · 51 nodes

Asking about symbols that are not there

The operation that enumerates an interval's distinct symbols in one descent, against the loop that asks about each symbol in turn, with the dead branches counted.

02040255075100125one sampled position in every …of the index, per centplain marks: 18.5%Elias-Fano: 3.8%16,384 characters14.9% off at one in 32

What the locating apparatus becomes

The share of a self-index that exists to answer where, measured across the sampling dial with the marks held two ways.

a loop over the alphabet1,478,400the compound walk, inside the loop134,40011xone descent per node18,91678x6 patterns · 1 error · sigma 2155.8% of the extensions were dead

Three savings on one structure

Three improvements to one index measured separately and then together, and what composes with what.

"t than"8 found · set 8 · frontier 4"of her"7 found · set 7 · frontier 4" that "16 found · set 16 · frontier 11" every"31 found · set 31 · frontier 19"the ev"8 found · set 8 · frontier 4"o of c"8 found · set 8 · frontier 5" than "15 found · set 15 · frontier 9"ime ra"7 found · set 7 · frontier 4the visited set, pale; the sweep's frontier, dark17,715 phrases examined either way1.67x on what is held

The same occurrences, less bookkeeping

A queue with a visited set against a text-order sweep, measured on the same queries, with the work held equal and the state not.

All essays