Concept

R-index — where it appears

A compressed self-index whose every part is a function of r, the number of equal-letter runs in the Burrows-Wheeler transform, including its suffix-array sampling. It is what removes a compressed index's last term proportional to the text, and it is larger than the array it replaces on any text whose transform has more than half a run per character.

Named by 11 essays across 5 fields — each of them below, with the objects they name alongside it.

sampled positions6,99041%phi predecessor3,95223%run starts2,09712%run lengths per symbol1,97012%run heads1,5389%C table5003%English-like · n = 16,385 · r = 233the sampling is the two shaded rowsone unit = one bit17,047 bits · 1.04 bits/char

The sampling that follows the runs

A run-length index over thirty-two copies of one text spends 11,286 bits on its suffix-array sampling and 5,605 on the transform it was built to compress. Sample at the run boundaries instead and the sampling is 10,942 bits that stop moving — two values per run, and a function that fills in everything between them.

indexes · Repeat
text position iphi(i)anchor: a run start25 of thema text that repeats itself · 4 copies of 32r = 26 · pieces 24

A function with r pieces

Computed at every one of eight thousand positions across four texts, a function defined on the whole suffix array agrees exactly with r−1 anchors and one addition. Anchor it at the successor instead of the predecessor — one character of code — and it disagrees at 506 of 800 positions while still returning plausible numbers.

machine · Index
patterns of 225%60 selectspatterns of 441%94 selectspatterns of 863%118 selectspatterns of 1682%117 selectsEnglish-like · 8 copies of 512share of steps needing no lookup40 patterns a row

The occurrence carried through the search

Backward search returns how many and not where, and every index on this site pays for the second question separately. Carrying one occurrence along with the interval costs a lookup on 75% of the steps for a two-character pattern and on 18% of them for a sixteen-character one, and it is what makes a run-boundary sampling usable at all.

indexes · Index
is·holds·no·the·the·of·of·rather·count·class·that·is·algorithm·o1111111211111151141131111111211122213111111212phrase lengths below each block · a shaded block is a literalEnglish-like · 64 charactersz = 46 · 17 literals

The phrases a text copies from itself

Four thousand characters of English-like text hold 604 phrases in the greedy parse and 1,219 runs in the transform. Thirty-two copies of one text hold 156 phrases and 233 runs. Two measures of repetition, neither of them an entropy, and they do not agree about which text is the more repetitive.

text · Parse
1,00010,00010⁴bits12481632characters in the collection · copies aboveFM-index, compressedr-indexphrase indexEnglish-like · divergence 0z 156 · r 233

An index with z in its size

Over thirty-two copies of one text, an index built on the parse is 8,892 bits, the r-index is 17,047 and the entropy-bounded index is 34,615. Over eight thousand characters of four-symbol text the same three are 7,844, 25,177 and 20,413, and the smallest of the three has changed places twice.

indexes · Parse
1,00010,000110occurrences12481632characters in the collection · copies aboveall occurrencessecondaryprimarypattern "ss is un" · z = 1561 primary · 31 secondary

The occurrences that cross a boundary

One pattern, thirty-two copies of a text, thirty-two occurrences. The search finds one of them and produces the other thirty-one by arithmetic, and the count it finds is the same one at two copies, at eight and at thirty-two — the searching does not grow when the answer does.

text · Parse
1,00010,00010⁴bits12481632characters in the collection · copies aboveFM-index, compressedr-indexphrase indexEnglish-like · divergence 0z 156 · r 233

The collection decides which index is small

Three compressed self-indexes over one text of five hundred characters measure 3,511, 7,285 and 10,974 bits. Repeat that text thirty-two times and the same three measure 34,615, 8,892 and 17,047 — the ordering has completely reversed, and nothing about any of the structures changed.

indexes · Repeat
10,000110sampling, bitssteps per occurrence1 in 21 in 41 in 81 in 161 in 321 in 641 in 128r-indexEnglish-like · 16 copies · r = 233pattern " time "

Every occurrence at the same price

A regular sampling has one dial and it moves two costs together — 1,495 bits at 63 LF steps an occurrence, 131,088 bits at none. A sampling at the run boundaries sits at 10,244 bits and one predecessor query, which is a point the curve reaches only at 69,649.

space · Repeat
1,00010,00010phrases followed12481632characters in the collection · copies aboveworst depthmean depthEnglish-like · z = 156median 18 · 90th 31

The character that costs a chain

The index over thirty-two copies is 8,892 bits and does not grow. Producing one character of the text it indexes costs 17.89 phrase-follows on average and 38 in the worst case, against 2.38 and 7 at one copy — the size stopped growing and the price of reading it did not.

space · Parse
English-like, 32 copies0.05x0.017 runs/charEnglish-like, 8 copies0.12x0.045 runs/chara text that repeats itself0.08x0.029 runs/charEnglish-like0.80x0.292 runs/charfour symbols, uniform2.07x0.753 runs/charthe whole suffix arraysampling bits, run boundaries / every valuen = 8,192

A sampling that costs more than the array

On four-symbol text the transform has 0.75 runs a character, so a sampling of two suffix-array values per run is one and a half values per position — 271,565 bits against the 131,088 that keeping every value costs. The structure built to remove a term proportional to the text is twice the thing it replaced.

wrong · Repeat
1001,00010characters in the wordruns · phrasesr, worse orderz, either orderr, better orderFibonacci words · two symbolsspread 1.50x to 3.17x

The measure that cannot see the alphabet

Take a Fibonacci word of 4,181 characters and transform it with a before b — six runs. Transform the same word with b before a — nineteen. The parse gives eighteen phrases either way, and the gap between the two run counts grows with every word in the family.

wrong · Parse

Named alongside it

The objects these essays reach for when they reach for this one.

MeasurementSelf-indexIndex sizeLocateRepetitionRun-lengthFM-indexTrade offBurrows-wheeler transformCompressibilityLempel ziv parsePhrase

All concepts