Every essay — page 13
What a bound is Counting The floors What the machine does Structures Two parameters The other axis When the algorithm flips a coin What the libraries do When it does not fit One pass, and no room The data that is not a number When the algorithm is a table The index that replaces the text What is taught wrongly Ladders Objects Search
When the algorithm is a table
A dynamic program's cost is settled before its input is touched, by how many distinct subproblems its recurrence has. The unit is the subproblem; the second number is how many must be held at once; and the methods worth knowing are the ones that compute more in order to keep less.
The order that has a depth
One hundred cells, filled in three orders, producing one table. Row order takes ninety-one steps and anti-diagonal order takes nineteen. Nineteen is not a property of the order — it is the longest chain of cells in the recurrence itself, no schedule can get under it, and every count taken until now was a total that could not see it.
A distance divided by a length is not a rate
Two substitutions turn "ab" into "ba", a distance of two over an alignment of two columns — a rate of 1.00. Deleting, matching and inserting also costs two, over three columns, for 0.67. Both are alignments of the same pair, the second has the better rate, and the optimal alignment is not the one that achieves it. Over every pair of strings up to three characters on three letters, 21% disagree.
The edit that reaches back two rows
Swapping two adjacent characters is one keystroke and costs two edits. Adding it as a fourth transition is four lines, it is what nearly everything ships, and the function those four lines compute is not the one they are named after. Over 1,600 pairs of short strings the two definitions differ on twelve, and the shipped one breaks the triangle inequality on twelve triples where the other breaks it on none.
The cells that were never worth having
Two three-hundred-character strings over twenty-six letters give a table of 90,601 cells, and 3,421 of them are pairs of positions whose characters agree. Only those can lengthen anything. A method that enumerates exactly those computes a twenty-sixth of the table — and on a two-letter alphabet it computes half of it and is worse than the table it replaced.
The matrix a corpus wrote
A substitution matrix is not a property of an alphabet. Fit one to four hundred pairs of sequences that rarely change and the dearest substitution costs five; fit the same model to four hundred pairs that often change and it costs two. Two hundred test pairs aligned under each matrix give different alignments in 115 cases — and a matrix fitted to eight pairs of the first kind moves 79 of them, from sampling alone.
The parameter plane has few answers
Sweep the cost of opening a gap against the cost of extending one over five hundred and seventy-six settings, and the optimal alignment of intention against execution takes four values — one of them at 571 of the settings. Under a linear model the plane divides into three wedges through the origin, because doubling every cost changes nothing and only the ratio is a parameter. Tuning an aligner is choosing a region, and most of the plane is one.
A triangle stored in a square
An interval table has a cell for every range of keys and nothing below its diagonal, and it can be stored as a square array, as packed rows, or as packed diagonals — the last matching the order it is filled in. On sixty-four keys, with every read replayed through a small cache, the square misses 39.7% of its reads, packed rows 38.8%, and packed diagonals 78.4%. Storing a table in the order it is written is storing it in the order it is not read.
The split scan cut into blocks
Every way of filling an interval table one cell at a time stops at about one cache miss per split point considered once the table outgrows the cache — 1.01 at 128 keys, whether the cells go by length, by rows, or in a recursive tiling. Cut each cell's scan into blocks instead, and apply a block of split points to a block of cells whose inputs are all in hand, recursively at every scale, and the same 357,760 split points cost 0.094 misses each. The fill is told nothing about the cache, blocks of one and of four do equally well, and it needs no extra memory, where storing the table twice gets to 0.151 by doubling it.
The bound the search finds for itself
A spelling checker that computes the full edit-distance table against every word in a 2,424-word vocabulary fills 156,714 cells for each misspelt query. Bound each table by the best distance found so far, and abandon it the moment a whole row exceeds that bound, and the same search fills 40,273 and finds the same words. Meet the candidates nearest in length first and it fills 26,203, starting a table for exactly the words a search that knew the answer in advance would start. The last factor of 1.7 is the price of not knowing, and it is largest when the misspelling is smallest.
The ties a rounded matrix makes
Measure how far each optimal alignment is from a tie — the smallest change to any one cost that makes another alignment win — and it predicts which alignments a refitted substitution matrix will move. A resample of the same corpus moves 30 of the 63 test alignments that sit on a tie and 3 of the other 137. A matrix fitted to a different divergence moves alignments far from a tie as well, and the prediction weakens to a chance of 0.62. And a third of the alignments were on a tie only because the matrix was rounded to whole bits — fitted without rounding, 15 of 200 are, and every prediction improves.
The columns the candidates share
Three thousand tables against one query, and most of them begin the same way. Stored as a trie, the 2,424-word vocabulary has 7,710 distinct prefixes holding 17,239 letters, and a search that computes one column per prefix reads 61,449 cells against 156,714 — before it applies any bound at all. Apply the bound at a prefix instead of at a word and it reads 16,958, beating a list search that was told the answer in advance.
The bound a block can and cannot have
Knuth's condition turns an interval table's cubic fill into a quadratic one by bounding each cell's best split between its two neighbours'. A blocked fill cannot use it a cell at a time, and the two cells that bound a block lie outside the block — one to its left, one below it. The schedule has finished both for ten per cent of the blocks, the bound then removes eleven per cent of the splits, and it removes half a per cent of the cache misses, because the splits it skips are the ones already in the cache.
The lattice that decides the ties
Rounding a fitted substitution matrix to whole bits puts 63 of 200 alignments on a tie where the exact fit puts 15. Rounding to half bits — a finer grain, and the obvious repair — puts 79. What tracks the ties is not how fine the lattice is but how many of the six fitted costs it keeps apart: whole and half bits both leave three, an eighth of a bit leaves all six, and matches the exact fit exactly.
A trie fitted to its queries
A nearest-word search over a trie of 2,424 words gives a column to about 30% of the prefixes each query, and it was proposed that a stream of queries would pick out a part of the trie worth keeping and a part worth collapsing. A hundred queries have visited 93% of it; two thousand, 99.5%. A stream does not choose a part of the trie. It chooses a distance: misspelt vocabulary words all lie within two edits, and a search that starts at the median past answer does a fifth of the work of one that discovers its bound. Misspellings of words the vocabulary lacks break that, unless the search tries the learned bound once and then gives it up.
The grain a corpus chooses for itself
A fitted substitution matrix can choose its own rounding lattice: the coarsest grain that keeps all six of its costs apart. On eight resamples of the same 400-pair corpus that grain comes out anywhere from an eighth of a bit to a forty-seventh, because it is set by whichever two costs happen to land closest, and those two differ by less than a tenth of their own standard error. Asked instead which costs it can actually tell apart, the corpus pools six into the generator's two and chooses an alignment the generator agrees with for every one of 200 test pairs on every resample.
A second try before any feature
A nearest-word search that learned one starting bound for a whole stream was proposed a bound for each query instead, read off features the search can compute before it walks. The proposed feature separates nothing: 58% of misspelt vocabulary words and 53% of words the vocabulary lacks begin with a real three-letter prefix. Length and unseen letter-triples do predict distance, and still buy about 1%, because the best start is two for almost every class. What closes half the gap to an oracle is a second cheap walk: one, then two, then discover.
The classes held-out pairs cannot rule out
A substitution matrix can choose how many distinct costs it has as part of the fit: try all 203 ways of grouping its six pairs into classes and keep the one that best predicts pairs it was not fitted to. On corpora from a generator with two costs, that choice finds the two on about a third of resamples, and no more often at 3,200 pairs than at 50. A spurious split gains the same bit or two of fit at every size, and held-out scoring charges it a fixed price. A penalty that grows with the corpus finds the two every time, and finds a third class half a bit away from 400 pairs.
A checker that stops at two edits
A nearest-word search that must always answer pays most of its price on the queries whose answer is useless: misspellings of words the vocabulary does not have, whose nearest vocabulary word can be twenty edits away. Capping the answer at two edits keeps every suggestion a misspelt vocabulary word could get and costs 42% of the uncapped search. What it does not do is stop the false suggestions. It declines 95% of unfamiliar words of eleven letters or more and 1% of those of three or four, because two edits is most of a short word, and 72% of what it still says to an unfamiliar word is wrong — against 77% with no cap at all.
A cost built from two properties
A substitution matrix fitted as a partition gives each class of pairs one cost, and the best number of classes can be chosen from the data. When the true costs are a sum of two terms — a transversion penalty and a graded difference between the letters — every pair costs something different, and the chosen partition's number of classes becomes a statement about the corpus: two classes at 25 sequence pairs, four at 200, five at 1,600. A model with the two terms and three parameters predicts a fresh corpus better than any partition at every size, and gets 191 of 200 alignments right from 50 pairs where the chosen partition needs 3,200. Told the wrong property it is worse than the plain two-class matrix, and nothing in its own fit says so.
The triples a typo writes
A spelling checker capped at two edits keeps every correction it could make and still gives a false suggestion to 72% of the unfamiliar words it answers, because on short words distance cannot tell a typo of a vocabulary word from a word the vocabulary lacks. The query's letters were proposed as the second test: decline a query whose letter-triples the vocabulary has never seen. They cannot separate the two either. One edit writes about one unseen triple into any word, familiar or not, while an unfamiliar word brings 0.59 of its own, and two thirds bring none. Declining at two unseen triples removes a quarter of the false suggestions and costs half the corrections — twenty-two corrections for every false suggestion removed.
The counts only short words carry
A spelling checker capped at two edits still gives a false suggestion to many words its vocabulary lacks, and the one evidence about a writer's intention it already holds is how often each vocabulary word is used. Scored by its nearest words' counts, weighed a tenth for each edit, an answer was predicted to separate a slip from an unfamiliar word on short queries. On the streams the checker was built against, whose typos are of words drawn evenly from the list, the false suggestions go to commoner words than the corrections do, and the test gives up sixteen corrections for each false suggestion it removes. Drawn as a writer mistypes, in proportion to use, the counts separate the two at last — and still trade worse than a plain cap of one edit, except on words of four letters or fewer, where a threshold removes a ninth of the false suggestions for one correction in ninety-six.
The data that is not a number
A string comparison costs characters, and how many depends on what the two strings share. A symbol stream repeats, and repetition is the only thing any compressor has ever used. Both are invisible to a counter that charges one for a comparison, and both change which algorithm wins.
The comparison that is not one comparison
Sorting 512 keys costs 3,955 comparisons whatever the keys are, and between 7,849 and 134,409 character examinations depending only on how much those keys have in common. The first number is the one every bound so far is stated in. The second is the one the machine pays, it grows without limit, and nothing here has ever counted it.
The shift the pattern already knows
On a text of 8,192 characters the naive scan examines 520,256 of them and Knuth–Morris–Pratt examines 16,321 — a factor of 32, and 16,321 is 99.6% of the 2n that no input can push it past. On ordinary random text the same two algorithms examine 20,862 and 20,833. Both measurements are of the same pair of algorithms and only one of them is the reason anybody uses the second.
The text that answers without reading it
Boyer–Moore–Horspool finds every occurrence of an eight-character pattern in a twenty-thousand-character text while examining 2,985 characters. Not 2,985 comparisons of eight characters each — 2,985 characters, 0.149 per character of text. It is a correct algorithm returning a complete answer about a text it has mostly not looked at, and the reason it can is a property of the alphabet rather than of the algorithm.
The model is the compressor
One stream of 32,768 symbols has an entropy of 3.886 bits per symbol, and 2.243, and 1.186, and 0.991, and 0.909. All five numbers are correct, all five are floors, and nothing about the data changed between them. The only thing that changed is how many preceding symbols the model was allowed to look at — which makes the entropy of a file a property of a decision rather than a property of a file.
The shift a set of patterns allows
Aho-Corasick reads every character of the text exactly once, whatever the number of patterns. Commentz-Walter reads backwards inside a window and steps over what it can, and on eight patterns of ten characters it looks at sixty-two per cent of a twenty-thousand-character text. The rule that does the skipping is not the one everybody implements.
The transform that emits nothing
The Burrows–Wheeler transform outputs exactly the characters it was given, in a different order. Its zeroth-order entropy is therefore identical to its input's, to fifteen decimal places, and by that measure it has done nothing at all. A Huffman coder handed the result spends 1.935 bits per symbol where the same coder on the same data spends 4.209, and the difference is entirely in the order.
A distance that is a path through a grid
How far apart two strings are is a shortest-path problem on a grid whose every edge is drawn by the recurrence — and finding a string in a text costs 4,988 character comparisons where measuring how far it is from one costs 96,000.
The filter that feeds the table
A self-index answers exact queries and nothing else. Approximate matching needs a table with twenty thousand columns in it. The pigeonhole joins them — cut the pattern into k+1 pieces and at least one occurs exactly, and the index that cannot answer the question decides where to ask it.
One pass for every pattern at once
Sixty-four patterns, one text, twenty thousand characters read. Running this site's KMP once per pattern reads 1,595,445 — and an implementation missing one kind of link finds fewer occurrences, reads exactly as much, and looks better on every counter.