Fitting — where it appears
Named by 13 essays across 6 fields — each of them below, with the objects they name alongside it.
The entropy that cannot see a copy
Two copies of a text have exactly the same symbol statistics as one, so every entropy on this site doubles when the second copy arrives and the second copy carries no information at all. The number of runs in the Burrows-Wheeler transform is 224 at two copies and 224 at thirty-two.
The matrix a corpus wrote
A substitution matrix is not a property of an alphabet. Fit one to four hundred pairs of sequences that rarely change and the dearest substitution costs five; fit the same model to four hundred pairs that often change and it costs two. Two hundred test pairs aligned under each matrix give different alignments in 115 cases — and a matrix fitted to eight pairs of the first kind moves 79 of them, from sampling alone.
What a second pass buys
Exact selection of a median from thirty-two thousand values needs the whole stream in one pass — a million bits — and eleven thousand in two. By nine passes it is three hundred and twenty. The state falls as n to the power one over p, which is a law with an exponent worth fitting, and on skewed data the deterministic rule misses it by three orders of magnitude.
The ties a rounded matrix makes
Measure how far each optimal alignment is from a tie — the smallest change to any one cost that makes another alignment win — and it predicts which alignments a refitted substitution matrix will move. A resample of the same corpus moves 30 of the 63 test alignments that sit on a tie and 3 of the other 137. A matrix fitted to a different divergence moves alignments far from a tie as well, and the prediction weakens to a chance of 0.62. And a third of the alignments were on a tie only because the matrix was rounded to whole bits — fitted without rounding, 15 of 200 are, and every prediction improves.
A floor with two variables in it
Under round-robin a Space-Saving summary's floor is 0.0203·n^1.018 over a hundred-and-twenty-eight-fold range of shard size, worst residual 2.7%. Under hashing the same measurement has no exponent at all — the local slope runs from n^5.17 to n^1.19 — and a least-squares line through it reports n^1.73 at a 441% residual.
The filter that proposes everything
Seed-and-extend saves two thousand times the work at zero errors and costs more than doing nothing at four. Between them the selectivity falls through the floor, and where it falls is set by two numbers that can be computed before the filter is run — one of which does not contain the length of the text at all.
The lattice that decides the ties
Rounding a fitted substitution matrix to whole bits puts 63 of 200 alignments on a tie where the exact fit puts 15. Rounding to half bits — a finer grain, and the obvious repair — puts 79. What tracks the ties is not how fine the lattice is but how many of the six fitted costs it keeps apart: whole and half bits both leave three, an eighth of a bit leaves all six, and matches the exact fit exactly.
The grain a corpus chooses for itself
A fitted substitution matrix can choose its own rounding lattice: the coarsest grain that keeps all six of its costs apart. On eight resamples of the same 400-pair corpus that grain comes out anywhere from an eighth of a bit to a forty-seventh, because it is set by whichever two costs happen to land closest, and those two differ by less than a tenth of their own standard error. Asked instead which costs it can actually tell apart, the corpus pools six into the generator's two and chooses an alignment the generator agrees with for every one of 200 test pairs on every resample.
The classes held-out pairs cannot rule out
A substitution matrix can choose how many distinct costs it has as part of the fit: try all 203 ways of grouping its six pairs into classes and keep the one that best predicts pairs it was not fitted to. On corpora from a generator with two costs, that choice finds the two on about a third of resamples, and no more often at 3,200 pairs than at 50. A spurious split gains the same bit or two of fit at every size, and held-out scoring charges it a fixed price. A penalty that grows with the corpus finds the two every time, and finds a third class half a bit away from 400 pairs.
What a heavier tail actually buys
Four decays, each tuned until it is equally quiet on a stream with nothing happening, then given one burst. At a burst of three hundred they see it for 54, 63, 68 and 104 seconds — indistinguishable. At nine thousand the exact polynomial reaches 384 seconds and the exponential 156. The shape of a tail does not decide how far back a scheme can see. It decides how fast that distance grows with the size of the event, and the exchange rate is the reciprocal of the exponent.
A cost built from two properties
A substitution matrix fitted as a partition gives each class of pairs one cost, and the best number of classes can be chosen from the data. When the true costs are a sum of two terms — a transversion penalty and a graded difference between the letters — every pair costs something different, and the chosen partition's number of classes becomes a statement about the corpus: two classes at 25 sequence pairs, four at 200, five at 1,600. A model with the two terms and three parameters predicts a fresh corpus better than any partition at every size, and gets 191 of 200 alignments right from 50 pairs where the chosen partition needs 3,200. Told the wrong property it is worse than the plain two-class matrix, and nothing in its own fit says so.
A line the corpus finds
Substitution costs built as a transversion penalty plus the distance between the two letters' values on a graded property beat every partition of the pairs, when the model is told the property, and fall below the plainest matrix when told the wrong one. Fitting each letter's position on a line instead was predicted to find the property's order from about four times the told model's data and then behave like it. It finds the order on every corpus of 100 sequence pairs or more but one, and every miss starts with the two letters whose values differ by a fifth of a bit. But knowing the order is not knowing the costs. Its alignments are no better than six free classes' until 800 pairs, sixteen times the data the told model needed for the same score, and on a generator with no graded property at all it finds a line anyway, a quarter as long as a real one at 200 pairs, and pays for it against the plain partition.
A line with a prior toward the classes
Substitution costs fitted as the letters' positions on a free line learn the letters' order quickly, and on a generator with no graded property they invent a line anyway. Charging the line for its length, at a rate growing with the corpus as the partitions' charge does, was predicted to collapse the invented line from 50 pairs and to cost the real one little. It does neither. On the generator with no property the charged line still has a median length of 0.12 bits at 3,200 pairs. On the generator with a property it takes the letters' order away at every size up to 400 pairs where the free line had it on sixteen seeds of sixteen. Charging the line for the number of gaps it keeps, the partitions' own rule, collapses the invented line on every seed at every size and matches the chosen partition to the bit — and then needs 1,600 pairs before it believes a real line is there.
Named alongside it
The objects these essays reach for when they reach for this one.
Honest limitEstimatorCost modelSubstitution matrixAlignmentCorpusOverfittingMeasurementParameter choiceModel selectionRoundingSampling