Field

When it does not fit

Every count in the other fields charges one for an array access, which is the right model for data in memory and the wrong one for data anywhere else. Here the unit of cost is the block, the model has two parameters that must be stated before anything is optimal, and the algorithms are the ones a database ships.
block transfersIn order, 0 to n−11,0241.0× a scan · 64.0 elements per transferEvery B-th element (B = 64)65,53664.0× a scan · 1.0 elements per transferUniformly at random61,40760.0× a scan · 1.1 elements per transfera scan of this array is 1,024 transfersB = 64, M = 4,096 (M/B = 64)64× between the cheapest order and the dearest

One access, eight kilobytes

The external-memory model with its two parameters stated — the block size B and the memory size M — and why the unit of cost that every analysis on this site assumed is the wrong one as soon as the data does not fit.

10³10⁴10010³n (elements)block transfersM = 256 · fan-in 7M = 1024 · fan-in 31B = 32, M as labelled4 passes against 3

Sorting what will not fit

External merge sort measured in block transfers, why the cost is a staircase in the memory available rather than a curve, and why the base of the logarithm is M/B.

1010010B (elements per block)block transfers per searchtuned for B = 64Sorted arrayB-tree tuned for B = 64van Emde Boas — told nothingM = 16,384, B as drawnone layout, 7 block sizes, no parameter

The layout that is told nothing

Cache-oblivious layout measured against a B-tree tuned for one block size, across seven block sizes — what a parameterless algorithm can claim, and the constant factor it pays for the claim.

elements of block written per key insertedB-tree, in place49.3Log-structured, T = 23.0 · 16× less than the treeLog-structured, T = 42.0 · 25× less than the treeLog-structured, T = 81.0 · 49× less than the treeLog-structured, T = 161.0 · 49× less than the treeB = 64, M = 4,096 (M/B = 64)25× between the two structures at T = 4

The writes nobody counted

Write amplification measured — an in-place B-tree against a log-structured merge tree — and why a read count and a write count are two resources rather than one.

1100ε — the exponent the fanout is B toelements written per key33445679ε = 1 — the B-tree ·fanout 256 · 3transfers a queryε = 0.5 · fanout 16 · 5a querythe number above eachpoint is what a querycostsB = 256, M = 16,384 (M/B = 64)131,072 random keys

One dial between two structures

The buffered tree, measured — how one exponent moves insertion cost from a whole block per key to a fraction of one, what it charges for that on every query afterwards, and where the trade stops paying.

1010010³10⁴B — elements to the blockblock transfersthey cross near B = 4one element at a timesorted by destination, 2passesM = 4,096, B as drawnthe lower bound is the smaller of the two, and it is proved rather than measured

Permuting is the harder problem here

Why moving elements one at a time costs one transfer each, why sorting by destination costs far less, where the two cross, and what a lower bound proved about both.

11010010³10⁴1010010³rows matching the queryblock transfersn/B rows — where the arithmetic says they meetindex, rows scatteredindex, file in key orderread the whole fileB = 64, M = 4,096 (M/B = 64)plus 3 transfers to descend the index

The index that is not worth reading

Three query plans priced in transfers, why the crossover the textbook arithmetic predicts is not the measured one, and what a clustered index changes about all of it.

10×10³10⁴the key side, as a multiple of memoryblock transfersblock nested looppartitioned hash joinsort–merge joinB = 64, M = 512 (M/B = 8)the other relation: 16,384 rows

Two ways to join, and the ratio that decides

Block nested loop, partitioned hash and sort–merge joins measured in transfers, why all three return identical answers, and why the choice between them is a ratio rather than a size.

11010010³10⁴12510rows the query actually matchescost ÷ the better plan's costrho plus the descent, 4.01estimate ÷64estimate ÷8estimate ×8estimate ×64n = 65,536, B = 64, M = 4,096 (M/B = 64), scattered read ×4dotted: the estimate switches at 512 rows

The estimate a plan rests on

One planner decision priced at the truth while its row estimate is held wrong by a fixed factor. Overestimates on narrow queries grow with the file, underestimates are capped by the read-cost ratio, and that ratio decides which mistake is the dangerous one.

10⁵10⁶10⁷10³10⁴inversions in the permutationblock transfers to carry it outsort by destination, 1,536w 8w 32w 128w 512w 2048w 819216 swaps64 swaps256 swaps1024 swapsshuffled inside windowsa few pairs swapped farn = 16,384, B = 64, M = 512 (M/B = 8)inversions do not order the cost

The permutation that moves almost nothing

Carrying out a permutation one element at a time, measured against the inversions it contains. The cost follows how far elements move relative to what fits in memory, not how many pairs are out of order, and two permutations can be ordered one way by inversions and the other way by transfers.

10×20×50×00.40.811.2skew of the join columnratio, logarithmicestimate off byregret, decided partregret, whole planR 4,000, S 40,000, T 2,000 rows, B = 64, M = 4,096 (M/B = 64)every order writes the same output

The join order is a guess

A three-way join ordered on an estimate that assumes a uniform join column, swept over skew. The regret over the whole plan stays under 1.5 while the regret on the part the order controls reaches 44 and then 244, because every order writes the same output and that shared cost hides the mistake.

0.020.050.10.20.5121235810filter bits a keywasted reads per absent lookupno filters: all 3 levelssame rate every levelrates sized to levelsn = 65,536, size ratio 4, first run 1,024 keysthe same memory at every point

The filter each run carries

Bloom filters on the runs of a log-structured store, with the same memory divided evenly or sized to each level. Sizing makes an absent lookup's wasted reads nearly independent of how many levels there are, while an even split pays for every level it adds.

10×20×50×100×200×00.250.50.7511.251.51.752skew of the join columnregret of the decided part, logarithmicuniform estimate4 counters a side16 counters a side64 counters a side256 counters a sideR 4,000, S 40,000, T 2,000 rows · 64-record blocks, 4,096 in memoryMisra–Gries on each side of the join column

The skew a few counters cannot repair

The uniform join-size estimate behind a three-way join order, repaired with Misra–Gries summaries of the join column, swept over skew and over the number of counters. A handful of counters removes a regret of 243.9 at high skew; at an exponent of one half 256 counters remove nothing and 512 are needed, because the sum of squared frequencies no longer has a small set of values carrying it.

trusts the estimateinsured ×2insured ×8always the index11.5235expected regret, logarithmic · label: worst within three standard deviationsrho 14.42.51.11.0rho 215.17.82.22.0rho 413.57.54.04.0rho 1616.016.016.016.0σ = 1.5, median error e^-1, 65,536 rowsexact over the error distribution

What insurance against an estimate costs

Four planners choosing between an index and a scan on estimates wrong by a log-normal factor, priced exactly over the error distribution. Insurance that shifts the crossing is free and halves the tail at a small read ratio, costs a third of the expected regret at a moderate one when estimates run low, and buys nothing at a large one — and when estimates run high it lowers the expected regret too.

1M2M4M16M64Mmean run length, in memoriesrandom33 runssorted1 runreversed64 runssorted, 1% arriving late2 runssorted, 10% arriving late7 runs262,144 records, 4,096 in memorydashed: two memories

Runs twice as long as memory

Replacement selection measured against loading, sorting and storing memory-sized runs for an external merge sort. Runs average two memories on random input, one run on sorted input and exactly one memory on reversed input; halving the runs saves a merge pass only in narrow windows of file size, while nearly sorted input collapses the run count and with it the merge.

0%25%50%75%100%ln 2mean leaf fillrandom, even splits2,906 leavesrandom, rightmost-split rule2,949 leavesascending, even splits3,971 leavesascending, rightmost-split rule2,048 leavesdescending, even splits4,095 leavesdescending, rightmost-split rule4,095 leavesbulk-loaded from sorted keys2,048 leaves131,072 keys, leaves of 64each bar names its rule

The keys that arrive late

A B+-tree with real node splits built from the same keys in several arrival orders, under even splits and under the rightmost-split rule, beside a tree bulk-loaded from sorted keys. Leaf fill runs from 50% to 100% and the leaf count by a factor of two; random insertion settles near ln 2; and one per cent of late keys in an ascending stream removes the whole benefit of the rightmost-split rule.

12481632641282561executed queries the fit sawexpected regrettrusting the estimatefitted from the queriesthe best divisor, 1.5ρ = 4, σ = 1.5, 120 fits a pointlower is better

What a planner pays to find out what to pay

The insurance divisor a query planner applies to its own row estimates, fitted from executed queries rather than given, measured in expected and worst regret against the number of observations — and across a plane of read ratio and error spread where learning ranges from paying at once to never paying.

50%60%70%80%90%100%0%1%2%5%10%25%50%share of keys arriving latemean leaf filleven splitsrightmost-split rulesibling first, two into threeln 2131,072 keys, leaves of 64late keys arrive at a random later point

The sibling a full leaf asks first

B*-tree redistribution at the leaves of a B+-tree, measured against even splits and the rightmost-split rule on random, ascending, descending and late-arriving key orders — with the extra sibling writes it costs counted per key.

01251025501002505000.00010.0010.010.1110100how far S's ranking is rotated from R'strue size ÷ estimatethe uniform estimate8 counters a side8 counters, using their silenceexactZipf 2, 8 counters a sideabove the line: the estimate is too low

A summary's silence is a bound, not a count

A three-way join whose two sides are skewed towards different values, swept from identical rankings to unrelated ones, with the uniform estimate, a Misra–Gries summary of each side, and a summary that also uses the values it does not hold — and why only one direction of error is correctable.

234681016110size ratioblock transfers · levelsa range of 100 keyslevelsan absent point lookup1,048,576 keys, 5 bits a keyfilters answer one of these two

The read a filter has no key for

Range queries over a log-structured store measured against the point lookups its Bloom filters were bought for, swept over the size ratio, the range's length and the filter's bits — and the size ratio a store should be tuned to once its queries are not all point lookups.

6080100how far a full leaf looks for room, in siblingsmean leaf fill, per cent12481664randomascendingdescendingascending, 1% late131,072 keys · leaves of 64 · fan-out 64a reach of 1 is the adjacent-sibling rule

A key passed along the row

Redistribution in a B+-tree of 131,072 keys widened from the adjacent siblings to a chain across the parent's children, at reaches of 1 to 64, on random, ascending, descending and nearly ascending insertion: the leaves' fill, the pages written per key, the trade between the two, and how far a key actually travels.

All essays