Generator

External merge sort, measured in block transfers

Rendered here at the parameters it defaults to, with every essay that calls it — which is the same list as the blast radius of changing it.
External merge sort, measured in block transfersEvery point is a real sort of n random values through a modelled memory, with the transfers counted as they happen rather than computed from a formula. The steps are the passes: each time the number of initial runs crosses a power of the fan-in, the whole file is read and written once more. At M = 256 the fan-in is 7 and the sort takes 4 passes at the largest size here; at M = 1024 it takes 3. More memory does not make the passes cheaper, it makes them fewer.10³10⁴10010³n (elements)block transfersM = 256 · fan-in 7M = 1024 · fan-in 31B = 32, M as labelled4 passes against 3

External merge sort, measured in block transfers

Every point is a real sort of n random values through a modelled memory, with the transfers counted as they happen rather than computed from a formula. The steps are the passes: each time the number of initial runs crosses a power of the fan-in, the whole file is read and written once more. At M = 256 the fan-in is 7 and the sort takes 4 passes at the largest size here; at M = 1024 it takes 3. More memory does not make the passes cheaper, it makes them fewer.

Drawn at 700 × 420, wide on the page. Everything above is what sort-passes returns with no arguments; the caption is the generator's own, computed from the numbers in the drawing rather than written beside it.

9 essays call sort-passes. The drawing above is what it returns with no arguments at all; every call below passes it something, because a placement that passes nothing draws whichever member of the family the generator happens to default to rather than the one its essay argues about — which is what optcheck and figfill exist to catch.

Where it is called

Changing this generator changes every one of these figures.

10³10⁴10010³n (elements)block transfersM = 256 · fan-in 7M = 1024 · fan-in 31B = 32, M as labelled4 passes against 3 When it does not fit

Sorting what will not fit

10⁴10010³n (elements)block transfersmeasured sortthe bound3.20×2.67×2.29×2.94×2.67×2.40×B = 32, M = 512 (M/B = 16)3.20× the floor at worst The floors

The floor under moving data

1010010³10⁴B — elements to the blockblock transfersthey cross near B = 4one element at a timesorted by destination, 2passesM = 4,096, B as drawnthe lower bound is the smaller of the two, and it is proved rather than measured When it does not fit

Permuting is the harder problem here

10×10³10⁴the key side, as a multiple of memoryblock transfersblock nested looppartitioned hash joinsort–merge joinB = 64, M = 512 (M/B = 8)the other relation: 16,384 rows When it does not fit

Two ways to join, and the ratio that decides

10⁵10⁶10⁷10³10⁴inversions in the permutationblock transfers to carry it outsort by destination, 1,536w 8w 32w 128w 512w 2048w 819216 swaps64 swaps256 swaps1024 swapsshuffled inside windowsa few pairs swapped farn = 16,384, B = 64, M = 512 (M/B = 8)inversions do not order the cost When it does not fit

The permutation that moves almost nothing

10×20×50×00.40.811.2skew of the join columnratio, logarithmicestimate off byregret, decided partregret, whole planR 4,000, S 40,000, T 2,000 rows, B = 64, M = 4,096 (M/B = 64)every order writes the same output When it does not fit

The join order is a guess

10×20×50×100×200×00.250.50.7511.251.51.752skew of the join columnregret of the decided part, logarithmicuniform estimate4 counters a side16 counters a side64 counters a side256 counters a sideR 4,000, S 40,000, T 2,000 rows · 64-record blocks, 4,096 in memoryMisra–Gries on each side of the join column When it does not fit

The skew a few counters cannot repair

1M2M4M16M64Mmean run length, in memoriesrandom33 runssorted1 runreversed64 runssorted, 1% arriving late2 runssorted, 10% arriving late7 runs262,144 records, 4,096 in memorydashed: two memories When it does not fit

Runs twice as long as memory

01251025501002505000.00010.0010.010.1110100how far S's ranking is rotated from R'strue size ÷ estimatethe uniform estimate8 counters a side8 counters, using their silenceexactZipf 2, 8 counters a sideabove the line: the estimate is too low When it does not fit

A summary's silence is a bound, not a count

The library, page 4 of 5 — where sort-passes sits