Series

Transfer — the series

23 essays on one idea, from the one that introduces it to the one that assumes the rest.
  1. block transfersIn order, 0 to n−11,0241.0× a scan · 64.0 elements per transferEvery B-th element (B = 64)65,53664.0× a scan · 1.0 elements per transferUniformly at random61,40760.0× a scan · 1.1 elements per transfera scan of this array is 1,024 transfersB = 64, M = 4,096 (M/B = 64)64× between the cheapest order and the dearest

    One access, eight kilobytes

    Every count on this site charges one for an array access. A machine charges for a block. The same 65,536 accesses cost 1,024 transfers in one order and 65,536 in another, with nothing about the algorithm's work changed — a factor of 64, which is exactly the number of elements in a block, and which no counter here could see until now.

    part 1 · applied
  2. levels on the path (outline) · transfers actually paid (filled)B = 411 levels · 6.99 transfers · 16,384 blocks residentB = 166 levels · 3.23 transfers · 4,096 blocks residentB = 644 levels · 1.99 transfers · 1,024 blocks residentB = 2563 levels · 1.30 transfers · 256 blocks residentB = 10243 levels · 1.01 transfers · 64 blocks residentbinary search over the same 4,194,304 keys: 20 transfersB = 1024, M = 65,536 (M/B = 64)20× between binary search and the widest tree

    A tree with nodes the size of a block

    A B-tree is a binary search tree that has read the hardware manual. Its node holds as many keys as fit in one transfer, so the height falls from log₂ n to log_B n — and the measured cost falls further still, to 1.01 transfers over four million keys, because the top of the tree is small enough to stay in memory. The comparison count goes up.

    part 2 · structures
  3. 10³10⁴10010³n (elements)block transfersM = 256 · fan-in 7M = 1024 · fan-in 31B = 32, M as labelled4 passes against 3

    Sorting what will not fit

    Merge sort's Θ(n log n) is a statement about comparisons and says nothing about a file larger than memory. Counted in transfers the answer is (n/B)·log_{M/B}(n/B), and the base of that logarithm is the number of blocks that fit in memory — so doubling the memory does not halve the work, it moves a staircase. The measured cost jumps by 32,768 transfers at one step and by nothing for the next four.

    part 3 · applied
  4. 1010010B (elements per block)block transfers per searchtuned for B = 64Sorted arrayB-tree tuned for B = 64van Emde Boas — told nothingM = 16,384, B as drawnone layout, 7 block sizes, no parameter

    The layout that is told nothing

    A B-tree is built around a block size somebody looked up. The van Emde Boas layout is given neither the block size nor the memory size, and across seven block sizes spanning a factor of 64 it tracks the best structure that was told them. An algorithm with no parameters making a claim at every level of the hierarchy at once is a strange thing to be able to measure, and this is what it costs.

    part 4 · applied
  5. elements of block written per key insertedB-tree, in place49.3Log-structured, T = 23.0 · 16× less than the treeLog-structured, T = 42.0 · 25× less than the treeLog-structured, T = 81.0 · 49× less than the treeLog-structured, T = 161.0 · 49× less than the treeB = 64, M = 4,096 (M/B = 64)25× between the two structures at T = 4

    The writes nobody counted

    Sixteen thousand keys inserted into a B-tree write 49.3 elements' worth of blocks for every key stored. The same keys into a log-structured store write 2.0. Every operation counter reports the two as the same work — the same insertions, the same comparisons, the same number of updates — and the factor of 24 decides which structure a storage engine is built from.

    part 5 · applied
  6. least recently usedthe offline optimumA loop over 7 blocks35 accesses35113.18×Random over 7 blocks35 accesses971.29×A straight sweep35 accesses35351.00×B = 8, M = 48 (M/B = 6)3.2× on the loop

    The block that is not a block

    This field's model has one memory, one block size, one processor, no prefetcher, no queue and no clock, and every number in it is a count of transfers rather than a duration. A real machine has five levels, reads ahead, issues a hundred requests at once, and charges four orders of magnitude more for one kind of transfer than another. What survives the difference is worth stating exactly, and so is what does not.

    part 6 · wrong
  7. 1100ε — the exponent the fanout is B toelements written per key33445679ε = 1 — the B-tree ·fanout 256 · 3transfers a queryε = 0.5 · fanout 16 · 5a querythe number above eachpoint is what a querycostsB = 256, M = 16,384 (M/B = 64)131,072 random keys

    One dial between two structures

    A B-tree writes 226 elements of block for every key stored and a log-structured store writes two. They are presented as rival designs. They are one design at two settings of an exponent that nothing in either description mentions, and every setting between them is available.

    part 7 · applied
  8. 1010010³10⁴B — elements to the blockblock transfersthey cross near B = 4one element at a timesorted by destination, 2passesM = 4,096, B as drawnthe lower bound is the smaller of the two, and it is proved rather than measured

    Permuting is the harder problem here

    Rearranging 65,536 elements into a stated order costs 63,601 transfers one at a time and 4,096 by sorting them into place. In the model every other field on this site uses, the first is the cheap method and beats the second by a factor of eight. The two models disagree about which problem is easy, and they disagree by about the same factor in opposite directions.

    part 7 · applied
  9. 11010010³10⁴1010010³rows matching the queryblock transfersn/B rows — where the arithmetic says they meetindex, rows scatteredindex, file in key orderread the whole fileB = 64, M = 4,096 (M/B = 64)plus 3 transfers to descend the index

    The index that is not worth reading

    An index turns a query over 65,536 rows from 1,024 transfers into four. At a thousand matching rows it costs 654 and still wins; at sixteen thousand it costs 1,027 and has lost. Where it turns is decided by the block size — a number the query does not contain, the schema does not mention, and nobody writing either has seen.

    part 8 · applied
  10. 10×10³10⁴the key side, as a multiple of memoryblock transfersblock nested looppartitioned hash joinsort–merge joinB = 64, M = 512 (M/B = 8)the other relation: 16,384 rows

    Two ways to join, and the ratio that decides

    The same join costs 260 transfers one way and 1,040 the other; at eight times the memory the same two costs are 2,880 and 1,280, the other way round. Neither number is a property of how large the tables are. The quantity that decides is how the smaller of them compares to memory, and a rule of thumb phrased in rows is a rule about somebody's machine.

    part 8 · applied
  11. 10⁵10⁶10⁷10³10⁴inversions in the permutationblock transfers to carry it outsort by destination, 1,536w 8w 32w 128w 512w 2048w 819216 swaps64 swaps256 swaps1024 swapsshuffled inside windowsa few pairs swapped farn = 16,384, B = 64, M = 512 (M/B = 8)inversions do not order the cost

    The permutation that moves almost nothing

    Two ways to scramble sixteen thousand elements. Shuffling them inside windows of five hundred and twelve puts two million pairs out of order and costs 3,095 block transfers to carry out. Swapping a thousand pairs across the whole array puts seven million out of order and costs 1,189. Inversions are the textbook measure of disorder, and on a disk they rank these two backwards.

    part 8 · applied
  12. 0.020.050.10.20.5121235810filter bits a keywasted reads per absent lookupno filters: all 3 levelssame rate every levelrates sized to levelsn = 65,536, size ratio 4, first run 1,024 keysthe same memory at every point

    The filter each run carries

    A log-structured store turns every lookup for a missing key into a read of every level, and a Bloom filter on each run buys those reads back with memory. Spread five bits a key evenly across three levels and a missing key still wastes 0.279 reads. Give the small levels more bits and the large one fewer — the same memory — and it wastes 0.201. At four levels the gap is 0.382 against 0.209, because sized filters stop the waste growing with the number of levels.

    part 8 · applied
  13. 11010010³10⁴12510rows the query actually matchescost ÷ the better plan's costrho plus the descent, 4.01estimate ÷64estimate ÷8estimate ×8estimate ×64n = 65,536, B = 64, M = 4,096 (M/B = 64), scattered read ×4dotted: the estimate switches at 512 rows

    The estimate a plan rests on

    A planner chooses between an index and a scan on how many rows it thinks will match, and the number it has is wrong by a factor. Guess sixty-four times too many on a narrow query and the scan it picks costs 13.5 times the index. Guess sixty-four times too few on a wide one and the index costs at most 4.01 times the scan — a ceiling that can be named before any query runs.

    part 9 · applied
  14. 10×20×50×00.40.811.2skew of the join columnratio, logarithmicestimate off byregret, decided partregret, whole planR 4,000, S 40,000, T 2,000 rows, B = 64, M = 4,096 (M/B = 64)every order writes the same output

    The join order is a guess

    Three tables, two orders, and an estimate of the first intermediate result that assumes the join column is uniform. When the column is skewed the estimate is out by seventy-two times, and the plan chosen on it costs 1.49 times the better one — which sounds tolerable until the shared output is taken away, and the part of the cost the order actually decided turns out to be 43.9 times worse.

    part 9 · applied
  15. 10×20×50×100×200×00.250.50.7511.251.51.752skew of the join columnregret of the decided part, logarithmicuniform estimate4 counters a side16 counters a side64 counters a side256 counters a sideR 4,000, S 40,000, T 2,000 rows · 64-record blocks, 4,096 in memoryMisra–Gries on each side of the join column

    The skew a few counters cannot repair

    A join order chosen on the textbook estimate costs 243.9 times the better order at a Zipf exponent of two, and two counters a side are enough to fix it. At an exponent of one half the estimate is out by less than a factor of two, the plan it picks costs 1.37 times the better one, and no number of counters up to 256 changes that. The easy case is the extreme one, and the reason the moderate one is hard is a series that stops converging at exactly one half.

    part 10 · applied
  16. trusts the estimateinsured ×2insured ×8always the index11.5235expected regret, logarithmic · label: worst within three standard deviationsrho 14.42.51.11.0rho 215.17.82.22.0rho 413.57.54.04.0rho 1616.016.016.016.0σ = 1.5, median error e^-1, 65,536 rowsexact over the error distribution

    What insurance against an estimate costs

    A planner that trusts its row estimate expects to pay 1.057 times the better plan and risks 7.76. One that insures itself by halving its estimate before it decides expects 1.057 and risks 4.10 — the insurance is free. At a read ratio of sixteen the same insurance costs six per cent in expectation and makes the worst case worse. Whether a conservative planner is paying a sensible premium depends on two numbers the planner can measure and usually does not — its device's read ratio and the direction its own errors run.

    part 10 · applied
  17. 1M2M4M16M64Mmean run length, in memoriesrandom33 runssorted1 runreversed64 runssorted, 1% arriving late2 runssorted, 10% arriving late7 runs262,144 records, 4,096 in memorydashed: two memories

    Runs twice as long as memory

    Feed 262,144 random records through a heap that holds 4,096 and the sorted runs that come out average 1.94 memories — the snowplow's famous factor of two. At a fan-in of 63 that saves a merge pass at 262,144 records, and at two of fourteen sizes in all. Feed the same heap a sorted file with one record in a thousand out of place and it writes two runs instead of sixty-four. And it spends 19 comparisons a record doing so, on every input, where sorting the chunks spends five on sorted data. The factor of two is the least of what the method does.

    part 10 · applied
  18. 0%25%50%75%100%ln 2mean leaf fillrandom, even splits2,906 leavesrandom, rightmost-split rule2,949 leavesascending, even splits3,971 leavesascending, rightmost-split rule2,048 leavesdescending, even splits4,095 leavesdescending, rightmost-split rule4,095 leavesbulk-loaded from sorted keys2,048 leaves131,072 keys, leaves of 64each bar names its rule

    The keys that arrive late

    Insert 131,072 keys into a B+-tree in random order and its leaves end up 70.5% full; in ascending order, 51.6%; in descending order, 50.0%. The rule databases use to fix ascending inserts — split a full leaf at its right-hand end — fills them completely, and it does nothing for descending keys. Let one key in a hundred arrive late in an otherwise ascending stream and the rule's leaves fall from 100% to 53.4% full. How much of an index is empty is decided by the order its keys arrived in, and a trickle of disorder undoes the fix.

    part 10 · applied
  19. 12481632641282561executed queries the fit sawexpected regrettrusting the estimatefitted from the queriesthe best divisor, 1.5ρ = 4, σ = 1.5, 120 fits a pointlower is better

    What a planner pays to find out what to pay

    Insurance against a row estimate is set from the error's median and spread, and a running system knows neither — it has to fit them from executed queries. Fitted from one query the divisor costs 1.263 times the better plan against 1.215 for a planner that never insures at all, so learning is worse than not learning until about sixteen queries have run. The tail, though, is bought immediately: one observation already holds the worst case to 24.7 against 36.6.

    part 11 · applied
  20. 50%60%70%80%90%100%0%1%2%5%10%25%50%share of keys arriving latemean leaf filleven splitsrightmost-split rulesibling first, two into threeln 2131,072 keys, leaves of 64late keys arrive at a random later point

    The sibling a full leaf asks first

    The rule databases use to fix ascending inserts fills their leaves completely and collapses to 53.4% when one key in a hundred arrives late. A leaf that offers a key to a sibling before it splits, and splits two full leaves into three when neither will take one, holds 84.2% on the same stream — and is better with a trickle of late keys than without one, because a perfectly ascending stream has no sibling with room.

    part 11 · applied
  21. 01251025501002505000.00010.0010.010.1110100how far S's ranking is rotated from R'strue size ÷ estimatethe uniform estimate8 counters a side8 counters, using their silenceexactZipf 2, 8 counters a sideabove the line: the estimate is too low

    A summary's silence is a bound, not a count

    When both sides of a join are skewed towards the same values the textbook estimate is 399 times too low and the order it picks costs 244 times the better one; two counters a side repair it. Skew them towards different values and the same estimate becomes 254 times too high — and the counters have nothing to say, because no value is in both summaries. Using their silence instead repairs the middle of the sweep and makes the far end six times worse.

    part 11 · applied
  22. 234681016110size ratioblock transfers · levelsa range of 100 keyslevelsan absent point lookup1,048,576 keys, 5 bits a keyfilters answer one of these two

    The read a filter has no key for

    A Bloom filter on every run of a log-structured store turns a lookup for a missing key from a read of every level into a fraction of one — 0.72 transfers across eight levels at five bits a key, and 0.00027 at twenty. A range query over the same store reads nine transfers at five bits and nine at twenty, because a filter answers whether one named key is in a run and a range has no key to name.

    part 11 · applied
  23. 6080100how far a full leaf looks for room, in siblingsmean leaf fill, per cent12481664randomascendingdescendingascending, 1% late131,072 keys · leaves of 64 · fan-out 64a reach of 1 is the adjacent-sibling rule

    A key passed along the row

    A full B+-tree leaf that offers a key to its immediate neighbours before splitting leaves an ascending stream 67.2% full. Let it look one sibling further and the same stream fills its leaves completely, 2,048 leaves where there were 3,048, for about the same key moves. On random keys each doubling of the reach adds a few points of fill and more writes than it saves, and at the whole parent a key travels sixteen leaves on average. On a stream that is mostly in order the same reach costs almost nothing.

    part 12 · applied

All series