Concept

Tail — where it appears

The extreme end of a distribution, which holds few items and most of the range, so a promise about the whole says little about it. It holds few items and most of the range, so a promise about the whole distribution says almost nothing about it.

Named by 7 essays across 5 fields — each of them below, with the objects they name alongside it.

quantilepromisedone summarymerge of 8q = 0.5100.01418q = 0.920.0183q = 0.992.005overq = 0.9990.214overrank error, in items out of 20,000ε = 0.01 · high-biased · 8 shards, round2 of 4 quantiles over the single summary's promise

The promise that does not survive the tree

A high-biased quantile summary at ε = 1% answers the 99th percentile of twenty thousand values to within two ranks. Merge thirty-two of them and the answer is seventeen ranks out — inside the merged bound, eight times outside the one anybody was quoted.

wrong · Rank
0%25%50%75%100%1.5×shareheadroom over the mean occupancyoverflowingstanding idledrifting · 4.0 s window · 100 Hz47% overflow at the mean

Sized for a rate that does not hold still

A four-second window on a stream at a hundred arrivals a second holds four hundred items on average and between 105 and 2,169 when the rate moves. An allocation set at that average overflows at 47 per cent of instants while 35 per cent of it stands empty, which is the same decision failing in both directions at once.

practice · Window
0.1%1%10%100%0.50.750.90.990.999quantile asked forrank error ÷ (1 − q)Greenwald–Khanna7,080 bitshigh-biased56,064 bitst-digest5,952 bits8 streams per point · ε = 0.01, δ = 100denominator: the tail

An error measured against the answer

A quantile summary asked for the 99.9th percentile answered 9,694 where the truth was 256, and violated nothing — its promise was a rank error under one per cent of the stream and it delivered a tenth of one per cent. One per cent of the stream is a thousand per cent of the tail, and no amount of extra state changes that.

streaming · Rank
1,000248163264high-biased, α = 0.56low-biased, α = 0.56none-biased, α = 0.74tuples keptshards mergedlog-normal, σ = 1.2 — a latency distribution · 20,000 valuesα 0.56 / 0.56 / 0.74 · worst residual 6.5%

The cheap tail and the expensive merge

A summary whose tolerance tightens towards the tail keeps seven times the tuples of a plain one on a single pass, and after merging sixty-four shards it keeps three and a half times as many. The error function that buys a useful tail promise is also the one that pays most for never having the values in one place.

space · Rank
11010⁴10⁵worst copy chain, phrases followedbits124816noneno capcap on each pointEnglish-like · 16 copies of 512z 156 to 7,351

What a ceiling costs in phrases

A cap of sixteen costs one phrase of a hundred and fifty-six and halves the worst chain. A cap of four costs six times the phrases. The curve between them is flat at one end and vertical at the other, and the elbow is where a structure should be built.

indexes · Parse
Greenwald–Khanna7,008 bits802high-biased55,008 bits802t-digest6,144 bits675the truth, 802ε = 0.01, δ = 100 · 40,000 valuesq = 0.52

The digest that promises nothing

The t-digest is the quantile structure most widely deployed and the only one with no proven bound on its rank error at any quantile. Measured, it beats the structure that does have one — and on two clusters with a gap between them it returns 431.5, where the data holds nothing at all between 40 and 800.

wrong · Rank
73 characters long24 runsto depth 7270 characters long24 runsto depth 69longest: 73 × "-" at position 11,0162,685 positions below depth 20, 2,685 of them inside a runeight source modules · worst depth 78100.0% in runs

The deepest text is punctuation

Eight source files reach depth 78 in a parse, on a collection with no version history in it at all. The positions between depth 20 and depth 72 are the same 48 characters at every level, and every one of them is a dash in a comment separator.

wrong · Parse

Named alongside it

The objects these essays reach for when they reach for this one.

GuaranteeMeasurementRank errorTrade offGreenwald–KhannaQuantileState bitsHeavy tailMergeable summaryParameter choiceQuantile summaryShard

All concepts