What a heavier tail actually buys

A decay measured from where it started ended with a curve and a suspicion. The curve put exponential decay, backward polynomial decay and a restarted forward counter on one pair of axes — how quickly each follows a change in the arrival rate, against how much its estimate wanders while nothing is changing — and the three sat on top of each other. Swept over their own parameters they traced nearly one line. The conclusion there was that the shape of a weight function does not buy a better trade, and only the timescale matters.

The suspicion was that the measurement had been aimed at the wrong thing. A step in the rate moves the arrivals a decay is currently averaging over, so a step is a question about the bulk of a weight function. The reason anybody reaches for a polynomial weight is the opposite case: one event, over in a second, that should still register long afterwards. Nothing on that page could see it, and the prediction left behind was that a polynomial tail would keep a large burst detectable several times longer than an exponential of the same spread.

It does not. At the burst size that page would have used, the four schemes are within a factor of two of each other and the exponential is not the worst of them. The prediction is wrong in the way that is worth being wrong: the quantity it named is real, it is just not the quantity the tail decides.

What has to be held fixed #

A decay that remembers longer sees a burst for longer, and is also quieter, because it averages over more arrivals. So a comparison between two weight shapes at two parameters somebody chose is a comparison between the parameters. Every number on this page is taken with the schemes matched on the thing that is not in question.

The spread is taken across streams at one instant, not along one stream. Two readings of the same decayed counter a second apart share nearly all of their arrivals, so a standard deviation taken along a stream measures the sampling interval as much as the estimator. Forty streams at four instants, and the spread is the mean of the four.

The four timescales that come out are not comparable as numbers and are not meant to be. A restart period of 112 seconds sounds enormous beside a half-life of 15, and it is: a forward counter’s landmark is somewhere between half a period and a period back, and its half-weight age is 29% of that, so it is averaging over something like twenty seconds. What the calibration produces is four settings at which the four estimators are equally uncertain, and from there on nothing separates them except the shape of the weight. Three of the four are also lazy in the sense the fading nobody computes measured — no counter is touched except on the way past an arrival or a reading — and the fourth is not lazy at all, which turns out to be the same fact as its state being unbounded.

The exponent chosen for the polynomial is α = 2 throughout, except where the essay says otherwise. The fourth scheme is the one a monitoring system would actually deploy if it wanted a tail: a weighted sum of three exponential counters, at half-lives spaced by a factor of eight, with the weights fitted by least squares to the polynomial’s own curve.

That fit is worth a sentence, because the obvious way to do it is wrong. Fitted by unweighted least squares over a wide range of ages it comes out at 809% worst relative error with a negative weight on the longest exponential — the residuals at small ages, where the target is of order one, swamp everything past a few seconds and the tail is fitted to nothing. Taken on the relative residual instead, and with the weights constrained not to go negative, three exponentials reproduce the polynomial to within 26% over the ages that matter. Twenty-six per cent is not close, and the plates below show where that slack goes.

Four shapes at one quietness #

The crossing is the whole geometry of the comparison. Before 77 seconds the exponential weights an old arrival more heavily than the polynomial does, because matching the spread forced the polynomial to spread its weight over a longer span and therefore to put less of it anywhere near the present. After 77 seconds the ordering reverses and never comes back. So a heavier tail is not a free addition to a weight function; it is bought from the recent past, and the calibration is what makes that visible.

The restarted forward counter is a different object from the other three. Its weight does not fall to a small number at the restart period. It falls to zero. Once the landmark has moved past an arrival, that arrival is not downweighted — it is gone, along with the counter that held it, and no amount of arithmetic recovers it. A decay measured from where it started needed the restarts to stop a fixed landmark’s memory growing with the stream’s age, and this is what they cost.

One burst, four descents #

The stream is Poisson at ten arrivals a second and the burst is three hundred extra arrivals inside one second. The background is drawn first and in full, and the burst is added to it afterwards, so a run with the burst and a run without it have identical background arrivals. A difference between them is the burst and nothing else.

That pairing is not only tidy. Every estimator here is a linear functional of the arrivals, so the estimate on a stream with a burst in it equals the estimate on the background plus the estimate on the burst alone. It is checked against the direct computation to fifteen decimal places, and it is what makes the burst-size sweep below affordable: the background — which for exact polynomial decay is a sum over every stamp the stream has produced, at every lag — is computed once per stream, and each further burst size costs a few hundred arrivals instead of fourteen thousand.

When is it still there? #

A burst is visible while the estimate is above what the same scheme does when there is no burst. That needs a threshold, and the threshold is read off the no-burst runs rather than assumed, at each lag separately, at a false-alarm rate fixed at 5%. Nothing here supposes the estimates are Gaussian, and they are not: the polynomial’s and the mixture’s distributions have their own tails.

Sixty-three seconds against sixty-eight. The exact polynomial, keeping every stamp the stream has ever produced, beats the one-number exponential counter by seven per cent. That is the prediction refuted, and it is refuted in the direction that matters: seven per cent is not worth a structure whose state grows without bound.

The mixture at 104 seconds is the odd one, because it is supposed to be an approximation of the scheme it beats by half as much again. The 26% fit error is where it comes from. Three exponentials cannot follow a power law exactly, and over the range where they are fitted they alternately overshoot and undershoot it; the longest of the three, at a half-life of 323 seconds, leaves more weight at a hundred seconds than the polynomial does. It is not standing in for the tail there. It is a different tail that happens to have been fitted nearby, and it is heavier.

That is worth separating from a claim about accuracy, because the mixture is less accurate as an approximation and better as a detector. A sketch that is allowed to be under drew the same distinction for an estimator whose error was one-sided by construction: the useful question is not how close a structure is to the thing it approximates but which direction it errs in when it errs, and against which question.

The restarted forward counter’s shape is the one worth looking at rather than reading off. It does not decline towards the threshold. It sits at complete detection and then, over about one sampling interval, stops.

The measurement the previous page should have made #

One burst size is one point, and the four horizons at one point are close enough that their ordering could be an accident of the seeds. Sweeping the burst size is what separates them, and it separates them completely.

The lines have different slopes because they obey different laws, and both laws follow from one line of algebra. A burst of $B$ arrivals contributes $B \cdot w(L)$ to the estimate at a lag of $L$, where $w$ is the scheme’s own weight at age $L$. It is detectable while that contribution is above a threshold $\theta$ set by the no-burst spread, which does not move with the lag. Setting the two equal:

$$B\,2^{-L/H} = \theta \;\Rightarrow\; L = \frac{H}{\ln 2}\,\ln\frac{B}{\theta}, \qquad B\left(1 + \frac{L}{s}\right)^{-\alpha} = \theta \;\Rightarrow\; 1 + \frac{L}{s} = \left(\frac{B}{\theta}\right)^{1/\alpha}.$$

A logarithm against a power. At any one burst size the two can be made to agree by choosing $H$ and $s$, and the matched-spread condition roughly does agree them; no choice of timescale can make the growth laws agree, because the timescale is a multiplier and the law is an exponent.

The exponential’s measured slope is 23.5 seconds an e-fold of burst size, against the $H/\ln 2$ of 22.1 seconds the law predicts — six per cent high, with the fit at $r^2 = 0.984$. That constant is not a new one. $H/\ln 2$ is the equivalent window the counter with no window in it computed from the steady state, the 1.44 H that made a decayed counter agree in the mean with a windowed count. It turns up here as the rate at which a burst’s visible lifetime grows with its size, which is a different question about the same weight.

The exponent, at four exponents #

A law with a parameter in it can be falsified by moving the parameter, and this one has α.

Four exponents, four predictions, and the worst disagreement is 8%. The scales the calibration chooses move by a factor of seven across the four — 8.6 s at α = 1.5 up to 57.9 s at α = 4 — because a weight that falls faster has to start further back to average over the same number of arrivals. The horizons at a fixed burst size therefore barely move at all, and the slopes move by a factor of nearly three. That is the finding in one sentence: the exponent of the tail is invisible in the horizon and is the whole of the growth rate.

The open rings are a warning about the fit rather than about the schemes. Taken through $\log L$ rather than $\log(1 + L/s)$ the same measurements give 0.56 where the prediction is 0.50, and the 12% is entirely the $+1$. It matters here because these horizons are only a few scales long — at α = 4 the longest is 226 seconds against a scale of 58 — and a law read off a log–log plot without its offset would have been judged wrong at every α. A limit is not a prediction found the same shape of error in a complexity fit: an asymptotic form fitted over a range where the asymptotics have not arrived reports the wrong exponent, confidently.

What the reach costs #

The middle row is the answer to the practical question. A tail is worth having when the events being watched are large, and the way to have one is three exponential counters, not a list of timestamps. At a burst of three thousand the mixture reaches slightly further than the object it approximates, on 232 bits against 696,000, and the reason it does is the same 26% fit error that made it win at three hundred. It is not a better approximation to the polynomial than the polynomial. It is a heavier tail, arrived at by accident, and a system that wanted a specific tail would have to fit it deliberately.

The cost of the exact polynomial is the one a window that is a duration priced for a sliding window: a stamp per arrival, at a width set by the clock’s resolution and the span. A window at least caps how many stamps it holds. Backward polynomial decay has no cap at all, because an arrival’s weight never reaches zero and dropping it is an approximation with no stated error. What a window costs in bits put the same trade on one axis for windowed counts and found the approximate structures winning by two orders of magnitude; the same gap is here, at 6,700 times.

The restarted forward counter is the cheapest of the three constant-state schemes to reason about and the only one with a hard ceiling on what it can see. Its horizon at a burst of thirty thousand is the same 112 seconds as at three thousand, and would be the same at three million.

A structure whose state grows with the stream is not automatically disqualified — a register that became a list found a case where the exact structure became cheaper than the approximate one once the question was narrowed to a window, because the window capped what exactness costs. Backward polynomial decay has no such cap, by construction: it is the one scheme here that has deliberately refused a window.

How heavy a tail may be #

There is a ceiling on the growth law, and it comes from the same weight function rather than from any measurement.

A backward polynomial counter on a stream at constant rate λ settles at $\lambda s/(\alpha - 1)$, which is the integral of its own weight. At α = 1 that integral diverges, and the counter does not settle at all: measured on a stream at ten a second with a scale of 17.24 s, it reads 360 two minutes in and 952 an hour in, on a stream whose rate never moved. At α = 1.2 it reads 296 and 589. At α = 2 it reads 154 and 182, the second within 5% of the $\lambda s/(\alpha-1)$ of 172 and the gap being warm-up rather than drift.

A counter with no steady state has no spread to be matched on, so it cannot appear on any plate here, and a monitoring system reading it would watch a number climb forever. That is not a limitation of this measurement: it is what a weight whose integral diverges means.

So every polynomial weight that can be used at all has α > 1, and its growth exponent $1/\alpha$ is therefore strictly below one. A burst’s detectable lifetime can never grow as fast as its size. Doubling the size of an event buys less than a doubling of how long it stays visible, for any decay with a steady state, and the closer a scheme comes to that ceiling the closer it comes to not having one.

Why the earlier measurement could not have found this #

It is worth being precise about why the trade curve on the page before this one was not merely underpowered but blind.

A step in the rate changes every arrival from a moment onwards. How quickly an estimate follows it is set by how much of the scheme’s weight sits in the recent past — the weighted mean age, roughly. How much the estimate wanders is set by how many arrivals effectively contribute — the square of the total weight over the sum of the squared weights. Both are integrals over the whole weight function, dominated by the region where the weight is large, and for any smooth shape that region is the first scale or two. Two weight functions can differ by a factor of a thousand at a hundred seconds and have nearly identical values of both integrals.

A burst asks about the weight at one age, and nothing else. The estimate rises by $B\,w(L)$ and the noise is unchanged, so the signal is a pointwise reading of the tail. That is why the same four schemes that were indistinguishable on the trade curve separate by a factor of four here, and it is why the separation only appears once the burst is large enough that the reading is taken far out.

There is a general rule in that. A summary statistic of a weight function cannot measure the weight function’s tail, and both of the statistics on the trade curve are summaries of the bulk. On average is not a number made the same complaint about reporting a mean when the distribution was the finding, and expected is not average made it about a guarantee that was true of a distribution and of no input. The same mistake at one level up: reporting two integrals of a shape when the shape was the finding.

What is settled and what is not #

Settled, by measurement on Poisson streams at ten a second with the four schemes matched to 5% spread: at a burst of 300 the half-detection horizons at a 5% false-alarm rate are 54, 63, 68 and 104 seconds for the restarted forward counter, the exponential, the exact polynomial and the three exponentials. At a burst of 9,000 they are 112, 156, 384 and 586. The exponential’s horizon grows as 23.5 seconds an e-fold of burst size against a predicted $H/\ln 2$ of 22.1. The polynomial’s grows as $B^{1/\alpha}$, measured at 0.654, 0.458, 0.310 and 0.246 against 0.667, 0.500, 0.333 and 0.250 for α of 1.5, 2, 3 and 4.

Settled, by construction: a restarted forward counter cannot see a burst older than its own landmark, so its horizon is capped by its restart period at every burst size — 112 seconds at 3,000 and the same at 30,000.

Settled, by arithmetic: every estimator here is linear in the arrivals, so a burst’s effect is exactly additive and the whole sweep is computed from one background pass per stream.

Not settled:

A burst that is not a spike. Every burst here is three hundred to nine thousand arrivals inside one second, which is a point mass at the scale of the horizons being measured. A rise that lasts a minute is a different object, and a scheme’s response to it is an integral of the weight rather than a reading of it.

More than three exponentials. The mixture wins by having a tail that is not quite the one it was fitted to. Whether a mixture fitted deliberately to a stated tail — more terms, or terms placed rather than spaced by a fixed factor — reaches further per bit than these three is not measured, and it is the design question a system would actually face.

Several bursts, and false alarms that are not independent. The false-alarm rate is held at 5% per reading, and a monitoring system reads continuously. The rate of firing at least once over an hour is a different quantity entirely, and the schemes’ estimates are strongly correlated from one reading to the next in a way that depends on their own timescales — which is to say, on the thing being compared.

The cost of the threshold. Every threshold here is read off sixty no-burst runs of the same background. A deployed detector has one stream and has to estimate its own quiet-state spread from that stream, which for a long-memoried scheme means waiting a long time.

Still open: the detector that does not know its own quiet #

The thresholds on this page come from an oracle — sixty independent runs of the background the burst was added to, which is exactly what a deployed system does not have. It has one stream, and the spread it must compare against is a property of an arrival process that is itself drifting.

The question that opens is whether a scheme’s own state can supply the threshold. An exponentially decayed counter and a second counter at a much longer half-life, read together, give a fast estimate and a slow one; the difference between them is a signal, and the slow one’s own variation over time is an estimate of the spread that signal should be judged against. That is one structure rather than a detector plus an oracle, and its whole state is two values and a stamp.

The measurement that follows builds that pair at a range of half-life ratios, calibrates the false-alarm rate from the stream itself rather than from repeated runs, and asks two things: how much horizon is lost against the oracle threshold measured here, and what happens when the background rate drifts slowly — since a self-calibrating detector must distinguish a burst from a drift, and the two differ only in their timescale, which is the one quantity every scheme on this page was matched on.