A check as slow as what it checks

The rise the correction kept subtracting followed a corrected burst detector past the end of a rise. The detector keeps a fast decayed counter with a half-life of 13.3 seconds and a slow one with sixteen times that, 213 seconds, and alarms when their difference $d$ exceeds 1.85 times a spread it has learned. On a rising background $d$ sits at an offset, and the slope the slow counter already shows removed it by reading the rise’s slope off the slow counter and subtracting the offset that slope predicts. When the rise stops, the slow counter goes on climbing towards the new level for several of its half-lives, the estimate reads that climb as slope, and the subtraction outlives the rise. After a 40% rise the corrected difference fell to 1.46 spreads below zero and stayed more than half a spread below for seventeen minutes, and bursts of thirty arrivals in that trough were caught as rarely as 27% of the time.

Its closing section noticed that the detector holds two readings of the same quantity. The estimate predicts the offset the rise should put into $d$, and the recent mean of the uncorrected $d$ is that offset, measured. While the rise lasts they agree. After it ends the measured offset falls as the slow counter catches up, and the prediction does not. It proposed subtracting the smaller of the two, with the measured offset taken as a decayed mean of the uncorrected difference over one slow half-life. The prediction was that the trough would fall from 1.46 spreads to under half a spread, and that bursts of thirty in the trough would be caught at least as often as by the positive clock, the design that stops learning its scale while $d$ stays positive. It named the risk: a burst raises the mean of $d$, and the cap with it.

The cap as proposed does very little, for a reason that is the same reason the trough exists. A faster cap does most of what was predicted, and pays for it while the rise is still going. And the risk the section named turns out to be a weakness of the correction itself, cap or no cap.

The smaller of two readings #

Every stream is the earlier page’s: a Poisson background at ten arrivals a second, rising by 20% or 40% of that rate every 1,000 seconds until 2,000 seconds in and level from then on. The detectors are the earlier page’s three, all at R = 16 with thresholds at 1.85 spreads, the spread the detector learns from its own quiet: the scale held while alarmed, the positive clock, and the held scale with the slope correction, its estimate decayed over one slow half-life. Two capped corrections join them. Each keeps a decayed mean of the uncorrected difference, the fast counter minus the slow, floored at zero, and subtracts the smaller of that mean and the estimate’s predicted offset. The proposal’s cap averages over one slow half-life, 213 seconds. A faster cap averages over a quarter of that, 53 seconds. A slower one over four half-lives is reported where it differs.

A burst is a number of extra arrivals in one second at a stated lag after the rise ends, the kind of burst four decays tuned to the same quiet were first given, caught when the detector alarms within ten seconds of its start. Traces are means over 24 streams and catch rates over 60, and the same streams serve every design, so every comparison between designs is paired. The capped detectors never subtract more than the uncapped one, reading by reading, and adding a cap leaves the estimate’s own prediction untouched; both are checked.

A trough lifted by a sixth #

After a 40% rise ends, the cap averaged over one slow half-life lifts the trough from 1.46 spreads below zero to 1.24, and the difference still spends eleven minutes more than half a spread below zero. The prediction said under half a spread. After a 20% rise the cap does as little, from 0.84 to 0.72. The slower cap, averaged over four half-lives, does less still, 1.29 after the steeper rise.

The cap averaged over 53 seconds lifts the trough to 0.70 spreads after a 40% rise and to 0.48 after a 20% rise, and the difference is more than half a spread below zero for 71 seconds instead of 917. That is most of what the prediction asked for, from a cap a quarter as slow as the one it proposed.

Why the proposed cap is as late as the estimate #

Two hundred seconds after a 40% rise ends, the uncorrected difference shows 1.35 spreads of offset, the estimate predicts 2.29, the cap over one slow half-life allows 2.12 and the cap over 53 seconds allows 1.71. The proposal treated the decayed mean of $d$ as a measurement of the offset and the estimate as a prediction of it, and expected the measurement to be the more current of the two. Both are averages over about one slow half-life of the past. The estimate is the slow counter’s change over the last half-life, decayed over one more; the cap is $d$ itself, decayed over one half-life. Each reports the offset as it was a few minutes ago, and a few minutes ago the offset was nearly as large as during the rise.

The uncorrected difference falls fast once the rise ends, from 2.22 spreads to 1.35 in the first two hundred seconds, because the fast counter reaches the new level within a minute and the slow counter starts closing the gap. Any reading averaged over 213 seconds has taken in barely half of that fall by then. The cap binds, since it is a little below the estimate, but only a little, and the subtraction stays more than twice what $d$ is showing. That is the same lag that opened the trough. The counter with no window in it put the reason in one line: a decayed counter behaves like a window of about 1.44 half-lives on a steady stream, and after a change it reports a weighted memory of the old level, not the new one. A check taken with the same memory as the thing it checks cannot catch its delay.

The 53-second cap has a quarter of that memory. At 200 seconds it allows 1.71 spreads, still above what $d$ shows but well under the estimate, and by 400 seconds it is within 0.3 of $d$. It follows the uncorrected difference down closely enough that the subtraction can no longer push $d$ far below zero, which is what the trough was.

Small bursts in the trough #

Bursts of thirty arrivals from 200 to 600 seconds after a 40% rise ends are caught 63% to 85% of the time under the 53-second cap, against 27% to 43% uncapped, 43% to 48% under the proposed cap, and 43% to 55% under the positive clock. The proposal’s second prediction, at least as often as the positive clock, holds narrowly for its own cap at 200 and 300 seconds and fails at 400 and 600. It holds comfortably for the faster one at every lag.

After a 20% rise, bursts of twenty are caught 55% to 70% of the time under the 53-second cap in the same stretch, against 37% to 43% uncapped and 58% to 62% under the positive clock. The shallower trough leaves less to recover, and the faster cap recovers most of it. On a level background with no rise at all, each design catches a burst of twenty 71% to 73% of the time, so the faster cap brings the trough’s losses to within a few points of a background that never moved.

The held scale catches more than any of them, 83% to 93% of the bursts of thirty. It does that by never subtracting anything and sitting on top of an offset that keeps carrying its difference over the threshold, and on the level stretch after a 40% rise it alarms on 20.5% of its readings when no burst is there. The corrected detectors are the designs that alarm rarely; among them, only the fast cap keeps small bursts in the trough.

What the fast cap gives back is quiet. On the level stretch after a 40% rise it alarms on 4.1% of readings, against 1.2% uncapped and 1.8% under the proposed cap, and after a 20% rise on 4.7% against 2.2%. The trough was suppressing alarms as well as bursts. A detector biased below zero is a quiet detector, and part of what the uncapped correction looked like on the earlier page’s level-stretch plate was that bias.

What the cap costs while the rise lasts #

While a rise lasts, the 53-second cap subtracts only 62% of the true offset on a 5% rise and 89% on a 40% rise, against the uncapped correction’s 97% and 96%, and alarms on 7% to 11% of readings against 5% to 8%. The proposal expected the cap to cost nothing during a rise, since the two readings agree there. They agree on average. Reading by reading, both are noisy, and the smaller of two noisy readings of one number is a reading biased low. A cap with a short memory is the noisier of the two, so it binds whenever its noise dips below the estimate, and the subtraction falls short of the offset by an amount set by that noise.

The shortfall is largest on the slowest rise, where the offset is smallest. A 5% rise puts only 0.31 spreads of offset into $d$, and a 53-second mean of $d$ wanders by more than that, so the cap binds on many readings and takes away a third of a small correction. On a 40% rise the offset is 2.5 spreads, the noise is a smaller share of it, and the cap takes away a tenth. The extra alarms follow: a rise under-subtracted is a rise partly left in $d$, which is the problem the correction was built to remove.

So the cap trades in both directions along the same axis. A long memory agrees with the estimate during the rise and lags with it after the rise ends. A short memory follows the end of the rise and wanders below the estimate during it. The clock a burst cannot outlast made the same trade with a clock in place of a cap: anything that can tell a rise has ended quickly can also mistake noise for an ending.

A large burst is read as a rise #

Two minutes after a burst of 300 arrivals in the trough, a burst of thirty is caught on 3% of streams under every correction, against 28% to 77% when the large burst was not there; five minutes after, the uncapped correction still catches 2%, the proposed cap 5%, and the 53-second cap 47%. The risk the closing section named was that a burst would lift the cap. It does, but that is the smaller effect. The larger one is in the estimate. A burst of 300 raises the slow counter by about one arrival a second within a few seconds, a step its change over the last half-life reads as a steep rise, and the predicted offset climbs, from 2.07 spreads to 2.82 two minutes later on the uncapped detector. The detector subtracts a rise that is a burst, and a small burst behind it starts nearly a spread deeper than the trough already put it.

The cap cannot stop that, because the burst raises the uncorrected difference as well, so the mean of $d$ that would cap the subtraction rises with the estimate. The fast cap recovers sooner for the same reason it lifted the trough: its memory of the burst fades in a few minutes, and by five minutes the subtraction is a sixth of a spread where the uncapped detector’s is two. The positive clock has its own version of the loss, for a different reason. A large burst keeps $d$ positive long enough for its schedule to relearn a wide scale, and a burst of thirty under that scale is caught 5% to 18% of the time.

The slope the slow counter already shows measured what reading a burst as slope costs the burst itself, and found at most 3.7 seconds of detection. It did not measure what it costs the next burst, and that is minutes. A burst detector whose correction is a slope estimate has a blind interval after every large burst. It is about as long as the estimate’s memory, because the estimate has to forget the burst before it stops subtracting it.

What the prediction got right #

The closing section predicted that capping the subtraction by a mean of $d$ over one slow half-life would close the trough after a 40% rise to under half a spread. It lifts it from 1.46 to 1.24, because a mean of $d$ over one slow half-life is as late as the estimate it was meant to check: both remember the rise for a few minutes after it ends. It predicted bursts of thirty in the trough would be caught at least as often as by the positive clock. Under that cap they are at two of four lags.

A cap with a quarter of the memory meets both predictions, 0.70 spreads and 63% to 85% of the bursts, and pays for it during the rise: it subtracts 62% to 89% of the offset where the uncapped correction subtracts 96%, and alarms on a few per cent more readings, because the smaller of two noisy readings is biased low.

The risk the section named is real and larger than it supposed. A large burst raises the cap, but it raises the estimate more, and the correction then subtracts the burst as if it were a rise, for as long as its estimate remembers it. No cap built from the same counters removes that, since the burst is in the cap’s reading too.

What the measurement leaves out #

Two cap lengths. The caps average over 213 and 53 seconds, with a slower one over 852 noted once. A cap between the two would sit between them on every plate, and nothing here finds the best one, which depends on how a system weighs small bursts after a rise against false alarms during one.

One burst size behind another. The large burst is 300 arrivals and the small one thirty, both one second long, 200 seconds into the trough of a 40% rise. A large burst on a level background, with no trough to deepen, would cost the next burst less, and that was not measured.

The correction’s own estimate is unchanged. The cap checks the estimate; it does not make the estimate ignore a burst. An estimate that discounted the readings taken while the detector was alarmed would attack the blind interval directly.

Still open: an estimate that does not read the alarms #

The blind interval after a burst comes from the estimate reading the burst as slope. A detector already knows when it is alarmed, and a reading taken while it is alarmed is a reading the burst is in. The estimate could skip those readings: hold its slope while the detector alarms, and for one slow half-life after, the time the burst takes to leave the slow counter’s change over a half-life, and resume reading the slope only once the burst can no longer be in it.

The measurement that follows gives the uncapped and the 53-second-capped corrections that hold and repeats the second-burst plate, the trough and the rising backgrounds. The prediction is that two minutes after a burst of 300 the second burst is caught as often as with no first burst, the 28% and 77% the dashed lines show, since the subtraction would not have moved. It could fail on the rise itself. A rise is also a stretch in which the detector alarms more often than usual, 5% to 11% of readings here, and an estimate that skips every alarmed reading, and the half-life after each, may stop reading the slope at all on a steep rise, where it is most needed. The question is whether a detector can learn its background from only the readings in which it has seen nothing, or whether a background that rises fast enough is always partly hidden behind the alarms it causes.