A hold that never lets go
A burst detector over a stream of arrivals keeps a fast decayed counter and a slow one, sixteen times slower, and alarms when their difference is more than 1.85 times a spread it has learned from its own quiet. The counters are the exponential kind the counter with no window in it introduced, which what a heavier tail actually buys found sees a modest burst as far back as any heavier-tailed decay does. A burst lifts the fast counter long before the slow one notices, so jumps. A background that rises slowly lifts too, by an offset proportional to the slope, and the slope the slow counter already shows removed that offset by reading the slope off the slow counter’s own change over one of its half-lives, decaying that reading into an estimate, and subtracting the offset the estimate predicts. On a steady rise the corrected difference sits centred on zero and the detector stays quiet.
The trouble starts when the background stops rising, which is what the rise the correction kept subtracting followed. The slow counter goes on climbing toward the new level, the estimate reads that climb as slope, and the subtraction outlives the rise. A check as slow as what it checks capped the subtraction by the offset itself shows, a decayed mean of the uncorrected difference, and found the cap over 53 seconds lifted most of the trough. But it found a weakness neither cap touched. A large burst steepens the slow counter too, so the estimate reads the burst as a rise and, for minutes afterwards, subtracts an offset that hides the next, smaller burst: two minutes after a burst of 300 extra arrivals, a burst of 30 was caught 3% of the time.
Its closing section proposed the obvious remedy. A detector knows when it is alarmed, and a reading taken while it is alarmed is a reading the burst is in. The estimate could skip those readings: hold its slope while the detector alarms, and for one slow half-life after, the time a burst takes to leave the slow counter’s change over a half-life, and only then resume. The prediction was that two minutes after a burst of 300 the second burst would be caught as often as with no first burst. The named risk was the rise itself: a rise is also a stretch of more frequent alarms, and an estimate that skips every alarmed reading, and a half-life after each, may stop reading the slope at all on a steep rise, where it is most needed.
Three detectors, held or not
Everything is the earlier essays’: a Poisson background of ten arrivals a second, rising by a stated share of that rate every thousand seconds until 2,000 seconds in and level after, or rising throughout; detectors at R = 16 and a threshold of 1.85 spreads; the slope estimate decayed over one slow half-life. Three corrections are run — uncapped, capped over 213 seconds and capped over 53 seconds — each with its estimate read on every reading, as before, or held.
The held estimate is held whenever the previous reading alarmed or the last alarm was less than one slow half-life, 213 seconds, ago. While held, no new reading of the slope enters it; it keeps whatever value it had. Everything else, including the learned spread and the cap, runs as before. A burst is a number of extra arrivals in one second, caught if the detector alarms within ten seconds of its start. Catch rates are over sixty streams and paired: the same streams serve every design.
The burst it no longer misses
Held through every alarm and a slow half-life after, the estimate subtracts almost nothing after a burst: two minutes after a burst of 300 a burst of 30 is caught 65% of the time against 3% read always — but not as often as with no first burst, 85%, which the prediction said it would be. At first sight the hold has done what it was for. The second burst, invisible to the detector that reads always for three minutes after the first, is caught two times in three from a minute after it.
The prediction’s own number misses by twenty points, and the dashed lines say something odder. With no first burst at all, the held detector catches the second burst 82% to 90% of the time, where the detector that reads always catches it 70% to 88%. The held detector is better at catching a lone burst in the aftermath of a rise than the one it was meant to improve. A design that skips readings should not be better on readings it does not skip.
The two oddities have one explanation, and it is the risk the proposal named.
Alarms on every rise
The risk arrives in full: on a background that keeps rising, holding the estimate lifts the alarms from 7%–11% of readings to 14%–64%, 64% on a 40% rise. On the steepest rise the held detector alarms on almost two readings in three. It is no longer a burst detector; it is a rise detector that cannot be turned off.
The read-always detector’s 7% to 11% is what a detector that learns its own quiet and the slope correction were built to achieve, false alarms near the rate the threshold implies whatever the background does. The hold undoes that, and undoes it most where the correction was most needed: the steeper the rise, the larger the offset, and the more readings alarm. A 5% rise only doubles the alarm rate; a 40% rise multiplies it by six.
So the held estimate is failing at its main job while it succeeds at its side job. The next plate shows how badly.
Held almost always, learning nothing
Held, the estimate is held on 97%–100% of readings on every rise and recovers −45% to 2% of the offset it exists to remove, where read always it recovers 95%–97%. It is not a slow estimate or a noisy one. It is an estimate that, once a rise has begun, almost never reads anything again, and keeps whatever value it had before the rise, which is near nought or, on the gentlest rise, a little negative from the noise of the quiet.
That explains both oddities in the first plate. The held detector subtracts almost nothing after a burst because it subtracts almost nothing at all: it never learned the rise, so it has no rise to over-subtract when the rise ends, no trough, and no burst-inflated estimate. It catches a lone burst after the rise better than the read-always detector because the read-always detector is in its trough there and the held one never had one. And it still misses a third of the second bursts because its spread, learned from a difference full of unremoved offset, and its alarms, firing on that offset, leave it in no state to see a burst of 30 clearly.
The held detector is, in effect, the detector with no slope correction, plus a mechanism that guarantees it stays that way.
How the hold locks in
The hold locks itself in: on a 20% rise the held estimate is held on over half its readings from minute 2 and never lets go, because the rise it has not learned keeps the detector alarming, and every alarm restarts a hold of a slow half-life. The first minutes of every rise alarm for both detectors, because no estimate can know about a rise before it has seen some of it. The read-always estimate learns from those minutes and the alarms fall away. The held estimate refuses those minutes, because they are alarmed, so it learns nothing, so the alarms continue, so it goes on refusing.
The mechanism is a feedback loop with the wrong sign. An estimate that corrects for a disturbance must learn from readings in which the disturbance shows; a rule that discards every reading in which the detector sees something discards exactly those. A hold of a slow half-life after each alarm makes the loop absolute: the alarms on a rise come more often than once every 213 seconds, so the hold never expires.
The proposal’s risk said the estimate might stop reading the slope on a steep rise. It stops on every rise, including a 5% one, because even a gentle rise produces one alarm per 213 seconds, and one is enough.
One stream, around a burst
On one stream the burst lifts the subtraction read always to 2.83 spreads for minutes, while the held estimate subtracts at most 0.00 — not because it skipped the burst, but because it never learned the rise before it. The trace makes the paradox concrete. The read-always estimate does exactly what the earlier essay found: it reads the burst’s steepening of the slow counter as slope and subtracts nearly three spreads for minutes, which is why a burst of 30 two minutes later is invisible to it.
The held estimate’s line is flat at nothing. A reader of this plate alone would say the hold worked, since it did not read the burst. But the trace starts a minute before the burst, more than two minutes after the rise ended, and the held estimate is already subtracting nothing there: it has been held through the whole rise.
That is why the comparison the prediction set up, the second burst against the same burst with no first one, cannot test the hold. Both arms are run on a detector that never had a correction to protect.
What the second burst sees
Two minutes after the burst every estimate read always subtracts 2.20–2.82 spreads, more than it would with no burst; the held estimate subtracts 0.07 with the burst and 0.08 without — almost nothing either way. The burst adds 0.75 to 1.03 spreads to every read-always subtraction, which is the damage the earlier essay found. The 53-s cap limits it best and still leaves 2.20.
The held line’s two bars are equal and near nought. The hold makes the burst’s damage zero by making the subtraction zero, and a fix that removes the problem by removing the thing that had it is not a fix. The question the earlier essay posed, how to subtract a rise’s offset without subtracting a burst’s, is untouched.
What a detector cannot learn from its quiet
The proposal asked whether a detector can learn its background from only the readings in which it has seen nothing, or whether a background that rises fast enough is always partly hidden behind the alarms it causes. The measurement answers more strongly than either alternative: it cannot learn a rising background from its quiet at all, because a background it has not yet learned is never quiet. Any rise produces alarms before the estimate catches up, and a rule that waits for quiet before learning waits for ever.
The principle reaches well past burst detectors. Any adaptive estimate that excludes the readings its own alarm flags inherits its alarm’s errors as blind spots. If the alarm fires because the estimate is wrong, excluding those readings removes exactly the evidence that would correct it. A detector that learns its own quiet got away with learning its spread only while not alarmed, because a spread that is too small makes the detector alarm more, the alarms are excluded, and the spread is learned from the remaining quiet, which is the right quiet. A slope that is too small also makes the detector alarm more, but the alarmed readings are the only ones that carry the slope. Holding the spread is safe; holding the slope is self-defeating.
The distinction between the two is what the earlier essays’ cases could not show, since each held one thing. What separates a safe hold from a lock-in is whether the quantity being held is one the alarms are evidence about. A burst’s arrivals tell the detector nothing about its quiet spread; a rise’s arrivals are the only evidence of the rise.
Down-weighted, not skipped
The hold treats a reading as all-or-nothing: either the estimate learns from it or it does not. The failure suggests the alternative a statistician would reach for first, which is to keep reading every reading but to stop any one of them from moving the estimate far. A burst and a rise differ in exactly the way robust estimators are built to exploit. A burst of 300 arrivals in one second puts a large, brief excursion into a few minutes of slope readings and leaves the rest untouched; a rise moves every reading a little, in the same direction, for as long as it lasts. A median of recent slope readings, or a mean that clips each reading at a few spreads before averaging it, would treat the first as an outlier and the second as the trend.
That is not what this essay measured, and it has its own failure to watch for. A slope reading here is the slow counter’s change over one of its half-lives, so a burst does not produce one outlying reading but a run of 213 of them, each carrying part of the burst; a median over a window shorter than that run would follow the burst as faithfully as the mean does. The window would have to be longer than a slow half-life to outvote a burst, and then it would be slow to follow the end of a rise, which is the trough the whole line of essays began with.
The measurement does settle what any such design must not do. It must not make the estimate’s learning conditional on the detector’s verdict, because the verdict depends on the estimate. A robust estimate reads every reading and decides for itself how much each counts; a held estimate lets the detector decide, and the detector is wrong for exactly as long as the estimate is.
What the measurement leaves out
One hold length. The hold is one slow half-life after the last alarm, as proposed. A shorter hold would expire between alarms on a gentle rise and let the estimate learn; how short it must be, and what it then gives back after a burst, was not measured.
One stream shape. The rises are linear in rate and level off abruptly. A rise that starts gently gives the read-always estimate time to learn before the alarms begin, and might give a held estimate a few quiet readings to start from.
Held means frozen. While held, the estimate keeps its old value. An estimate that decayed toward nought while held, or toward the cap’s measured offset, would fail differently.
Paired streams, one burst size. Every burst pair is 300 then 30. A first burst small enough not to alarm would not start a hold, and a hold that only large bursts trigger is a different design.
Still open: a hold with a ceiling
The hold locked in because a rise’s alarms recur faster than the hold expires. A burst is short: the clock essays measured how long a burst of a given size keeps the detector alarmed, and the clock a burst cannot outlast set a ceiling on it. An alarm that has lasted longer than that ceiling, or alarms that keep recurring for longer than it, are not a burst, and an estimate that knew this could resume reading.
The measurement that follows caps the hold: the estimate is held while the detector alarms and for a slow half-life after, but never for longer in total than the burst ceiling, after which it reads every reading until the detector has been quiet for a full slow half-life again. It runs the same bursts and the same rises. The prediction is that on every rise the alarms fall back to within two points of the read-always detector’s 7% to 11%, because the hold expires at the ceiling, and that two minutes after a burst of 300 the second burst is caught at least 50% of the time against 3% read always, because a single burst ends inside the ceiling and its readings are skipped. It could fail on the large burst itself: a burst of 300 keeps the detector alarmed, or alarming on and off, for longer than smaller bursts, and if it outlasts the ceiling the estimate reads its tail as slope after all. The question is whether the length of an alarm can tell a burst from a rise, or whether a burst large enough to matter is long enough to pass for one.
Named alongside this one
Essays reaching for the same objects. Nobody chose these; they are what the concept index makes visible.
- A decay measured from where it started design parameter · estimator · half life · streaming model
- A choice the fences cannot make design parameter · prediction
- A division chosen before the queries arrive design parameter · prediction
- A floor under every level design parameter · prediction
- A guess that costs more than the walk design parameter · prediction
- A level priced on its own sample design parameter · prediction
The objects this essay names
Each one links to every other essay that touches it.
Design parameterEstimatorFalse alarmHalf lifePredictionStreaming model