The clock a burst cannot outlast

A detector that learns its own quiet built a burst detector from three numbers. A fast decayed counter $f$ has the half-life $H = 13.3$ seconds that what a heavier tail actually buys matched to a 5% spread at ten arrivals a second. A slow counter $s$ runs at $R$ times that half-life, and a scale $v$ is a running mean square of their difference $d = f - s$. The detector is read once a second and alarms when $d$ exceeds $z\sqrt{v}$. On a stream that does not move it sets its own false-alarm rate without an oracle. Its trouble is what the scale should do with an alarm.

A scale that learns every reading takes the burst into itself and forgets it within 20 seconds at $R = 16$, whatever the burst’s size. A scale held while alarmed keeps the burst, up to a ceiling that does not depend on the scale at all:

$$L^* = H \log_2 R \cdot \frac{R}{R-1},$$

the lag past which the slow counter holds more of any burst than the fast one, so that $d$ has turned negative. That is 56.8 seconds at $R = 16$ and 81.1 at $R = 64$. But a held scale meets a slowly rising background as a burst that never ends, and alarms on it for good.

The closing section of that page proposed a schedule between the two, and gave its reason in one line. No burst can hold the detector’s attention past $L^*$, so a scale that learns again once an excursion has lasted longer than $L^*$ forgets nothing a burst could explain and learns everything a burst could not. It named two clocks the schedule could run on — how long the alarm has lasted, and how long $d$ has stayed positive — and predicted that the first would fail on gentle rises. It was right about the first clock. The second does most of what the schedule promised, and the part it cannot do has a size that can be written down.

Two clocks on the same excursion #

Both schedules hold the scale while the detector alarms, exactly as the held design does, and differ only in when they let it learn again. The alarm clock counts how long the current alarm has lasted and releases the scale when that passes $L^*$. The positive clock counts how long $d$ has been above zero without a break and releases the scale when that passes $L^*$, whether or not every reading in between alarmed. Each keeps one more number than the held design: a count of seconds. Like the counters themselves, it needs no work between readings, which the fading nobody computes showed is what makes decayed state cheap to keep. Held designs read a scale biased low, since the largest ordinary excursions are kept out of it, so their threshold is raised once from $z = 1.645$ to $1.85$. That constant is fixed from a still stream for every $R$ at once — the kind of measured constant the threshold somebody chose went looking for in library code — and brings the false-alarm rate there back to 5.0–5.3% for all three held designs.

The trace shows the whole difference between them. On a background rising 5% every 1,000 seconds at $R = 64$, the slow counter lags the rise by more than the fast one does. The counter with no window in it found that a counter fading by half every $H$ averages over about $1.44H$ of the past, so a counter with a longer half-life is reading an older, lower rate. The difference $d$ then sits above zero by a steady amount, 0.6 arrivals a second, and the ordinary noise of $d$ rides on top of it. The noise is about as large as the offset, so readings wander above and below the threshold every few seconds. A quarter of them alarm, but no single alarm lasts longer than 19 seconds, so an alarm clock is reset long before it could reach 81. The scale stays held, and the detector keeps alarming on a quarter of its readings for as long as the rise goes on.

The positive clock is reset only when $d$ falls to zero, and with the noise riding on an offset of more than a spread, that is rare. In these ten minutes $d$ stays positive for up to 150 seconds at a stretch. That is almost twice the ceiling, and nothing a burst can do, because a burst’s $d$ has turned negative by $L^*$ whatever its size. The alarm clock asks how long the detector has been sure. The positive clock asks how long the stream has been on one side, which is the question $L^*$ was derived to answer.

What each clock does to a rising background #

At $R = 16$ the alarm clock is nearly the held scale on gentle rises — 11.1% against 11.2% at 5%, 17.4% against 18.3% at 10% — and helps only once a rise is steep enough to hold a single alarm past $L^*$. At 40% it does, and the rate falls to 13.7%. This is the earlier page’s 60-second hold limit measured again. That page found the limit helped little at gentle rises, and $L^*$ at $R = 16$ is 57 seconds, close enough that nothing changes. The prediction that the alarm clock would fail on the gentle rises was correct, and the trace above is the mechanism.

The positive clock alarms on 9.2%, 11.4%, 9.5% and 3.1% of readings as the rise steepens, and its rate falls as the drift grows rather than climbing. The held scale goes the other way, from 11.2% to 63.3%. A steep rise holds $d$ positive for long stretches, so the positive clock releases the scale quickly and it learns the offset as part of the spread. Once it has, the threshold stands above the offset and the alarms stop. A gentle rise does not hold $d$ positive for long, and the scale stays held against an offset it never learns. So the positive clock does worst on the gentlest drifts, and its rate peaks between them, near 11% at a rise of 10%.

That is still twice the 5% the threshold was set for, and it is no accident that the scale learning every reading sits at about the same place, 6.6% to 10%. Once released, the positive clock’s scale is a learning scale, and it learns a drift the same way: as spread. It cannot do better on a drift than a scale that learns everything, because releasing the scale is all it does. What it adds is the burst. The learning scale forgot a burst of 900 in 20 seconds. The positive clock, as the next section measures, keeps it for 53.

At $R = 64$ the same designs separate much further, because the offset a rise puts into $d$ grows with the slow half-life. The held scale alarms on 34.5% of readings at a 5% rise and 63.7% at 10%. The alarm clock alarms on 29.8% and 21.3%. The positive clock alarms on 11.3%, then 3.1%, 0.3% and none as the rise steepens. From a rise of 10% every 1,000 seconds upwards, the positive clock at $R = 64$ is below the target it was calibrated to on a still stream. Past about two spreads of offset the drift barely alarms at all.

The price of the schedule, paid by the burst #

The case for any schedule rests on the other half: that it does not give back the horizon the held scale bought. The positive clock is more exposed to that than the alarm clock. A burst that lands while $d$ happens to be above zero already inherits the seconds already on the clock, and its alarm can be released a little early.

At $R = 16$ the positive clock keeps a burst of 900 visible for 53.3 seconds against the held scale’s 54.7, and a burst of 9,000 for 54.2 against 55.4. The loss is about a second at every size, and it is the inherited seconds: a burst that arrives on top of an ordinary positive stretch starts its excursion with the clock already running. At $R = 64$ the numbers are 71.9 against 73.0 at 900 and 77.3 against 78.4 at 9,000. That is the same second, against a ceiling of 81.1. Both designs sit a few seconds under $L^*$ at the largest bursts. The ceiling is where $d$ crosses zero on average. Noise brings the crossing forward on some streams and back on others, and the half-detection lag reports where half of them have crossed.

The alarm clock keeps exactly the held scale’s horizon at every size and both ratios: 54.7 seconds at 900 for $R = 16$ and 73.0 at 64. No burst’s alarm reaches $L^*$, so the alarm clock never releases on a burst. That is the property it was designed around. The trouble is that it releases on almost nothing else either.

So the positive clock trades about one second of horizon for most of the drift problem. At $R = 64$ and a 10% rise, where the held scale alarms on nearly two readings in three and the alarm clock on one in five, it alarms on one in thirty. The summary that has to forget found that a structure asked only about the recent past has to forget on a schedule. The positive clock is such a schedule, and it is the cheapest one the detector could hold: one count of seconds, reset to zero whenever $d$ turns.

One quantity that places every drift #

The two ratios behave so differently that they look like different detectors. They are not: they are the same detector meeting drifts of different sizes, and the size that matters can be computed.

An exponential counter with rate constant $k$, watching a rate that rises at $\beta$ arrivals a second per second, reads $\beta/k$ behind it. The slow counter lags by more than the fast one, so the drift puts a steady offset into their difference:

$$\bar d = \beta\left(\frac{1}{k_s} - \frac{1}{k_f}\right) = \frac{\beta\,(R-1)\,H}{\ln 2}.$$

At ten arrivals a second, a rise of 10% every 1,000 seconds is $\beta = 0.001$. The formula gives offsets of 0.29 arrivals a second at $R = 16$ and 1.21 at $R = 64$. On eight rising streams the measured mean of $d$ is 0.285 and 1.200. Over all eight settings the formula and the measurement agree to within about 4%, except the gentlest rise at $R = 16$. There the offset is small beside the noise and eight streams measure it 10% high. The spread the offset is judged against was computed in the earlier page from the same Poisson arithmetic: 0.46 arrivals a second at $R = 16$ and 0.50 at $R = 64$. So the offset in units of spread is the one number that places a drift for this detector.

Placed on that axis, the two ratios fall onto one curve. A rise of 40% at $R = 16$ and 10% at $R = 64$ put nearly the same offset into $d$, 2.48 and 2.43 spreads, and the positive clock alarms on 3.1% of readings at both. A rise of 20% at $R = 16$ and 5% at $R = 64$ give 1.24 and 1.21 spreads, and 9.5% and 11.3%. The held scale lines up the same way, at 63.3% and 63.7% for the first pair and 32.4% and 34.5% for the second.

The curve says what the positive clock can and cannot do. Its alarms rise as the offset grows towards about one spread, and fall away steeply above two. Below one spread, the noise pushes $d$ back through zero often enough that the clock seldom reaches $L^*$, and the scale stays held against an offset it never learns. Above two, $d$ almost never touches zero during a drift, the clock passes $L^*$ on the first long excursion, and the offset becomes part of the spread. The worst drift for this detector is one that moves its difference by about one spread, and at either ratio that drift costs roughly 11% of readings in alarm.

The axis also says why $R = 64$ looked so much better. A larger $R$ multiplies the offset by $R - 1$ but barely changes the spread (0.46 to 0.50), so every drift is moved rightwards along the same curve, past its peak. The detector at $R = 64$ is not cleverer. It meets every drift as a bigger one. The price is the one the earlier page found: a slow counter at 64 half-lives follows nothing that happens within a quarter of an hour.

Learning a drift as a level instead #

Releasing the scale answers a drift by treating it as spread: the threshold rises until the offset sits beneath it. There is a second way to answer it, and it is closer to what a drift actually is. An excursion that outlasts $L^*$ is not a burst, so it is a change in the background, and the background is what the slow counter is for. The rebasing design uses the same positive clock, but when it passes $L^*$ while alarmed it sets the slow counter equal to the fast one and leaves the scale alone. It needs its own $z$, lowered once to 1.65, since clearing long positive runs lowers its alarm rate on a still stream. A reset is the move a decay measured from where it started made with its landmark: forget everything before a chosen moment and measure from there. Here the moment is chosen by the stream rather than by a timer.

Rebasing is the best of the designs on the gentlest drifts and the worst on the steepest. At $R = 16$ it alarms on 6.6% and 6.5% of readings at rises of 5% and 10%, against the positive clock’s 9.2% and 11.4%. It rises to 9.0% at 20% and 14.4% at 40%, where the positive clock has fallen to 3.1%. The trace shows why. A reset clears the offset, but the rise has not stopped. The slow counter starts lagging again at once, $d$ climbs back towards the same offset, and the detector alarms and waits out $L^*$ before resetting again. A drift learned as level becomes a staircase. Every step costs one alarm lasting at least $L^*$, and the steeper the rise, the more steps there are in a thousand seconds: 1.2 at a rise of 10% and 2.7 at 40%. At $R = 64$ the staircase costs more, 12.5% and 17.9% at the two steepest rises. The offset there is four times larger, and every step waits out a ceiling of 81 seconds rather than 57.

Rebasing also gives back more of the burst than the positive clock. It keeps a burst of 900 for 50.7 seconds at $R = 16$ and 60.6 at $R = 64$, against the positive clock’s 53.3 and 71.9. The reason is the same inherited seconds, with a larger penalty. A burst on top of a long positive stretch can trip the reset while the burst is still in the fast counter, and a reset sets the slow counter to a fast one that contains the burst.

The two ways of learning a drift fail in opposite places. Learned as spread, a drift is never alarmed on again once the clock has released, but the scale is inflated for as long as the offset lasts. Learned as level, the scale stays honest, but the drift is alarmed on again at every step. The error of a difference found a subtraction whose error stayed a fixed share of a quantity that stopped being the one asked about. The staircase is the detector’s version of that: the offset is cleared each time, but it is not the quantity that is still growing.

Where these numbers stop applying #

Rises that are linear. Every drift here rises at a constant rate. A window that is a duration found that on a drifting stream even the meaning of recent shifts, and a linear rise is the gentlest such stream there is. A rise that accelerates would move along the offset curve during the stream, and a detector could be released at a gentle offset and then face a steep one with a scale that learned the gentle one. A rise that stops is a step, which is closer to the rebasing design’s own model of the world than to the spread-learning one.

Poisson arrivals, and one burst shape. Bursts are a stated number of arrivals spread over one second, on backgrounds of ten Poisson arrivals a second, as on the earlier pages. A clumped background has a larger spread of $d$ and longer ordinary positive runs. The inherited seconds that cost the positive clock about one second of horizon here would cost it more, and the drift curve would move left, since the same offset would be fewer spreads.

$z$ fixed from a still stream. Each design’s threshold constant is set once, for both ratios, from twenty-four still streams, and gives 5.0–5.3% there for the held designs and 4.0–5.2% for rebasing. No design was tuned against the rising streams it is scored on.

Horizons from sixty paired streams, alarm rates from twenty-four. Each burst run shares its background with a no-burst run, so a horizon measures the burst and nothing else. A result the size of its own noise found a difference smaller than its own spread across repeated runs; the differences reported here were checked against that failure by their sign. The one-second loss the positive clock pays is a difference between two numbers each measured on the same sixty streams, and it has the same sign at every size and both ratios.

The ceiling is exact only for the mean. $L^*$ is where a burst’s contribution to $d$ crosses zero, and it holds on average. On one stream, noise moves the crossing by a few seconds either way, which is why no held design quite reaches it.

Still open: a difference corrected for the slope it can see #

Every design here meets a drift through the offset it puts into $d$, and that offset has a formula: $\beta(R-1)H/\ln 2$. The detector does not know $\beta$, but it can estimate it. The slow counter’s own rate of change is a reading of $\beta$, delayed and noisy. A third decayed quantity — the slow counter’s difference from its value one slow half-life ago, itself decayed — would give an estimate from the detector’s own state, with no clock and no reset.

The measurement that follows subtracts the predicted offset from $d$ before comparing it with the threshold, so that a steady rise leaves the corrected difference centred on zero. It asks where on the offset curve the corrected detector alarms, and what the correction costs a burst. The prediction is that it flattens the curve’s peak near one spread, which neither clock touches, because it needs no long positive run to act. The cost should be the other side of the same coin. A burst steepens the slow counter too, so part of every burst is read as slope and subtracted. The question is whether the slope estimate can be made slow enough to ignore a burst’s minute and still fast enough to follow a drift’s hour, and the ratio between those two timescales is the one this whole family of detectors keeps returning to.