A detector that learns its own quiet

The counter with no window in it kept a rate as one number that fades by half every half-life, and found it settles on a steady stream at exactly the count of a window 1.44 half-lives long. What a heavier tail actually buys asked how long such a counter keeps a single burst visible. It matched each decay scheme to the same 5% spread on a steady stream of ten arrivals a second — for the exponential, a half-life of 13.3 seconds — added a burst, and measured the lag at which the estimate stopped standing out. “Standing out” needed a threshold. The page took it from an oracle: sixty independent runs of the same background with no burst, and the 95th percentile of the estimate across them at each lag.

Its closing section named that oracle as the thing a deployed detector does not have. A real system watches one stream, and the spread it must compare against belongs to an arrival process that may itself be drifting. The section proposed a structure that supplies its own threshold. It keeps the fast counter, adds a second at a much longer half-life to estimate the background, and uses the slow counter’s own variation as the spread — “two values and a stamp”. It asked two things: how much horizon is lost against the oracle, and what happens when the background drifts slowly, since a burst and a drift differ only in their timescale.

The detector can be built and it calibrates itself. The horizon it loses is most of the horizon, for a reason that the obvious repair turns into a different failure. The question the section asked last — whether one structure can tell a burst from a drift — has a plain answer, and it is no.

The detector, and the stream it calibrates on #

The detector keeps three numbers. The fast rate $f$ is the earlier page’s matched exponential counter, half-life $H = 13.3$ seconds. The slow rate $s$ is the same counter with half-life $RH$. The scale $v$ is an exponentially weighted mean square of the difference $d = f - s$, with the slow half-life. It is read once a second and alarms when $d > z\sqrt{v}$, with $v$ as it stood before the current reading, so the reading being judged is not yet part of the scale it is judged against. $z = 1.645$ is the one-sided 5% point of a normal distribution, and the false-alarm rate that results is measured, not assumed. The detector calibrates for its first minute, starting both counters at the rate of that minute, which removes a start-up transient without an oracle. Streams are long enough for the slow counter to settle, twelve slow half-lives and more.

On a stream that does not move, the detector calibrates itself: it alarms on 4.8% to 5.6% of readings at every R, against the 5% its threshold was set for. No independent runs were used. The difference between two counters on one stream has a spread the stream itself reveals, and a running mean square of that difference is a sufficient estimate of it at every ratio of half-lives tried. The difference is not quite normal, since it is a difference of Poisson-driven sums, but it is close enough that the normal’s 5% point gives 5%.

The scale it learns can be checked against a formula. A decayed counter with rate constant $k = \ln 2 / H$ on a Poisson stream of rate $\lambda$ has variance $\lambda k/2$, and two counters on the same stream share arrivals, so their difference has variance $\lambda\,(k_1/2 + k_2/2 - 2k_1k_2/(k_1+k_2))$. At ten arrivals a second that predicts a spread for $d$ of 0.342, 0.464 and 0.499 arrivals a second at $R$ = 4, 16 and 64. The detector’s own scale, averaged over eight settled streams, is 0.336, 0.456 and 0.502. The running mean square finds the same number the formula does, without being told the rate or that the stream is Poisson. The spread grows with $R$ because a slower background counter shares less of the fast one’s noise, so less of it cancels in the difference. The error of a difference found the same thing for subtracted sketches: what a difference’s error depends on is how much the two estimates share.

That half of the proposal works as stated, and it is the half that would have been hardest to believe without measuring. A detector can set its own false-alarm rate from its own readings. The other half is the horizon.

The burst, and the scale that learns it #

The trace shows what goes wrong. At the burst the difference leaps from its usual fraction of an arrival a second to 42.6. The detector alarms at once. At the next reading, the scale has taken in the square of that leap. A mean square with a slow half-life of 213 seconds moves only a little at each reading, but the leap is nearly fifty times the threshold’s usual size, and its square over two thousand times the usual mean square. The threshold climbs more than tenfold, to a peak of 11.6. The difference is falling at the fast counter’s rate, halving every 13 seconds, and 24 seconds after the burst it passes under a threshold that is now measuring the burst itself. The scale has learned the burst as part of the stream’s ordinary variation. The held scale, which took none of the leap in, keeps its threshold at 0.9, and its alarm ends 51 seconds after the burst. By then the difference itself has fallen under that unchanged threshold, a few seconds before the lag at which, as the next section shows, it would have turned negative anyway. The two detectors see the same difference; they disagree only about what to compare it with.

A learning scale’s horizon does not grow with the burst: 20 seconds at R = 16 and 39 at R = 64, whether the burst is 300 arrivals or 9,000. A larger burst leaps higher and inflates the scale in proportion, so the ratio between them — which is all the alarm tests — is the same. The oracle’s threshold does not move with the burst, and its horizon grows as the earlier page found for an exponential: by a fixed number of seconds for every factor of $e$ in the burst, 55 seconds at 300 and 119 at 9,000. Against it, the learning detector gives up between a third of the horizon, at the smallest burst and R = 64, and five sixths of it, at the largest and R = 16.

This is not a defect of the proposal’s arithmetic. It is the proposal doing what it said. The scale is the spread of $d$ over the recent past, and a burst is part of the recent past. A detector that knows only its own stream has no way to say that one excursion was not variation, except by deciding so — which is what an alarm is.

Holding the scale, and its ceiling #

The obvious repair acts on that decision. While the detector is alarmed, the scale is not updated: a reading judged to be a burst is kept out of the scale the next reading is judged against. That biases the scale low, because the largest ordinary excursions are now excluded as well. The false-alarm rate on a still stream rises to 7.5–8.7%. Raising $z$ once, to 1.85, for every $R$ at once, brings it back to 5%. That is a constant of the design, fixed from a still stream in advance, not a threshold drawn per stream — the kind of constant the threshold somebody chose went looking for in library code, except that this one has a measurement behind it.

With the scale held, the horizon comes back. At a burst of 300 and $R = 64$ it is 52.7 seconds against the oracle’s 54.8, which is essentially the oracle. At larger bursts the held detector’s horizon grows at first and then stops: 73.0 seconds at 900, then 78.4 at 3,000 and at 9,000. At $R = 16$ it stops at 55.4.

The ceiling has a closed form, and it does not depend on the scale at all. A burst of $B$ arrivals adds $B \ln 2 / H \cdot 2^{-L/H}$ to the fast rate $L$ seconds later, and $B \ln 2 / (RH) \cdot 2^{-L/(RH)}$ to the slow one. The difference is positive only while the first exceeds the second, which holds until

$$L^* = H \log_2 R \cdot \frac{R}{R-1}.$$

At $H = 13.3$ seconds that is 56.8 seconds for $R = 16$ and 81.1 for $R = 64$, against 55.4 and 78.4 measured. Past $L^*$ the slow counter holds more of the burst than the fast one, and the difference the detector watches has turned negative whatever the burst’s size. A larger $R$ raises the ceiling, but only by the logarithm of $R$. The formula also says what reaching the oracle’s horizon would take. To keep a burst of 9,000 visible for the oracle’s 119 seconds, the ceiling must be at least that, which needs $\log_2 R$ near 9 — a slow half-life of about five hundred fast ones, almost two hours. A background estimate that slow cannot follow any change in the stream that happens within a working day, which is the drift problem below in its strongest form. A decay measured from where it started found that a counter’s response to a step is set by the shape of its weights. Here the response that matters is the difference of two counters, and its shape is fixed by their ratio of half-lives alone.

At a single burst size the three designs agree completely for the first 16 seconds, and part in order: the learning scale by 24 seconds, the held scale between 48 and 64, and the oracle between 64 and 96. The held detector’s loss against the oracle is the ceiling, and the learning detector’s loss is the scale. Detection falls from all to nothing within one step of the lag grid in both detectors, because each fails for a deterministic reason — the difference passing under the scale, or under zero — rather than by noise.

A drift is a burst that does not end #

The closing section’s second question was drift. A background whose rate rises slowly makes the fast counter run ahead of the slow one by a steady amount: the slow counter lags the rise by about its own half-life. To the detector, that looks like a small burst that never decays. A window that is a duration found that on a drifting stream even the meaning of recent shifts — a window of arrivals and a window of time disagree — and a detector comparing two timescales of the same stream meets the same shift as a signal.

The learning detector survives a drift; the held one does not. With the background rising 10% every 1,000 seconds, the learning detector alarms on 9.5% of readings, twice its target. The steady offset between the counters enters its scale and raises its threshold with it. At steeper rises the offset dominates the scale so completely that the alarm rate falls again, to 6.6%: a drift the scale has learned entirely is invisible. The held detector alarms on 18.3% at a 10% rise and 63.3% at 40%. The first time the offset pushes it over its threshold it stops learning, and the offset keeps it over, so it alarms for good. Limiting the hold to 60 seconds lets it relearn and brings the steepest case down to 14.8%. At the gentler rises, where the offset is comparable to the ordinary spread and the detector crosses in and out, it helps little.

So the two versions of the detector fail in opposite directions, and each fails where the other succeeds. The learning scale treats every sustained excursion as the new normal: it adapts to drift and forgets bursts. The held scale treats every excursion as an event until told otherwise: it remembers bursts and cannot adapt to drift. A limit on the hold is a timescale that decides which of the two an excursion is, and a burst and a drift differ in exactly that — the timescale. The proposal predicted the problem in its last sentence, and the measurement puts numbers on it. There is no setting of this structure that holds a burst of 900 for the oracle’s 80 seconds and alarms on a 10% rise per 1,000 seconds at no more than 5%.

What was measured and what was not #

Poisson backgrounds at ten arrivals a second. Every stream is Poisson with a steady or linearly rising rate. A bursty background — arrivals in clumps — has a larger spread of $d$ than Poisson, which the learning scale would learn correctly, and larger ordinary excursions, which the held scale would mistake for events. The formula that matched the learned scale so closely is a Poisson formula, and on a clumped stream it would no longer be the thing to check against.

One burst shape. Every burst is its arrivals spread over one second, as on the earlier page. A burst spread over a minute is closer to a drift and would be forgotten sooner by the learning scale and held longer by the held one.

Readings once a second. The detector is read once a second. It could be read at every arrival; the scale would then be a mean over arrivals rather than over time, and on a rising background it would weigh the rise more.

The oracle is the earlier page’s, and it is generous. Its threshold is the 95th percentile of the fast counter across sixty independent streams at exactly the lag being tested, which assumes the detector knows both the background’s distribution and how long ago the burst happened. The self-calibrating detectors know neither, and the gap between them and the oracle is partly the price of not knowing when to look.

Three numbers of state. The proposal said two values and a stamp; the detector here keeps three values — the two rates and the scale — and a stamp. The fading nobody computes showed that decayed values need no work between arrivals, and that holds for all three.

Still open: a scale that forgets bursts on a schedule #

The summary that has to forget found that a structure asked about only the recent past must forget on a schedule, and pays bits for the schedule that nothing else pays. The scale here is such a structure, and the question is what its schedule should be.

The two failures suggest a structure between them. The held scale fails on drift because it never learns what it has once alarmed on. The learning scale fails on bursts because it learns them immediately. A scale that learns what it has alarmed on, but only after the alarm has lasted longer than a burst could — longer than the ceiling $L^*$ — would forget nothing a burst could explain and learn everything a burst could not.

The measurement that follows sets the hold’s limit at $L^*$, 57 seconds at $R = 16$ and 81 at $R = 64$, and repeats both measurements. The prediction is that the burst horizon stays at the held scale’s ceiling, since no burst’s alarm outlasts it, and that on drift the alarm rate falls towards the learning scale’s, since the relearning starts as soon as the excursion has outlasted anything a burst could produce. It could fail on the gentle rises, where the offset is small enough that the detector crosses in and out of alarm and never holds long enough to relearn. If it does, the right quantity to wait for is not how long the alarm has lasted but how long the difference has stayed positive, which a burst’s difference, decaying and then turning negative past $L^*$, never does for long.