The slope the slow counter already shows

A detector that learns its own quiet built a burst detector from two decayed counters on a stream of arrivals. A fast counter $f$ has a half-life $H$ of 13.3 seconds and a slow one $s$ has $R$ times that. The detector alarms when their difference $d = f - s$ exceeds $z$ times a scale it learns from $d$ itself. On a background that does not move, it sets its own false-alarm rate. On one that rises, it does not. A steady rise of $\beta$ arrivals a second, every second, leaves both counters lagging the true rate, the slow one by more, and $d$ settles at an offset of $\beta(R-1)H/\ln 2$ above zero. The detector reads that offset as a burst that never ends.

The clock a burst cannot outlast found the best of several schedules for the scale. It is held while the detector alarms, and learns again once $d$ has stayed positive longer than the ceiling $L^*$ that no burst’s excursion can outlast. That schedule cleared most of the false alarms a rise causes. It also found the one quantity that says where the rest come from, the rise’s offset measured in spreads of $d$. Its closing section proposed going at the offset directly. The detector does not know $\beta$, but the slow counter’s own rate of change is a reading of it, delayed and noisy. Subtract the predicted offset from $d$ before comparing, and a steady rise should leave the corrected difference centred on zero, with no clock and no reset. The prediction was that this flattens the offset curve’s peak near one spread, which neither clock touched. The cost was predicted too: a burst steepens the slow counter, so part of every burst would be read as slope. The question was whether one estimate could be slow enough to ignore a burst’s minute and quick enough to follow a drift’s hour.

Reading the slope off the slow counter #

The correction uses nothing the detector did not already hold. Once a second the slow counter’s value is kept, and its change over the last slow half-life, divided by that half-life, is a reading of how fast the background is rising: a counter that tracks a rate rising at $\beta$ rises at $\beta$ itself, once it has settled. The reason is the one the earlier page used to derive the offset. A decayed counter with rate constant $k$, on a rate rising at $\beta$, settles at a fixed lag of $\beta/k$ behind the rate. A fixed lag behind a line that rises at $\beta$ is itself a line that rises at $\beta$. So the slope of either counter, once settled, is the background’s slope, and the slow counter is the less noisy of the two to read it from. That reading is noisy all the same, since the slow counter carries the noise of every arrival it has seen, so it is decayed into an estimate $\hat\beta$ with a half-life of its own, $K$ slow half-lives. The predicted offset $\hat\beta(R-1)H/\ln 2$ is subtracted from $d$, and the corrected difference is compared with $z$ times a scale learned from the corrected difference while the detector is not alarmed. Everything else — the streams, their ten arrivals a second, the bursts, the minute of warm-up and the five slow half-lives before alarms are counted — is the earlier pages’. On a still stream the corrected detector alarms on 5.1% of readings at $z = 1.85$, the same $z$ and the same rate as the held design.

On a stream rising 20% every 1,000 seconds at R = 16, the estimated offset averages 0.60 arrivals a second against the 0.58 the rise predicts, and the corrected difference sits at −0.03. The uncorrected difference is where the earlier page’s formula said it would be, 0.57, and the estimate follows it from the counter alone. The corrected difference wanders around zero with the noise of the stream, the threshold sits above it, and 3.1% of the readings on this stretch alarm. At this rise the held scale alarms on 32% of readings across twenty-four streams.

The offset curve, flattened #

At R = 16 the corrected detector alarms on 5.5% to 7.8% of readings across rises whose offsets run from 0.31 to 2.48 spreads of $d$. The held scale alarms on 11.2% to 63.3% of them, and the positive clock on 9.2% to 11.4% below two spreads. The peak the positive clock left near one spread, 11.4% at 0.62 spreads and 9.5% at 1.24, is gone: the corrected detector sits at 5.6% and 6.5% there, a point or so above a still stream. That is the prediction, and it holds as stated.

At R = 64 the result is split. Near one spread the correction is better than either clock: 6.6% at 1.21 spreads against the positive clock’s 11.3%. On the steepest rises it is worse, 11.5% at 4.85 spreads and 16.7% at 9.70, where the positive clock has fallen to 0.3% and none. Those rises are so steep that $d$ stays positive for long stretches whatever the correction does, the positive clock’s schedule learns them, and a learned scale is wide enough to hold them. The correction leaves a residue proportional to its own error, and at R = 64 the error is multiplied by a gain four times larger. The gain $(R-1)H/\ln 2$ is 288 seconds at R = 16 and 1,210 at R = 64, and a small error in the slope becomes a large one in the offset. The two repairs answer different parts of the curve. The correction flattens it where offsets are moderate, and the clock handles the steep end, where waiting costs less than being right.

What a burst loses to the correction #

The correction costs a burst at most 3.7 seconds of detection: at R = 16 a burst of 900 arrivals stays detected for 51.0 seconds against the held scale’s 54.7, and bursts of 3,000 or more stay detected to the same 55.4 seconds. The prediction was right that a burst steepens the slow counter and that part of it is read as slope. It is a small part, and it lands at the end of the burst’s detection rather than the start. The arithmetic, from the detector’s constants rather than a separate run: a one-second burst of 900 arrivals raises the slow counter by about three arrivals a second, which over one slow half-life of readings is a slope of about 0.014 a second each second. Decayed with a half-life of one slow half-life, the estimate has taken in about a sixth of that fifty seconds later, which the gain of 288 seconds turns into about 0.6 arrivals a second of false offset. Early in the burst, when $d$ stands tens of arrivals a second above zero, that is nothing. Fifty seconds in, the burst’s own part of $d$ has decayed to about one arrival a second against a threshold near 0.9, and 0.6 subtracted from it ends the detection a few seconds early. The largest bursts are detected up to the ceiling $L^*$ either way, since past $L^*$ the slow counter holds more of any burst than the fast one and $d$ goes negative on its own.

At R = 64 the cost is 2.0 seconds at most, 71.0 against 73.0 at 900 arrivals, and nothing at 3,000 and above. The prediction framed the question as a trade between ignoring a burst’s minute and following a drift’s hour. At one slow half-life the estimate does both, because a burst reaches the estimate only slowly and only in its last seconds of detection, while a drift is the whole of what the estimate is reading.

How quick the estimate has to be #

Decayed with a half-life of one slow half-life, the estimate recovers 96% of the offset over the settled stream; four times slower, 71%; sixteen times, 29% — and the alarms follow. At four slow half-lives the corrected detector alarms on up to 22.1% of readings, at sixteen on up to 49.1%, nearly as many as the held scale. A slower estimate is less noisy once it has settled. On these streams it has not settled. The prediction worried that a quick estimate would read bursts as slope, and the measurement found that a slow one fails to read slope at all.

By the end of the warm-up, at 1,124 seconds, the quick estimate has reached 89% of the true offset; four times slower, 43%; sixteen times, 13%. An estimate decayed with a half-life of sixteen slow half-lives needs about an hour of R = 16 stream to reach half of a constant slope, longer than these streams last. The counter with no window in it described a decayed counter as a window without edges, weighting every past arrival by its age. The slope estimate is such a counter over slope readings, and its effective window is its half-life: an estimate whose window is longer than the rise has been going on is averaging the rise with the flat stretch before it. The summary that has to forget put the same trade in terms of a decayed summary’s memory: it remembers what the half-life lets it remember and not what came before. Here the thing to be remembered is the start of the rise, and a long memory starts from zero and has to fill.

That qualifies the answer to the section’s question. On streams whose rise began within the last hour, the quick estimate is the only one that works, and it costs bursts almost nothing. On a rise that had gone on for many hours, a slower estimate would have settled, and its lower noise might pay at R = 64, where the quick estimate’s noise is multiplied by the large gain. That was not measured, since every stream here starts flat and rises from its first second.

Why the gain matters #

The corrected detector’s weakness at R = 64 is arithmetic, and the same arithmetic says where the correction belongs. The offset is the slope times $(R-1)H/\ln 2$, so an error in the slope becomes an error in the offset multiplied by that gain. The slope is read from the slow counter, whose noise grows as $R$ grows: it averages over more arrivals, but a slope is a difference of two noisy values, and the gain multiplies what is left. The error of a difference found a difference of two estimates carrying the errors of both. The correction is a difference of a difference: the counters’ difference, less a multiple of one counter’s difference over time.

The residue can be measured directly. On eight streams rising 20% every 1,000 seconds, the estimated offset at R = 16 averages 0.554 arrivals a second against the 0.576 predicted, and varies from reading to reading with a standard deviation of 0.081, a sixth of the spread of $d$. At R = 64 it averages 2.321 against 2.418 and varies by 0.122, a quarter of the spread. Both estimates fall about 4% short on average, since each starts from zero when the stream starts and approaches the slope along its own decay. A shortfall of 4% of the offset is a tenth of a spread at R = 16’s steepest rise and four tenths at R = 64’s. That is why the corrected detector’s alarms at R = 64 rise with the slope, from 6.6% to 16.7%. The residue is a fixed share of an offset that grows, and the plate places it at the right end of the curve.

A result the size of its own noise asked when a measured difference is larger than the measurement’s own error. The same question applies inside the detector: the correction is worth making while its error is smaller than what it removes. At R = 16 it removes an offset of up to 2.5 spreads and leaves an error of about a sixth of one. At R = 64 on the steepest rise it removes 9.7 spreads and leaves nearly half of one, which is enough to double the still-stream alarm rate. It is still far better than removing nothing, which is what the held scale does, but it is no longer better than learning the rise, which is what the clock does.

So the correction suits small $R$, where the gain is modest and the offsets of ordinary rises sit near one spread. The positive clock suits large $R$ and steep rises, where offsets are many spreads and a scale that learns them is cheaper than an estimate precise enough to cancel them. The earlier page named the offset in spreads as the quantity that decides where a design fails. It also decides which repair to use: at 1.2 spreads or less the correction alarms less at both values of $R$, and at 2.4 spreads or more the clock does. The threshold somebody chose warned against constants picked without a reason, and this one has a reason, since the offset in spreads can be computed from $R$, $H$ and a guess at the steepest rise a system expects.

What the measurement leaves out #

Linear rises only. Every rising stream here is a straight ramp from its first second. A background that rises and then levels off, or rises in steps, would leave the estimate reading a slope that has ended, and the correction would then push the corrected difference below zero until the estimate decayed. How far below, and whether that hides bursts arriving at the change, was not measured.

Streams under an hour. The span is set by the earlier pages’ streams, about 53 minutes at R = 16 and longer at R = 64, with alarms counted from the end of a warm-up of five slow half-lives. The slow estimates’ failure is partly a statement about that span.

The held schedule for the scale. The corrected detector’s scale is learned while it is not alarmed, as in the held design. A corrected detector with the positive clock’s schedule, learning again once the corrected difference has stayed positive past $L^*$, would combine the two repairs, and the plates suggest it would take the better of each end of the curve. It was not built.

Twenty-four streams a point for alarms, sixty for bursts. As on the earlier pages. The horizons at R = 16 differ from the held design’s by 1.6 and 3.7 seconds, a few per cent of the lag, and the lags are interpolated between the eleven sampled at every stream.

Still open: a correction that learns where the rise stops #

The correction’s untested weakness is a rise that ends. The estimate keeps reading the slope for about one slow half-life after the background levels off, and for that long it subtracts an offset that is no longer there. The corrected difference is pushed below zero, and a burst arriving in that stretch starts from a deficit. A decay measured from where it started found a decayed counter’s reading depending on where the stream began; here the question is where a rise ended.

The measurement that follows runs streams that rise for a stated time and then hold level, and places bursts at stated lags after the rise stops. It measures the corrected detector’s alarms on the level stretch and the bursts it misses there, against the held scale and the positive clock. The prediction is that for about one slow half-life after the rise ends the corrected detector misses bursts of 300 arrivals that the others catch, and that bursts of 900 and more are caught regardless, since their excursion dwarfs the leftover offset. The number that decides whether the correction is safe to deploy is the smallest burst it still catches in that stretch, set against the smallest burst anyone wanted to catch.