Efficient Regression Models for Scan Statistics

CosmicAI Researchers Gazi Abdur Rakib, Tristan Ashton, Dr. Ryan Loomis, Dr. Brian Mason, Dr. Eric Murphy, Dr. Ci Xue, and Prof. Jeff Phillips explored a new class of scan statistics for detecting interval anomalies in one-dimensional signals and applied it to the identification of platforming anomalies in ALMA bandpass calibration solutions. The team generalized the classical Gaussian scan statistic to fit regression models to the background signal rather than a constant mean, and developed linear-time algorithms that make the approach practical at the survey scale.   

What the team did

Bandpass calibration corrects a radio telescope's instrumental response across the observing frequency range. At ALMA, the correlator processes each spectral window in segments called subbands. Platforming occurs when those segments fall out of alignment, leaving a discrete step in a stretch of channels that should form a smooth curve. Unflagged, it propagates into the reconstructed data cube as an artifact. Catching it is harder than it sounds, since the expected bandpass is itself curved, and real spectra also carry atmospheric absorption features and edge effects. Until recently, this was done by eye, with data reducers missing roughly 49% of cases. The automated detector ALMA, deployed in October 2025, raises fewer false alarms but misses about 58% of events.

What researchers found

Scan statistics test every possible interval in a signal, asking at each one whether the data are better explained by separate models inside and outside that interval than by a single model fit to the entire signal. The largest improvement is in the detection. The classical version, developed for problems like disease cluster detection, assumes a constant background. A bandpass is not constant, so that assumption makes ordinary variation look anomalous and lets real defects hide inside it.

The team replaced the constant background with two richer families: a low-degree polynomial capturing gentle global drift, and Nadaraya-Watson kernel regression, a smoother that estimates each point as a weighted average of its neighbors within a chosen bandwidth. The kernel version adapts to local structure rather than imposing a single global shape and proved clearly the better of the two.

The barrier was cost. With millions of candidate intervals per spectrum, evaluating each from scratch takes over ten minutes. But adjacent intervals overlap almost completely, so the score can be updated incrementally, and truncating the kernel confines each update to a small neighborhood. Runtime drops to about 70 milliseconds, a thousandfold speedup.

Why the work matters

Half of all platforming defects currently reach downstream science products undetected. As next-generation facilities come online with data rates well beyond ALMA's, manual inspection of each calibration solution becomes infeasible, and the accuracy gap must close without a corresponding increase in false alarms that would discard usable observations.

This work also generalizes beyond the original application. Any measurement that drifts smoothly and is occasionally interrupted by a localized offset fits the same model. The team demonstrated this on solar radiation time series from ground-based weather stations, where passing shadows produce the same signature.

Code Repository / Scan Statistics Library