We study the fundamental tradeoffs between computational tractability and statistical accuracy for a general family of hypothesis testing problems with combinatorial structures. Based upon an oracle model of computation, which captures the interactions between algorithms and data, we establish a general lower bound tha…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Statistical detection problems”
New method for valid and exact statistical inference of multi-dimensional change-points.
problem Statistical inference of change-points in multi-dimensional sequences.
method Proposes a method to guarantee the statistical reliability of both location and components of detected changes.
result Demonstrates the effectiveness of the method in genomic abnormality identification and human behavior analysis.
The study defines backdoor detection in ML and proves its infeasibility.
problem Backdoor detection in machine learning systems.
method Formal statistical definition and analysis of feasibility.
result Backdoor detection is impossible except for very small alphabet sizes.
SI-CLAD improves clustering-based anomaly detection by controlling false positives.
problem Lack of reliability in clustering-based anomaly detection.
method SI-CLAD (Statistical Inference for CLustering-based Anomaly Detection) using Selective Inference framework.
result SI-CLAD rigorously controls false detection probability below a specified significance level.
Proposes a framework for OOD detection combining multiple statistics.
problem Detecting out-of-distribution (OOD) samples reliably during inference.
method Multiple hypothesis testing with conformal p-values.
result Uniformly outperforms threshold-based tests across different datasets and neural networks.
The study formalizes temporal precision and recall for anomaly detection in sequences.
problem Insufficient understanding of precision and recall in sequential anomaly detection.
method Formalized temporal precision and recall measures, developed time-tolerant confusion matrices, and demonstrated statistical significance.
result Precision and recall may overestimate performance with temporal tolerance.
Simple methods combine statistical tests for out-of-distribution detection.
problem Detecting data points not following the training distribution.
method Combining classical parametric tests (Rao's score test) and a typicality test.
result Combining Fisher's method of test statistics improves out-of-distribution detection accuracy.
Detecting the emergence of an abrupt change-point is a classic problem in statistics and machine learning. Kernel-based nonparametric statistics have been used for this task which enjoy fewer assumptions on the distributions than the parametric approach and can handle high-dimensional data. In this paper we focus on th…
Paper proposes a scoring function for detecting anomalies in large datasets.
problem Detecting outliers in large, feature-rich datasets.
method Binary classification problem with a two-sample linear rank statistic.
result Empirical results show the effectiveness of the proposed scoring function.
Testing for latent structure via the Wilcoxon--Wigner random matrix of normalized rank statisticsstat.ME
Paper develops new method for detecting latent structure in large symmetric data matrices.
problem Testing for latent structure in large symmetric data matrices.
method Introduces Wilcoxon--Wigner random matrices based on normalized rank statistics.
result Establishes asymptotic Gaussian fluctuations for leading eigenvalue and eigenvector of Wilcoxon--Wigner matrices.
Data-driven anomaly detection methods suffer from the drawback of detecting all instances that are statistically rare, irrespective of whether the detected instances have real-world significance or not. In this paper, we are interested in the problem of specifically detecting anomalous instances that are known to have …
Paper introduces a statistical framework for watermarking LLM-generated text.
problem Detecting LLM-generated text from human-written text with statistical signals.
method Hypothesis testing formulation, pivotal statistic selection, secret key, closed-form expression of false negative rate, minimax optimization.
result Derives optimal detection rules for LLM-generated text, demonstrating higher power than existing methods.
This paper considers the problem of detection in distributed networks in the presence of data falsification (Byzantine) attacks. Detection approaches considered in the paper are based on fully distributed consensus algorithms, where all of the nodes exchange information only with their neighbors in the absence of a fus…
Unified framework for online LLM watermark detection using e-processes.
problem Detecting AI-generated text from human-written content in online settings.
method Unified framework based on e-processes for anytime-valid hypothesis testing on independence.
result Proposed methods achieve competitive performance in watermark detection.
We develop a novel method for detection of signals and reconstruction of images in the presence of random noise. The method uses results from percolation theory. We specifically address the problem of detection of multiple objects of unknown shapes in the case of nonparametric noise. The noise density is unknown and ca…
Paper explores limits of high-order clustering with planted structures.
problem Statistical and computational limits of high-order clustering with planted structures.
method Developed methods for detection and recovery of clusters, identified signal-to-noise ratio boundaries.
result Sharp boundaries of signal-to-noise ratio for statistical and computational feasibility.
The accurate measurement of security metrics is a critical research problem because an improper or inaccurate measurement process can ruin the usefulness of the metrics, no matter how well they are defined. This is a highly challenging problem particularly when the ground truth is unknown or noisy. In contrast to the w…
On Rank Energy Statistics via Optimal Transport: Continuity, Convergence, and Change Point Detectionstat.ML
Paper uses optimal transport-based statistics for change point detection.
problem Change point detection in multivariate data.
method Soft rank energy and entropically regularized optimal transport.
result Soft rank energy performs better in real datasets with strong continuity and convergence properties.
Valid inference method for DTW distance for abnormal time-series detection.
problem Statistical inference on DTW distance under uncertain conditions.
method Conditional selective inference framework to derive valid p-values.
result First method to provide valid p-values for DTW distance.
The problems of outliers detection and robust regression in a high-dimensional setting are fundamental in statistics, and have numerous applications. Following a recent set of works providing methods for simultaneous robust regression and outliers detection, we consider in this paper a model of linear regression with i…
New findings on computational limits for estimating hidden structures.
problem Estimating hidden structures in noisy data.
method Use of low-degree polynomials as a restricted model of computation.
result Established low-degree hardness of recovery problems for easy detection problems.
The paper develops methods for novelty detection on path space using signature-based statistics.
problem Novelty detection on path space as a hypothesis testing problem.
method Signature-based test statistics, transportation-cost inequalities, CVaR, one-class SVM algorithms.
result Established lower bounds on type- error and general power bounds.
Paper introduces an unsupervised tensor-based anomaly detection method for spatiotemporal data.
problem Challenges in detecting anomalies in spatiotemporal data, especially in urban traffic monitoring and medical imaging.
method Formulates anomaly detection as a regularized robust low-rank + sparse tensor decomposition, incorporating spatiotemporal smoothness and local dependencies.
result Demonstrates improved anomaly detection performance on both synthetic and real data.
NN-CUSUM detects changes in high-dimensional data using neural networks.
problem Detecting abrupt changes in high-dimensional data.
method Neural network-based CUSUM for online change-point detection.
result NN-CUSUM performs well in detecting changes in high-dimensional data.
New method detects uncertainty in neural networks for out-of-distribution detection.
problem Detecting out-of-distribution inputs to ensure model reliability.
method Predictive topological uncertainty (pTU) based on persistent homology.
result pTU provides a statistical framework for OOD detection.
RealStats detects fake images rigorously, combining multiple detectors for robustness.
problem Detecting AI-generated images remains challenging due to evolving generative models.
method Combines training-free statistics to compute p-values and aggregate them for a unified real-image distribution.
result Framework produces interpretable probability scores for robust fake image detection.
A new method speeds up community detection in graphs.
problem Efficiently detecting communities in large graphs.
method Subsampled semidefinite programming for faster clustering.
result Statistical limits of sketching for community detection established.
Study phase transitions in identifying infected individuals using group testing.
problem Identifying a set of k infected individuals from a population using pooled tests.
method Two random assignment designs (constant-column and Bernoulli) and polynomial-time inference procedures.
result Sharp phase transitions in statistical and computational limits for detection and recovery problems.
Paper reproduces a kernel-based scan B-statistic for online change-point detection.
problem Continuous detection of distribution changes in online data streams.
method Efficient kernel-based scan B-statistic for online change-point detection.
result Scan B-statistic outperforms parametric methods in challenging scenarios.
Improves signal detection in non-Gaussian noise using transformed data.
problem Signal detection in rank-one signal-plus-noise data matrices.
method Pre-transforming matrix entries and using linear spectral statistics for hypothesis testing.
result Sharp phase transition of largest eigenvalues in spiked rectangular matrices.
A new method detects concept drift in streaming data using k-means space partitioning.
problem Detecting distribution changes in streaming data.
method Equal intensity k-means space partitioning (EI-kMeans) and heuristic sensitivity improvement.
result EI-kMeans improves drift detection accuracy and sensitivity.
A new method detects interactions in neural networks using topological analysis.
problem Detecting interactions between input features in neural networks.
method Topological analysis of neural network connectivity to quantify interaction strength.
result The PID algorithm outperforms state-of-the-art baselines in interaction detection tasks.
We consider the change-point detection problem of deciding, based on noisy measurements, whether an unknown signal over a given graph is constant or is instead piecewise constant over two connected induced subgraphs of relatively low cut size. We analyze the corresponding generalized likelihood ratio (GLR) statistics a…
Community detection is a fundamental statistical problem in network data analysis. Many algorithms have been proposed to tackle this problem. Most of these algorithms are not guaranteed to achieve the statistical optimality of the problem, while procedures that achieve information theoretic limits for general parameter…
New CUSUM method detects changes in Hawkes networks efficiently.
problem Detecting abrupt changes in Hawkes networks.
method Efficient online recursive implementation of CUSUM statistic.
result Proposed CUSUM method outperforms existing methods.
New method detects information leakage using approximate Bayes predictor.
problem Unintentional exposure of sensitive information via observable data.
method Statistical learning theory and information theory framework, approximating Bayes predictor's log-loss and accuracy.
result MI can be accurately estimated to detect ILs, outperforming state-of-the-art baselines.
The paper tackles data misappropriation in LLMs by embedding watermarks and testing for their presence.
problem Detecting data misappropriation in LLMs trained on copyrighted data.
method Embedding watermarks, formulating as hypothesis testing, developing statistical framework, constructing test statistics, determining optimal thresholds, controlling errors, establishing asymptotic optimality.
result The proposed statistical testing framework effectively detects data misappropriation in LLMs.
Cyclostationary Statistical Models and Algorithms for Anomaly Detection Using Multi-Modal Dataeess.SP
A framework is proposed to detect anomalies in multi-modal data. A deep neural network-based object detector is employed to extract counts of objects and sub-events from the data. A cyclostationary model is proposed to model regular patterns of behavior in the count sequences. The anomaly detection problem is formulate…
User authentication and intrusion detection differ from standard classification problems in that while we have data generated from legitimate users, impostor or intrusion data is scarce or non-existent. We review existing techniques for dealing with this problem and propose a novel alternative based on a principled sta…
Machine Learning (ML) models are applied in a variety of tasks such as network intrusion detection or Malware classification. Yet, these models are vulnerable to a class of malicious inputs known as adversarial examples. These are slightly perturbed inputs that are classified incorrectly by the ML model. The mitigation…
Detects changes in classifier scores to identify shifts in class priors.
problem Label shift changes in classification data.
method Sequential changepoint detection of classifier scores.
result Outperforms other detection procedures in label shift settings.
In this paper, we explore various statistical techniques for anomaly detection in conjunction with the popular Long Short-Term Memory (LSTM) deep learning model for transportation networks. We obtain the prediction errors from an LSTM model, and then apply three statistical models based on (i) the Gaussian distribution…
Detects dense subhypergraphs in random hypergraphs using low-degree polynomials.
problem Detecting a planted dense subhypergraph in a random hypergraph model.
method Degree-n^o(1) polynomials of adjacency tensor entries.
result Thresholds for detection in different density regimes.
AdaDetectGPT improves text authorship detection with statistical guarantees.
problem Determining if text is authored by a human or an LLM.
method Adaptive learning of a witness function from training data to enhance logits-based detectors.
result AdaDetectGPT nearly uniformly improves text authorship detection, with up to 37% improvement.
WISER detects watermarked segments in text via epidemic change-point analysis.
problem Identifying watermarked segments in mixed-source texts.
method Epidemic change-point perspective, WISER algorithm.
result WISER outperforms state-of-the-art methods in accuracy and speed.
Optimized Franz-Parisi criterion matches SQ lower bounds for various statistical models.
problem Understanding computational hardness in statistical inference.
method Proposed and refined Franz-Parisi criterion, established equivalence with SQ lower bounds.
result Optimized Franz-Parisi criterion is equivalent to Statistical Query (SQ) lower bounds.
SAAD enhances anomaly detection in automotive systems with high accuracy.
problem Anomaly detection in automotive systems.
method Integrates statistical techniques with machine learning.
result SAAD achieves 88.3% accuracy and 0.921 F1 score.
Feature Shift Detection: Localizing Which Features Have Shifted via Conditional Distribution Testscs.LG
Detects which features have shifted in data distributions.
problem Identifying which specific features have caused a distribution shift.
method Formalizes the problem as multiple conditional distribution hypothesis tests, proposes non-parametric and parametric statistical tests, and uses a test statistic based on the density model score function.
result Demonstrates methods for identifying when and where a shift occurs in multivariate time-series data.