Survey of robust clustering methods for hotspot detection.
problem Detecting false positives in spatial hotspot mapping.
method Statistically rigorous clustering techniques.
result Survey of models and algorithms for robust clustering.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “statistical clustering”
ElbowSig assesses clustering structure at multiple scales.
problem Selecting optimal number of clusters in unsupervised learning.
method Formalizes elbow heuristic with a normalized discrete curvature statistic.
result Validates multiscale clustering structure over various resolutions.
Study proposes a statistical testing framework for evaluating clustering pipelines.
problem Quantifying the statistical reliability of clustering results from data analysis pipelines.
method Selective inference-based statistical testing framework for clustering pipelines.
result The proposed test controls the type I error rate and is effective in validating clustering results.
A method for clustering small datasets in high dimensions using random projections.
problem Challenges in clustering small datasets in high-dimensional spaces.
method Random projection followed by binary clustering in one-dimensional space.
result Statistically significant clustering structures can be found with as few as 100-200 points.
The paper introduces a statistical distance matrix for better feature representation and clustering.
problem Lack of detailed distance representation between feature elements.
method Extended traditional statistical distance to a matrix form (statistical distance matrix) and applied hierarchical clustering.
result The statistical distance matrix with clustering (Information Mandala) provides clearer and geometrically arranged feature representations.
Neural clustering learns time series affinity from statistical features.
problem Challenging time series clustering with unknown cluster shapes and structures.
method Amortized neural inference using statistical features.
result Competitive clustering accuracy without manual specification of cluster shapes.
Paper explores limits of high-order clustering with planted structures.
problem Statistical and computational limits of high-order clustering with planted structures.
method Developed methods for detection and recovery of clusters, identified signal-to-noise ratio boundaries.
result Sharp boundaries of signal-to-noise ratio for statistical and computational feasibility.
New computational lower bounds for clustering and related problems.
problem Statistical-computational gaps in high-dimensional clustering problems.
method Investigation of low-degree polynomials in latent space models to derive lower bounds.
result New and sharper computational lower bounds for clustering, sparse clustering, and biclustering.
Replicable clustering algorithms for k-medians, k-means, and k-centers are proposed.
problem Designing clustering algorithms that produce the same partition on repeated runs under the same distribution.
method Utilizing approximation routines for combinatorial clustering problems in a black-box manner.
result Replicable algorithms for statistical -medians, -means, and -centers with specified approximation and sample complexities.
Paper proposes efficient methods for high-order clustering in tensor block models.
problem High-order clustering of multiway datasets in neuroimaging, genomics, etc.
method Tensor block model and computationally efficient algorithms (HLloyd, HSC)
result Achieves high-order exact clustering with statistical optimality and computational efficiency.
SI-CLAD improves clustering-based anomaly detection by controlling false positives.
problem Lack of reliability in clustering-based anomaly detection.
method SI-CLAD (Statistical Inference for CLustering-based Anomaly Detection) using Selective Inference framework.
result SI-CLAD rigorously controls false detection probability below a specified significance level.
Paper improves clustering risk bounds for kernel k-means.
problem Improving clustering risk bounds for kernel k-means.
method Analyzes kernel k-means and Nyström approximation.
result Achieves nearly optimal excess clustering risk bound.
Polynomial-time algorithm finds planted hypercube vectors in Gaussian mixtures.
problem Clustering d-dimensional Gaussian mixtures with unknown covariance.
method Lattice-based methods using Lenstra--Lenstra--Lovasz reduction.
result Achieves statistically-optimal sample complexity of d+1 samples.
The paper analyzes -means clustering for missing data, proving statistical guarantees under MCAR.
problem Statistical guarantees for -means clustering with missing data, especially under Missing Completely at Random (MCAR).
method Established -excess risk bound and consistency of cluster centers under general missing mechanisms; derived -convergence rate and asymptotic normality for MCAR.
result Achieving -rate and converging to true cluster centers requires distinct true cluster centers in every dimension under MCAR.
Paper improves robust spectral clustering for noisy data.
problem Noisy data and heavy-tailed entries hinder traditional clustering methods.
method Robust spectral clustering with rank statistics for latent structure recovery.
result Provable recovery of latent block structure in large data matrices.
A novel multi-resolution cluster detection (MCD) method is proposed to identify irregularly shaped clusters in space. Multi-scale test statistic on a single cell is derived based on likelihood ratio statistic for Bernoulli sequence, Poisson sequence and Normal sequence. A neighborhood variability measure is defined to …
A cluster tree provides a highly-interpretable summary of a density function by representing the hierarchy of its high-density clusters. It is estimated using the empirical tree, which is the cluster tree constructed from a density estimator. This paper addresses the basic question of quantifying our uncertainty by ass…
New algorithm improves clustering accuracy without sacrificing scalability.
problem Improving clustering accuracy for large datasets.
method Nonnegative low-rank semidefinite programming with Burer-Monteiro factorization.
result Significantly smaller mis-clustering errors compared to existing methods.
Quantum GBS boosts asset clustering for robust statistical arbitrage portfolios.
problem Identifying co-moving assets from correlation matrices for statistical arbitrage.
method Mapping S&P 500 correlation data to GBS-compatible adjacency matrices, benchmarking classical and quantum clustering algorithms.
result Quantum GBS generates superior alpha during high volatility periods, persisting under low-loss conditions.
Topology of Currencies: Persistent Homology for FX Co-movements: A Comparative Clustering Studystat.ML
TDA improves FX clustering quality over traditional methods.
problem Capturing complex currency co-movements in FX markets.
method Topological Data Analysis (TDA) compared to traditional statistical methods on monthly FX returns.
result TDA-based clustering yields more compact and well-separated clusters.
Counting the number of clusters, when these clusters overlap significantly is a challenging problem in machine learning. We argue that a purely mathematical quantum theory, formulated using the path integral technique, when applied to non-physics modeling leads to non-physics quantum theories that are statistical in na…
The problem of finding groups in data (cluster analysis) has been extensively studied by researchers from the fields of Statistics and Computer Science, among others. However, despite its popularity it is widely recognized that the investigation of some theoretical aspects of clustering has been relatively sparse. One …
Characterizes optimal reconstruction error in high-dimensional Gaussian mixtures.
problem Optimizing reconstruction error in high-dimensional sparse Gaussian mixtures.
method Exact asymptotic characterization using state evolution of AMP algorithm.
result Identification of statistical-to-computational gap between AMP and information-theoretic threshold.
Cluster algorithms are increasingly popular in biomedical research due to their compelling ability to identify discrete subgroups in data, and their increasing accessibility in mainstream software. While guidelines exist for algorithm selection and outcome evaluation, there are no firmly established ways of computing a…
Spectral clustering is a fast and popular algorithm for finding clusters in networks. Recently, Chaudhuri et al. (2012) and Amini et al.(2012) proposed inspired variations on the algorithm that artificially inflate the node degrees for improved statistical performance. The current paper extends the previous statistical…
A cluster driven log-volatility factor model: a deepening on the source of the volatility clusteringq-fin.ST
We introduce a new factor model for log volatilities that performs dimensionality reduction and considers contributions globally through the market, and locally through cluster structure and their interactions. We do not assume a-priori the number of clusters in the data, instead using the Directed Bubble Hierarchical …
Researchers have used from 30 days to several years of daily returns as source data for clustering financial time series based on their correlations. This paper sets up a statistical framework to study the validity of such practices. We first show that clustering correlated random variables from their observed values i…
Assessment of risk levels for existing credit accounts is important to the implementation of bank policies and offering financial products. This paper uses cluster analysis of behaviour of credit card accounts to help assess credit risk level. Account behaviour is modelled parametrically and we then implement the behav…
Clustering analysis has become a ubiquitous information retrieval tool in a wide range of domains, but a more automatic framework is still lacking. Though internal metrics are the key players towards a successful retrieval of clusters, their effectiveness on real-world datasets remains not fully understood, mainly beca…
Develops a new tensor model for clustering with degree correction.
problem Clustering with unknown degree heterogeneity in multiway data.
method Degree-corrected tensor block model with estimation guarantees.
result Demonstrates an intrinsic statistical-to-computational gap for tensors of order three or greater.
Asymptotics for The -meansstat.ML
A new -means method improves clustering accuracy and robustness.
problem Insufficient asymptotic properties in existing -means methods.
method Introducing clustering consistency and proposing a new -means method.
result The new -means method has lower clustering error rates and is more robust.
Study provides guarantees for kernel clustering under non-parametric mixtures.
problem Statistical guarantees for kernel-based clustering without strong assumptions.
method Non-parametric mixture models, kernel-based clustering, consistency guarantees.
result Necessary and sufficient separability conditions for consistent clustering recovery.
Multi-level conformal clustering: A distribution-free technique for clustering and anomaly detectionstat.ML
In this work we present a clustering technique called \textit{multi-level conformal clustering (MLCC)}. The technique is hierarchical in nature because it can be performed at multiple significance levels which yields greater insight into the data than performing it at just one level. We describe the theoretical underpi…
We give complete algorithms and source code for constructing (multilevel) statistical industry classifications, including methods for fixing the number of clusters at each level (and the number of levels). Under the hood there are clustering algorithms (e.g., k-means). However, what should we cluster? Correlations? Ret…
We address the problem of communicating domain knowledge from a user to the designer of a clustering algorithm. We propose a protocol in which the user provides a clustering of a relatively small random sample of a data set. The algorithm designer then uses that sample to come up with a data representation under which …
We present *K-means clustering algorithm and source code by expanding statistical clustering methods applied in https://ssrn.com/abstract=2802753 to quantitative finance. *K-means is statistically deterministic without specifying initial centers, etc. We apply *K-means to extracting cancer signatures from genome data w…
Identification of clusters of investors from their real trading activity in a financial marketq-fin.TR
We use statistically validated networks, a recently introduced method to validate links in a bipartite system, to identify clusters of investors trading in a financial market. Specifically, we investigate a special database allowing to track the trading activity of individual investors of the stock Nokia. We find that …
Paper proposes efficient methods for clustering and signal recovery in high-dimensional data with block structures.
problem High-dimensional clustering and signal recovery under block signal structures.
method CFA-PCA and MA-PCA methods for sparse and dense block signals.
result Proposed methods achieve computational minimax optimality for clustering and signal recovery.
Clusters of financial market states identified over 2006-2019.
problem Understanding the statistical properties of financial markets.
method Clustering analysis of correlation matrices constructed from sliding epochs.
result Financial markets can be classified into distinct states with transitions indicating precursors to catastrophic events.
K-ARMA models cluster time series data robustly.
problem Clustering time series data effectively.
method Model-based K-ARMA clustering algorithm with robust outlier detection.
result K-ARMA models outperform existing methods for time series clustering.
New clustering method adapts to data structure.
problem Natural adaptability of clustering order in EM algorithm.
method Minimizes mixing entropy to define relative entropic order.
result Empirical relative entropic order is consistent.
We consider two closely related problems: planted clustering and submatrix localization. The planted clustering problem assumes that a random graph is generated based on some underlying clusters of the nodes; the task is to recover these clusters given the graph. The submatrix localization problem concerns locating hid…
Unified framework for clustering with auxiliary data.
problem Clustering with datasets reflecting similar but different latent structures.
method Adaptive Transfer Clustering (ATC) algorithm that optimizes bias-variance decomposition.
result ATC proves optimal under Gaussian mixture model and shows transfer benefits.
This paper provides a new unimodality test with application in hierarchical clustering methods. The proposed method denoted by signature test (Sigtest), transforms the data based on its statistics. The transformed data has much smaller variation compared to the original data and can be evaluated in a simple proposed un…
This paper introduces individual fairness in clustering using -divergence.
problem Ensuring fair clustering by treating similar individuals similarly.
method Uses -divergence to measure statistical similarity and assigns individuals to probability distributions over cluster centers.
result Provides an algorithm with provable approximation guarantee for clustering with individual fairness constraints.
Network Lasso clusters sparse graph clusters efficiently.
problem Local graph clustering of sparse and chain-like clusters.
method Network Lasso minimizes total variation of cluster indicator signals.
result Network Lasso handles sparse clusters difficult for spectral clustering.
In this work we develop a theory of hierarchical clustering for graphs. Our modeling assumption is that graphs are sampled from a graphon, which is a powerful and general model for generating graphs and analyzing large networks. Graphons are a far richer class of graph models than stochastic blockmodels, the primary se…
Study provides selective inference method for latent block models.
problem Challenges in constructing a test on a block structure selected by clustering algorithms.
method Developed a selective inference method for latent block models using squared residue minimization and simulated annealing.
result Proposed tests effectively handle selective bias in block structures compared to naive tests.