Paper analyzes and accelerates Langevin Monte Carlo methods using large deviations theory.
problem High-dimensional sampling problems in machine learning.
method Unified approach using large deviations theory to study and accelerate Langevin dynamics variants.
result Efficiency of Langevin dynamics variants demonstrated through numerical experiments.
Infinitesimal boosting converges to a deterministic process in large sample limit.
problem Characterizing the asymptotic behavior of infinitesimal gradient boosting in large sample sizes.
method Proving convergence to a deterministic process using large sample theory and differential equations.
result The test error decreases over time in the population limit.
We analyze SGAs for statistical inference via asymptotics, improving tuning methods.
problem Improper tuning of SGAs for optimization and sampling.
method Characterize large-sample asymptotics of SGAs via step-size and sample-size scaling limits.
result Iterate averaging with large step size is robust and asymptotically has covariance proportional to MLE's.
The paper introduces a sampling theory for graphons with a Poincaré inequality and proves consistency.
problem Sampling on large graphs is challenging due to their non-Euclidean nature.
method The paper introduces a signal sampling theory for graphons, proving a Poincaré inequality and showing consistency.
result Unique sampling sets for graphon signals are consistent across graph sequences.
Estimates covariance matrices with correlations between samples.
problem Estimating large-dimensional covariance matrices with correlated samples.
method Generalized Marcenko-Pastur equation and Ledoit-Peche shrinkage estimator using random matrix theory and free probability. Developed an efficient algorithm based on Ledoit-Wolf kernel estimation.
result Efficient algorithm for estimating large covariance matrices with correlations.
Develops large-sample theory for non-stationary source separation.
problem Lack of large-sample results for non-stationary source separation methods.
method Large-sample theory for NSS-JD method under specific assumptions.
result Consistency of unmixing estimator and its convergence to Gaussian distribution.
New framework analyzes SGD dynamics in large samples and dimensions.
problem Analyzing stochastic gradient descent in large-scale settings.
method Inspired by random matrix theory, new framework for fixed stepsize and finite sum settings.
result SGD dynamics become deterministic in the large sample and dimensional limit, governed by a Volterra integral equation.
A new method improves active learning for large batch sizes.
problem Challenges in scaling Bayesian active learning to large batch sizes.
method Derives Partial Batch Label Sampling (ParBaLS) for EPIG algorithm.
result ParBaLS EPIG outperforms top- B B B selection and BatchBALD. Sharp concentration bounds for i.i.d. variables.
problem Controlling the tail probabilities of independent variables.
method Extension of Sanov's theorem using large deviations and information theory.
result Matching concentration and anti-concentration bounds for i.i.d. samples of any size.
LoCoV reduces portfolio optimization errors from sample covariance matrices.
problem Large errors in sample covariance matrix for optimal portfolio weights.
method LoCoV (low dimension covariance voting) algorithm to reduce these errors.
result LoCoV outperforms classical methods in portfolio optimization experiments.
Improved learning theory for kernel distribution regression with two-stage sampling.
problem Distribution regression problem and two-stage sampling setting.
method Kernel methods, near-unbiased condition, new error bounds, convergence rates.
result Strictly improved convergence rates for three important classes of kernels.
Paper introduces SPADE method to protect classifiers from OOD and adversarial samples.
problem Protecting classifiers from out-of-distribution and adversarial samples.
method SPADE method based on GEV model in latent space.
result Provable protection against OOD and adversarial samples.
Sharp statistical theory for conditional diffusion models.
problem Lack of theoretical foundation for conditional diffusion models.
method Sharp statistical theory with approximation of conditional score function.
result Sample complexity bound that adapts to data distribution smoothness.
We introduce a simple framework for designing private boosting algorithms. We give natural conditions under which these algorithms are differentially private, efficient, and noise-tolerant PAC learners. To demonstrate our framework, we use it to construct noise-tolerant and private PAC learners for large-margin halfspa…
Study precise sample covariance error for Gaussian centered data.
problem Precise characterization of sample covariance error for Gaussian data.
method Developed a Random Duality Theory (RDT) framework to determine upper and lower bounds.
result Upper and lower bounds match in large-dimensional contexts, matching the spectral norm's limiting value.
The paper explores learning good policies from past data in large state spaces.
problem Learning good policies from historical data in large state spaces.
method Introduces expressivity assumptions and data coverage for function approximation and algorithmic design.
result A variety of algorithms and their guarantees are presented based on assumptions and desired complexity.
LIIT uses large deviations to train neural networks faster.
problem Training neural networks requires large amounts of data, leading to inefficiencies.
method LIIT uses large deviations principle to generate and update training samples.
result LIIT achieves similar classification performance with smaller, more representative samples.
New method improves uncertainty quantification for large batch sizes and misspecified models.
problem Challenges in tuning algorithms for accurate uncertainty quantification in large batch sizes and misspecified models.
method Proposes new discrete-time approximations to SGD and SGLD, proving error bounds for practical tuning.
result Quantitative, non-asymptotic error bounds for accurate predictions of covariance and autocorrelation time.
The study proves sampling-based GNNs can approximate training on full graphs with small subgraphs.
problem Training Graph Neural Networks (GNNs) on large graphs is computationally expensive.
method Theoretical framework using graph local limits to prove approximation of GNN training on small samples.
result Parameters learned from sampling-based GNNs on small subgraphs are close to those on full graphs.
Particle-optimization-based sampling (POS) is a recently developed effective sampling technique that interactively updates a set of particles. A representative algorithm is the Stein variational gradient descent (SVGD). We prove, under certain conditions, SVGD experiences a theoretical pitfall, {\it i.e.}, particles te…
The paper improves Monte Carlo methods for optimization problems.
problem Efficiently solving optimization problems with biased Monte Carlo estimators.
method Introduces Multilevel Monte Carlo (MLMC) within Sample Average Approximation (SAA).
result Establishes uniform convergence and sample complexity for MLMC in SAA.
Unified framework for finite-sample RL algorithms using Lyapunov theory.
problem Finite-sample convergence guarantees of asynchronous RL algorithms.
method Reformulate RL algorithms as Markovian SA, develop Lyapunov analysis.
result Mean-square error bounds and convergence for various RL algorithms.
Study shows how large neural networks avoid overfitting through decoupling of feature learning and complexity growth.
problem Understanding inductive bias and generalization in large neural networks.
method Dynamical mean field theory applied to large two-layer networks.
result Training dynamics of large networks exhibit a separation of timescales, decoupling feature learning and overfitting.
A novel distributed adaptive NN classifier for large data sets.
problem Handling large and distributed data for efficient classification.
method Distributed adaptive nearest neighbor classifier with stochastic tuning parameter selection and early stopping rule.
result Achieves nearly optimal convergence rate under large sub-sample sizes.
Study on kernel tests for high-dimensional data, focusing on MMD and CLT.
problem Asymptotic behavior of kernel two-sample tests in high dimensions and large samples.
method Maximum mean discrepancy (MMD) with isotropic kernels, deriving asymptotic expansions and CLT.
result Interplay between moment discrepancy and dimension-and-sample orders in kernel tests.
Exact learning of tree-structured models with side info and noise.
problem Learning tree-structured graphical models with side information and noise.
method Probabilistic tools from strong large deviations theory.
result Exact asymptotics of structure learning from samples.
This work accelerates constrained sampling using large deviation principles.
problem Sampling constrained probability distributions efficiently.
method Large deviation principles applied to skew-reflected non-reversible Langevin dynamics.
result The skew-symmetric matrix accelerates convergence and reduces asymptotic variance.
HyperAgent improves RL exploration in large-scale problems.
problem Efficient exploration in large-scale reinforcement learning problems.
method Hypermodel framework for incremental posterior approximation without conjugacy.
result HyperAgent achieves logarithmic per-step computational complexity and sublinear regret.
Approximate Markov chain Monte Carlo (MCMC) offers the promise of more rapid sampling at the cost of more biased inference. Since standard MCMC diagnostics fail to detect these biases, researchers have developed computable Stein discrepancy measures that provably determine the convergence of a sample to its target dist…
For spherically symmetric distributions, efficient quantisation can be achieved with moderate sample sizes.
problem Optimal quantisation in high dimensions requires large sample sizes, making it impractical.
method Uniformly distributed random quantisers on a sphere of suitable radius achieve exceptional performance.
result For moderate sample sizes, quantisation error can be efficiently computed and approximated.
Blend-ASC improves self-consistency efficiency by dynamically allocating samples, reducing costs.
problem Efficiently applying self-consistency to large datasets is computationally expensive.
method Blend-ASC dynamically allocates samples during inference, improving efficiency.
result Blend-ASC reduces sample usage by 6.8x on average compared to vanilla self-consistency.
Large deviations for fat tailed distributions, i.e. those that decay slower than exponential, are not only relatively likely, but they also occur in a rather peculiar way where a finite fraction of the whole sample deviation is concentrated on a single variable. The regime of large deviations is separated from the regi…
New model predicts stock performance in large equity markets.
problem Predicting stock performance in large equity markets over long time horizons.
method Rank-based volatility stabilized models calibrated to empirical data.
result The model exhibits relative arbitrage and statistically fits empirical features.
The hidden tail of empirical distributions is analyzed using extreme value theory.
problem Understanding the bias between in-sample mean and true statistical mean for large n n n . method Extreme value theory applied to empirical distributions and their moments.
result The hidden moment of order 0 for power law distributions follows an exponential distribution with expectation 1 / n 1/n 1/ n . Marginal maximum likelihood (MML) estimation is the preferred approach to fitting item response theory models in psychometrics due to the MML estimator's consistency, normality, and efficiency as the sample size tends to infinity. However, state-of-the-art MML estimation procedures such as the Metropolis-Hastings Robbi…
New auditors assess f f f -DP privacy with adaptive sampling, avoiding large sample sizes.
problem Empirical auditing of f f f -DP privacy with adaptive sampling. method Shift focus to f f f -DP, develop adaptive auditors for whitebox and blackbox settings. result Adaptive auditors detect f f f -DP violations across the privacy spectrum with statistical guarantees. Relying on recent advances in statistical estimation of covariance distances based on random matrix theory, this article proposes an improved covariance and precision matrix estimation for a wide family of metrics. The method is shown to largely outperform the sample covariance matrix estimate and to compete with state…
New insights into negative sampling for graph representation learning.
problem Challenges in generating high-quality graph representations for large node sets.
method Theoretical analysis and derivation of negative sampling distribution correlation, proposing MCNS method.
result The negative sampling distribution should be positively but sub-linearly correlated to the positive sampling distribution.
Study non-monotonic loss functions in CRC, achieving valid risk control with large calibration samples.
problem Non-monotonic loss functions in CRC, violating existing theory's monotonicity assumption.
method Finite grid selection, calibration sample size analysis, Lipschitz continuity, monotonicity, distribution shift.
result Valid CRC achieved with large calibration samples, optimal excess risk rate of log ( m ) / n \sqrt{\log(m)/n} log ( m ) / n . Cer-Eval saves LLM evaluation costs while maintaining accuracy.
problem Challenges in evaluating large language models due to large dataset requirements.
method Adapts to different evaluation objectives, uses test sample complexity, and develops a partition-based algorithm.
result Cer-Eval can save 20-40% test points with comparable accuracy and 95% confidence guarantee.
Hypothesis testing for graphs has been an important tool in applied research fields for more than two decades, and still remains a challenging problem as one often needs to draw inference from few replicates of large graphs. Recent studies in statistics and learning theory have provided some theoretical insights about …
New theory explains contrastive learning via overlapping augmented views.
problem Lack of theoretical understanding of contrastive learning.
method Augmentation overlap perspective to improve downstream performance.
result Asymptotically closed bounds for downstream performance under weaker assumptions.
New neural network class reduces VC dimension, leading to better generalization.
problem VC theory struggles with explaining small generalization errors in overparametrized neural networks.
method Developed hyperplane arrangement neural networks (HANNs) and used sample compression analysis.
result HANNs can have significantly smaller VC dimension than the number of weights, yet remain highly expressive.
Theory for RLHF generalization under reward shift and clipped KL.
problem Theoretical understanding of RLHF generalization, especially with reward shift and clipped KL.
method Developed generalization theory for RLHF, accounting for reward shift and clipped KL.
result Presented generalization bounds for RLHF, suggesting generalization error from sampling, reward shift, and KL clipping.
This paper optimizes sampling for least-squares approximation.
problem Optimizing sampling for least-squares approximation in arbitrary linear spaces.
method Introducing the Christoffel function to construct near-optimal random sampling strategies.
result The number of samples scales log-linearly in the dimension of the approximation space.
GIV methodology extends instrumental variable estimation for high-dimensional data.
problem Estimating structural parameters in high-dimensional models with endogeneity and latent factors.
method Extends GIV methodology to large N and T, treats factors and loadings as unknown, and uses additional instruments for efficiency.
result Efficiency gains and negligible sampling errors in estimated instrument and factors.
Entropy regularized OT test assesses independence between samples.
problem Testing independence between two samples.
method Entropy regularized optimal transport.
result Non-asymptotic bounds for test statistic established.
In this work we develop a theory of hierarchical clustering for graphs. Our modeling assumption is that graphs are sampled from a graphon, which is a powerful and general model for generating graphs and analyzing large networks. Graphons are a far richer class of graph models than stochastic blockmodels, the primary se…