Paper studies M-estimators with derivatives and residual distribution for robust adaptive tuning.
problem Tackles robustness and adaptive tuning of M-estimators with heavy-tailed noise.
method Provides formulae for derivatives, characterizes residual distribution, proposes adaptive criterion.
result Characterizes distribution of residuals and proposes adaptive criterion as out-of-sample error proxy.
Paper improves ML estimation from incomplete data with robust M-estimator.
problem Estimating parameters from incomplete data with improved accuracy.
method Developed a robust M-estimator and a sandwich estimator for standard errors.
result Improved estimation accuracy with smaller standard errors than ML estimates.
Improved Sparse Polyak for high-dimensional M-estimation with sparser solutions.
problem High-dimensional M-estimation problems with potential loss of sparsity and accuracy.
method Variant of Sparse Polyak with optimal thresholding operators.
result Retains desirable scaling properties while achieving sparser and more accurate solutions.
Theoretical framework for M-posteriors connects Bayesian and frequentist statistics.
problem Connecting Bayesian and frequentist approaches in statistical inference.
method Developed a theoretical framework for M-posteriors, showing asymptotic normality and frequentist consistency.
result M-posteriors are robust and contract around M-estimators under mild conditions.
Unified representation of density-power-based divergences simplifies estimation to M-estimation.
problem Outliers in density estimation.
method Define a norm-based Bregman density power divergence (NB-DPD) that reduces to M-estimation.
result NB-DPD connects and generalizes existing divergences, highlighting robustness properties.
Regularized M-estimators are used in diverse areas of science and engineering to fit high-dimensional models with some low-dimensional structure. Usually the low-dimensional structure is encoded by the presence of the (unknown) parameters in some low-dimensional model subspace. In such settings, it is desirable for est…
When recovering an unknown signal from noisy measurements, the computational difficulty of performing optimal Bayesian MMSE (minimum mean squared error) inference often necessitates the use of maximum a posteriori (MAP) inference, a special case of regularized M-estimation, as a surrogate. However, MAP is suboptimal in…
Paper proves robust M-estimators' coordinates' normality in high dimensions.
problem High-dimensional robust M-estimators' asymptotic normality.
method Develops Stein formulae for high-dimensional random vectors on the sphere.
result Asymptotic normality holds for most coordinates of robust M-estimators with convex penalty.
This paper considers the problem of robust subspace recovery: given a set of N points in RD, if many lie in a d-dimensional subspace, then can we recover the underlying subspace? We show that Tyler's M-estimator can be used to recover the underlying subspace, if the percentage of the inliers is larger t…
A new robust PCA estimator combining M-estimators and minimum divergence estimators.
problem Adverse effect of outlying observations in PCA for high-dimensional data.
method Minimum density power divergence estimator combined with a computationally efficient algorithm.
result High breakdown guarantee regardless of data dimension with theoretical support and practical applications.
A popular regularized (shrinkage) covariance estimator is the shrinkage sample covariance matrix (SCM) which shares the same set of eigenvectors as the SCM but shrinks its eigenvalues toward its grand mean. In this paper, a more general approach is considered in which the SCM is replaced by an M-estimator of scatter ma…
Study examines influence diagnostics in high-dimensional M-estimation.
problem Understanding influence diagnostics in high-dimensional settings.
method Characterized the distribution of leave-one-out influences in high-dimensional Gaussian M-estimation.
result The distribution of influences converges to a limiting measure in high-dimensional settings.
Optimizes private statistics with noisy methods.
problem Private inference in statistical models.
method Noisy optimization for M-estimators and confidence regions.
result Private estimators converge to non-private ones with high probability.
A large dimensional characterization of robust M-estimators of covariance (or scatter) is provided under the assumption that the dataset comprises independent (essentially Gaussian) legitimate samples as well as arbitrary deterministic samples, referred to as outliers. Building upon recent random matrix advances in the…
Tyler's M-estimator's phase transition at DS-SNR = 1 is resolved.
problem Robust Subspace Recovery
method Tyler's M-estimator
result TME converges exactly to the true subspace for DS-SNR >= 1 under a new stability condition.
This paper analyzes M-estimators under infinite-variance noise in high dimensions.
problem High-dimensional M-estimation with infinite-variance noise.
method Study of the Fenchel conjugate domain and its impact on risk.
result Exact risk of M-estimators under infinite-variance noise is derived.
In this paper, we investigate the adversarial robustness of multivariate M-Estimators. In the considered model, after observing the whole dataset, an adversary can modify all data points with the goal of maximizing inference errors. We use adversarial influence function (AIF) to measure the asymptotic rate at which t…
This paper improves robust cluster enumeration for RES data.
problem Challenges in determining optimal clusters in noisy data.
method Generalizes robust Bayesian cluster enumeration for RES mixtures.
result Significant robustness improvement over existing methods.
Paper proposes robust estimators for heavy-tailed data with infinite variance.
problem Developing robust estimators for heavy-tailed data with infinite variance.
method Proposes two robust estimators: ridge log-truncated M-estimator and elastic net log-truncated M-estimator.
result Demonstrates robustness of log-truncated estimations over standard estimations through simulations and real data analysis.
We propose a method for nonparametric density estimation that exhibits robustness to contamination of the training sample. This method achieves robustness by combining a traditional kernel density estimator (KDE) with ideas from classical M-estimation. We interpret the KDE based on a radial, positive semi-definite ke…
Estimates error for robust M-estimators with convex penalties.
problem Estimating out-of-sample error for robust M-estimators in high-dimensional linear regression.
method Proposes a generic out-of-sample error estimate for robust M-estimators with convex penalties, using observed data and derivatives. result The out-of-sample error estimate has a relative error of order n−1/2 under certain conditions. The paper analyzes the risk of bagging regularized M-estimators under proportional asymptotics.
problem Characterizing the risk of ensemble estimators trained with subsamples and regularizers.
method Developed a consistent estimator for the risk of ensemble estimators under proportional asymptotics.
result Optimal subsample size k⋆ tends to be in the overparameterized regime for the full-ensemble estimator. We study theoretical properties of regularized robust M-estimators, applicable when data are drawn from a sparse high-dimensional linear model and contaminated by heavy-tailed distributions and/or outliers in the additive errors and covariates. We first establish a form of local statistical consistency for the penalize…
Study proposes an active subsampling method for estimating individualized thresholds in high-dimensional data.
problem Estimating optimal individualized thresholds in high-dimensional data with limited labeled samples.
method Developed a K-step active subsampling algorithm to iteratively select and label the most informative data points.
result Revealed a phase transition phenomenon in the estimation of θ with respect to the smoothness of the conditional density. New algorithm improves learning in noisy networks with robust performance.
problem Improving learning in noisy, distributed networks.
method Diffusion normalized least mean M-estimate algorithm with sparse-aware variant.
result The proposed algorithms outperform existing diffusion algorithms in impulsive noise scenarios.
New method approximates M-estimator and predictions without solving fixed-point equations.
problem Characterize behavior of M-estimator and predictions in single index models.
method Develops data-driven observable adjustments to proximal operators.
result Empirical distributions of M-estimator and predictions are approximated without solving fixed-point equations.
Adaptive inference for M-estimators in bandit data with model misspecification.
problem Challenges in off-policy inference for adaptively collected bandit data with a misspecified model.
method A novel approach to define a projected solution over a stationary evaluation policy, stabilizing variance with flexible methods.
result Valid inference for M-estimators in adaptive settings, even with unstable treatment policies. Improved statistical inference for expensive data using machine learning predictions.
problem Statistical inference under adaptive two-phase multiwave sampling with expensive measurements.
method Multiwave Predict-Then-Debias estimator combining proxy information and expensive measurements.
result Valid estimators and confidence intervals for M-estimation under adaptive sampling.
New risk class defined based on loss location and deviation.
problem Risk assessment in loss distributions.
method Wrapper around smooth loss functions, M-estimators, stochastic gradient methods.
result Finite-sample stationarity guarantees for stochastic gradient methods.
We consider the class of optimization problems arising from computationally intensive L1-regularized M-estimators, where the function or gradient values are very expensive to compute. A particular instance of interest is the L1-regularized MLE for learning Conditional Random Fields (CRFs), which are a popular class of …
Paper improves parameter estimation of continuous distributions using preference feedback.
problem Improving parameter estimation of continuous distributions.
method Preference-based M-estimators and deterministic preferences.
result Preference-based estimators achieve an estimation error scaling of O(1/n), significantly faster than sample-only methods.
Develop a comprehensive theory for regularized M-estimation in reproducing kernel Hilbert spaces.
problem Regularized M-estimation in reproducing kernel Hilbert spaces
method Existence and measurability of the estimator, sharp rates of convergence
result New rates for tensor product Sobolev spaces
For the problem of high-dimensional sparse linear regression, it is known that an ℓ0-based estimator can achieve a 1/n "fast" rate on the prediction error without any conditions on the design matrix, whereas in absence of restrictive conditions on the design matrix, popular polynomial-time methods only guarante…
Study improves BN TTA under distribution shift using higher-order asymptotics.
problem Improving BN TTA for changing data distributions.
method Integrates Edgeworth expansion and saddlepoint approximation with one-step M-estimation.
result Derives optimal weighting parameter for minimized mean-squared error.
The paper develops methods to handle missing data using regularized M-estimation in reproducing kernel Hilbert space.
problem Handling missing data in statistical analysis.
method Kernel ridge regression for imputation and maximum entropy method for propensity score estimation.
result The proposed methods achieve statistical consistency and asymptotic equivalence.
Efficiently estimates shrinkage coefficient for RTME using LOOCV approximation.
problem Estimating optimal shrinkage coefficient for Regularized Tyler's M-estimator.
method Proposes an approximate LOOCV method to estimate α efficiently. result Significant speedup and accuracy improvement over existing methods.
The paper develops an asymptotic theory of self-supervised pre-training.
problem Sharpness of current rates in self-supervised pre-training and their accuracy.
method Two-stage M-estimation and tools from Riemannian geometry.
result Characterization of the limiting distribution of the downstream test risk.
New algorithms improve robust covariance estimation efficiency.
problem Efficiently compute robust covariance estimation for large datasets.
method Frank-Wolfe-based algorithms for Tyler's M-estimator.
result AFW and GAFW converge linearly under mild assumptions.
In certain situations that shall be undoubtedly more and more common in the Big Data era, the datasets available are so massive that computing statistics over the full sample is hardly feasible, if not unfeasible. A natural approach in this context consists in using survey schemes and substituting the "full data" stati…
Extends Infinitesimal Jackknife for model covariance, enhancing ensemble model analysis.
problem Estimating uncertainty in combinations of models.
method Extends Infinitesimal Jackknife to estimate covariance between models.
result Theoretical consistency of Infinitesimal Jackknife covariance estimate demonstrated.
We consider the problem of robustifying high-dimensional structured estimation. Robust techniques are key in real-world applications which often involve outliers and data corruption. We focus on trimmed versions of structurally regularized M-estimators in the high-dimensional setting, including the popular Least Trimme…
c-lasso is a Python tool for robust and sparse regression with linear constraints.
problem Sparse and robust linear regression with linear constraints.
method Estimates coefficients and scale under linear constraints using perspective M-estimators.
result Provides estimators for various loss functions with linear constraints.
New GIC improves model selection for structured sparse models.
problem Model selection and regularization in high-dimensional scenarios.
method Proposes a new Generalized Information Criteria (GIC) for structured sparse models.
result Obtains non-asymptotic model selection bounds and sufficient conditions for model selection consistency.
We introduce a novel regression framework which simultaneously models the quantile and the Expected Shortfall (ES) of a response variable given a set of covariates. This regression is based on a strictly consistent loss function for the pair quantile and ES, which allows for M- and Z-estimation of the joint regression …
Undirected graphical models, or Markov networks, are a popular class of statistical models, used in a wide variety of applications. Popular instances of this class include Gaussian graphical models and Ising models. In many settings, however, it might not be clear which subclass of graphical models to use, particularly…
Starting at a saddle tower surface, we give a new existence proof of the Lawson surfaces ξm,k of high genus by deforming the corresponding DPW potential. As a byproduct, we obtain for fixed m estimates on the area of ξm,k in terms of their genus g=mk≫1.
Paper presents a new framework for covariance matrix estimation with geometric insights.
problem Challenges in covariance matrix estimation, especially in finding suitable models and efficient estimation methods.
method General framework for linear restrictions on different transformations of the covariance matrix, including matrix logarithm and its inverse.
result Yields an M-estimator with M-estimation allowing for straightforward asymptotic and finite sample analysis. Develops asymptotic theory for adversarial estimators.
problem Estimating unknown functions in machine learning and econometrics.
method Derives convergence rates and normality of A-estimators under various conditions.
result Normality of neural-net M-estimators, overcoming previous technical issues.