Regularized M-estimators are used in diverse areas of science and engineering to fit high-dimensional models with some low-dimensional structure. Usually the low-dimensional structure is encoded by the presence of the (unknown) parameters in some low-dimensional model subspace. In such settings, it is desirable for est…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Regularized M-estimators”
Paper studies M-estimators with derivatives and residual distribution for robust adaptive tuning.
problem Tackles robustness and adaptive tuning of M-estimators with heavy-tailed noise.
method Provides formulae for derivatives, characterizes residual distribution, proposes adaptive criterion.
result Characterizes distribution of residuals and proposes adaptive criterion as out-of-sample error proxy.
The paper analyzes the risk of bagging regularized M-estimators under proportional asymptotics.
problem Characterizing the risk of ensemble estimators trained with subsamples and regularizers.
method Developed a consistent estimator for the risk of ensemble estimators under proportional asymptotics.
result Optimal subsample size tends to be in the overparameterized regime for the full-ensemble estimator.
This paper analyzes M-estimators under infinite-variance noise in high dimensions.
problem High-dimensional M-estimation with infinite-variance noise.
method Study of the Fenchel conjugate domain and its impact on risk.
result Exact risk of M-estimators under infinite-variance noise is derived.
When recovering an unknown signal from noisy measurements, the computational difficulty of performing optimal Bayesian MMSE (minimum mean squared error) inference often necessitates the use of maximum a posteriori (MAP) inference, a special case of regularized M-estimation, as a surrogate. However, MAP is suboptimal in…
We study theoretical properties of regularized robust M-estimators, applicable when data are drawn from a sparse high-dimensional linear model and contaminated by heavy-tailed distributions and/or outliers in the additive errors and covariates. We first establish a form of local statistical consistency for the penalize…
New method approximates M-estimator and predictions without solving fixed-point equations.
problem Characterize behavior of M-estimator and predictions in single index models.
method Develops data-driven observable adjustments to proximal operators.
result Empirical distributions of M-estimator and predictions are approximated without solving fixed-point equations.
Paper proposes a new method for covariance estimation using M-estimators with eigenvalue shrinkage.
problem Estimating covariance matrices in heavy-tailed distributions.
method Replaces shrinkage sample covariance matrix with M-estimator of scatter matrix and optimizes shrinkage parameter.
result Shrinkage M-estimators outperform shrinkage SCM in heavy-tailed distributions.
We consider the class of optimization problems arising from computationally intensive L1-regularized M-estimators, where the function or gradient values are very expensive to compute. A particular instance of interest is the L1-regularized MLE for learning Conditional Random Fields (CRFs), which are a popular class of …
Optimal prediction for sparse linear models? Lower bounds for coordinate-separable M-estimatorsmath.ST
For the problem of high-dimensional sparse linear regression, it is known that an -based estimator can achieve a "fast" rate on the prediction error without any conditions on the design matrix, whereas in absence of restrictive conditions on the design matrix, popular polynomial-time methods only guarante…
Efficiently estimates shrinkage coefficient for RTME using LOOCV approximation.
problem Estimating optimal shrinkage coefficient for Regularized Tyler's M-estimator.
method Proposes an approximate LOOCV method to estimate efficiently.
result Significant speedup and accuracy improvement over existing methods.
Paper proves robust M-estimators' coordinates' normality in high dimensions.
problem High-dimensional robust M-estimators' asymptotic normality.
method Develops Stein formulae for high-dimensional random vectors on the sphere.
result Asymptotic normality holds for most coordinates of robust M-estimators with convex penalty.
New GIC improves model selection for structured sparse models.
problem Model selection and regularization in high-dimensional scenarios.
method Proposes a new Generalized Information Criteria (GIC) for structured sparse models.
result Obtains non-asymptotic model selection bounds and sufficient conditions for model selection consistency.
The paper develops methods to handle missing data using regularized M-estimation in reproducing kernel Hilbert space.
problem Handling missing data in statistical analysis.
method Kernel ridge regression for imputation and maximum entropy method for propensity score estimation.
result The proposed methods achieve statistical consistency and asymptotic equivalence.
Develop a comprehensive theory for regularized M-estimation in reproducing kernel Hilbert spaces.
problem Regularized M-estimation in reproducing kernel Hilbert spaces
method Existence and measurability of the estimator, sharp rates of convergence
result New rates for tensor product Sobolev spaces
Regularized M-estimators with nonconvexity: Statistical and algorithmic theory for local optimamath.ST
We provide novel theoretical results regarding local optima of regularized -estimators, allowing for nonconvexity in both loss and penalty functions. Under restricted strong convexity on the loss and suitable regularity conditions on the penalty, we prove that \emph{any stationary point} of the composite objective f…
Study proposes an active subsampling method for estimating individualized thresholds in high-dimensional data.
problem Estimating optimal individualized thresholds in high-dimensional data with limited labeled samples.
method Developed a K-step active subsampling algorithm to iteratively select and label the most informative data points.
result Revealed a phase transition phenomenon in the estimation of with respect to the smoothness of the conditional density.
Estimates error for robust M-estimators with convex penalties.
problem Estimating out-of-sample error for robust M-estimators in high-dimensional linear regression.
method Proposes a generic out-of-sample error estimate for robust -estimators with convex penalties, using observed data and derivatives.
result The out-of-sample error estimate has a relative error of order under certain conditions.
MTLRRC improves MTL by robustly clustering tasks and detecting outliers.
problem Improving MTL by handling outlier tasks and sharing common information.
method Robust regularized clustering with non-convex group penalties.
result MTLRRC effectively detects and clusters tasks, improving overall performance.
Many statistical -estimators are based on convex optimization problems formed by the combination of a data-dependent loss function with a norm-based regularizer. We analyze the convergence rates of projected gradient and composite gradient methods for solving such problems, working within a high-dimensional framewor…
Paper improves ML estimation from incomplete data with robust M-estimator.
problem Estimating parameters from incomplete data with improved accuracy.
method Developed a robust M-estimator and a sandwich estimator for standard errors.
result Improved estimation accuracy with smaller standard errors than ML estimates.
We consider the problem of robustifying high-dimensional structured estimation. Robust techniques are key in real-world applications which often involve outliers and data corruption. We focus on trimmed versions of structurally regularized M-estimators in the high-dimensional setting, including the popular Least Trimme…
Improved Sparse Polyak for high-dimensional M-estimation with sparser solutions.
problem High-dimensional M-estimation problems with potential loss of sparsity and accuracy.
method Variant of Sparse Polyak with optimal thresholding operators.
result Retains desirable scaling properties while achieving sparser and more accurate solutions.
Theoretical framework for M-posteriors connects Bayesian and frequentist statistics.
problem Connecting Bayesian and frequentist approaches in statistical inference.
method Developed a theoretical framework for M-posteriors, showing asymptotic normality and frequentist consistency.
result M-posteriors are robust and contract around M-estimators under mild conditions.
Unified representation of density-power-based divergences simplifies estimation to M-estimation.
problem Outliers in density estimation.
method Define a norm-based Bregman density power divergence (NB-DPD) that reduces to M-estimation.
result NB-DPD connects and generalizes existing divergences, highlighting robustness properties.
We introduce a novel regression framework which simultaneously models the quantile and the Expected Shortfall (ES) of a response variable given a set of covariates. This regression is based on a strictly consistent loss function for the pair quantile and ES, which allows for M- and Z-estimation of the joint regression …
This paper considers the problem of robust subspace recovery: given a set of points in , if many lie in a -dimensional subspace, then can we recover the underlying subspace? We show that Tyler's M-estimator can be used to recover the underlying subspace, if the percentage of the inliers is larger t…
A new robust PCA estimator combining M-estimators and minimum divergence estimators.
problem Adverse effect of outlying observations in PCA for high-dimensional data.
method Minimum density power divergence estimator combined with a computationally efficient algorithm.
result High breakdown guarantee regardless of data dimension with theoretical support and practical applications.
In nonparametric classification and regression problems, regularized kernel methods, in particular support vector machines, attract much attention in theoretical and in applied statistics. In an abstract sense, regularized kernel methods (simply called SVMs here) can be seen as regularized M-estimators for a parameter …
Generative approach speeds hyperparameter tuning for machine learning models.
problem Computational infeasibility of cross-validation and difficulty of fully Bayesian hyper-parameter learning.
method Combines optimization-based approximations and amortization techniques.
result Rapid evaluation of hyper-parameters over grids or ranges, supporting predictive tuning and uncertainty quantification.
New method identifies subgroups in censored data.
problem Identifying meaningful patterns in heterogeneous populations.
method Combining inverse probability weighting, M-estimation, and concave pairwise fusion penalization.
result Robust approach for censored data under heterogeneous AFT models.
Ever since the proof of asymptotic normality of maximum likelihood estimator by Cramer (1946), it has been understood that a basic technique of the Taylor series expansion suffices for asymptotics of -estimators with smooth/differentiable loss function. Although the Taylor series expansion is a purely deterministic …
Many statistical estimators for high-dimensional linear regression are M-estimators, formed through minimizing a data-dependent square loss function plus a regularizer. This work considers a new class of estimators implicitly defined through a discretized gradient dynamic system under overparameterization. We show that…
Study examines influence diagnostics in high-dimensional M-estimation.
problem Understanding influence diagnostics in high-dimensional settings.
method Characterized the distribution of leave-one-out influences in high-dimensional Gaussian M-estimation.
result The distribution of influences converges to a limiting measure in high-dimensional settings.
Optimizes private statistics with noisy methods.
problem Private inference in statistical models.
method Noisy optimization for M-estimators and confidence regions.
result Private estimators converge to non-private ones with high probability.
A large dimensional characterization of robust M-estimators of covariance (or scatter) is provided under the assumption that the dataset comprises independent (essentially Gaussian) legitimate samples as well as arbitrary deterministic samples, referred to as outliers. Building upon recent random matrix advances in the…
We propose a novel, efficient approach for distributed sparse learning in high-dimensions, where observations are randomly partitioned across machines. Computationally, at each round our method only requires the master machine to solve a shifted ell_1 regularized M-estimation problem, and other workers to compute the g…
Tyler's M-estimator's phase transition at DS-SNR = 1 is resolved.
problem Robust Subspace Recovery
method Tyler's M-estimator
result TME converges exactly to the true subspace for DS-SNR >= 1 under a new stability condition.
In this paper, we investigate the adversarial robustness of multivariate -Estimators. In the considered model, after observing the whole dataset, an adversary can modify all data points with the goal of maximizing inference errors. We use adversarial influence function (AIF) to measure the asymptotic rate at which t…
Robust M-Estimation Based Bayesian Cluster Enumeration for Real Elliptically Symmetric Distributionseess.SP
This paper improves robust cluster enumeration for RES data.
problem Challenges in determining optimal clusters in noisy data.
method Generalizes robust Bayesian cluster enumeration for RES mixtures.
result Significant robustness improvement over existing methods.
Paper proposes robust estimators for heavy-tailed data with infinite variance.
problem Developing robust estimators for heavy-tailed data with infinite variance.
method Proposes two robust estimators: ridge log-truncated M-estimator and elastic net log-truncated M-estimator.
result Demonstrates robustness of log-truncated estimations over standard estimations through simulations and real data analysis.
We propose a method for nonparametric density estimation that exhibits robustness to contamination of the training sample. This method achieves robustness by combining a traditional kernel density estimator (KDE) with ideas from classical -estimation. We interpret the KDE based on a radial, positive semi-definite ke…
We study Empirical Risk Minimizers (ERM) and Regularized Empirical Risk Minimizers (RERM) for regression problems with convex and -Lipschitz loss functions. We consider a setting where $|\cO|$ malicious outliers contaminate the labels. In that case, under a local Bernstein condition, we show that the -error rat…
New algorithm improves learning in noisy networks with robust performance.
problem Improving learning in noisy, distributed networks.
method Diffusion normalized least mean M-estimate algorithm with sparse-aware variant.
result The proposed algorithms outperform existing diffusion algorithms in impulsive noise scenarios.
Adaptive inference for -estimators in bandit data with model misspecification.
problem Challenges in off-policy inference for adaptively collected bandit data with a misspecified model.
method A novel approach to define a projected solution over a stationary evaluation policy, stabilizing variance with flexible methods.
result Valid inference for -estimators in adaptive settings, even with unstable treatment policies.
M-estimation under Two-Phase Multiwave Sampling with Applications to Prediction-Powered Inferencestat.ME
Improved statistical inference for expensive data using machine learning predictions.
problem Statistical inference under adaptive two-phase multiwave sampling with expensive measurements.
method Multiwave Predict-Then-Debias estimator combining proxy information and expensive measurements.
result Valid estimators and confidence intervals for M-estimation under adaptive sampling.
New risk class defined based on loss location and deviation.
problem Risk assessment in loss distributions.
method Wrapper around smooth loss functions, M-estimators, stochastic gradient methods.
result Finite-sample stationarity guarantees for stochastic gradient methods.
The time-evolving precision matrix of a piecewise-constant Gaussian graphical model encodes the dynamic conditional dependency structure of a multivariate time-series. Traditionally, graphical models are estimated under the assumption that data is drawn identically from a generating distribution. Introducing sparsity a…