arXiv research

New algorithm improves learning in noisy networks with robust performance.

We propose a method for nonparametric density estimation that exhibits robustness to contamination of the training sample. This method achieves robustness by combining a traditional kernel density estimator (KDE) with ideas from classical $M$-estimation. We interpret the KDE based on a radial, positive semi-definite ke…

The ratio of two probability densities can be used for solving various machine learning tasks such as covariate shift adaptation (importance sampling), outlier detection (likelihood-ratio test), and feature selection (mutual information). Recently, several methods of directly estimating the density ratio have been deve…

Paper proposes a new method for covariance estimation using M-estimators with eigenvalue shrinkage.

Understanding efficiency in high dimensional linear models is a longstanding problem of interest. Classical work with smaller dimensional problems dating back to Huber and Bickel has illustrated the benefits of efficient loss functions. When the number of parameters $p$ is of the same order as the sample size $n$, $p \…

When recovering an unknown signal from noisy measurements, the computational difficulty of performing optimal Bayesian MMSE (minimum mean squared error) inference often necessitates the use of maximum a posteriori (MAP) inference, a special case of regularized M-estimation, as a surrogate. However, MAP is suboptimal in…

Adaptive inference for $M$-estimators in bandit data with model misspecification.

New risk class defined based on loss location and deviation.

For the problem of high-dimensional sparse linear regression, it is known that an $\ell_0$-based estimator can achieve a $1/n$ "fast" rate on the prediction error without any conditions on the design matrix, whereas in absence of restrictive conditions on the design matrix, popular polynomial-time methods only guarante…

We consider the problem of robustifying high-dimensional structured estimation. Robust techniques are key in real-world applications which often involve outliers and data corruption. We focus on trimmed versions of structurally regularized M-estimators in the high-dimensional setting, including the popular Least Trimme…

Study improves BN TTA under distribution shift using higher-order asymptotics.

Paper studies M-estimators with derivatives and residual distribution for robust adaptive tuning.

Generative approach speeds hyperparameter tuning for machine learning models.

Paper improves ML estimation from incomplete data with robust M-estimator.

Improved Sparse Polyak for high-dimensional M-estimation with sparser solutions.

Theoretical framework for M-posteriors connects Bayesian and frequentist statistics.

Unified representation of density-power-based divergences simplifies estimation to M-estimation.

Regularized M-estimators are used in diverse areas of science and engineering to fit high-dimensional models with some low-dimensional structure. Usually the low-dimensional structure is encoded by the presence of the (unknown) parameters in some low-dimensional model subspace. In such settings, it is desirable for est…

Paper proves robust M-estimators' coordinates' normality in high dimensions.

This paper considers the problem of robust subspace recovery: given a set of $N$ points in $\mathbb{R}^D$, if many lie in a $d$-dimensional subspace, then can we recover the underlying subspace? We show that Tyler's M-estimator can be used to recover the underlying subspace, if the percentage of the inliers is larger t…

A new robust PCA estimator combining M-estimators and minimum divergence estimators.

The stochastic multi-armed bandit problem is well understood when the reward distributions are sub-Gaussian. In this paper we examine the bandit problem under the weaker assumption that the distributions have moments of order 1+ε, for some $ε\in (0,1]$. Surprisingly, moments of order 2 (i.e., finite variance) are suffi…

Study examines influence diagnostics in high-dimensional M-estimation.

Optimizes private statistics with noisy methods.

A large dimensional characterization of robust M-estimators of covariance (or scatter) is provided under the assumption that the dataset comprises independent (essentially Gaussian) legitimate samples as well as arbitrary deterministic samples, referred to as outliers. Building upon recent random matrix advances in the…

New method estimates and completes tensors from ordinal data, improving accuracy and efficiency.

Tyler's M-estimator's phase transition at DS-SNR = 1 is resolved.

While optimizing convex objective (loss) functions has been a powerhouse for machine learning for at least two decades, non-convex loss functions have attracted fast growing interests recently, due to many desirable properties such as superior robustness and classification accuracy, compared with their convex counterpa…

This paper analyzes M-estimators under infinite-variance noise in high dimensions.

In this paper, we investigate the adversarial robustness of multivariate $M$-Estimators. In the considered model, after observing the whole dataset, an adversary can modify all data points with the goal of maximizing inference errors. We use adversarial influence function (AIF) to measure the asymptotic rate at which t…

This paper improves robust cluster enumeration for RES data.

Paper proposes robust estimators for heavy-tailed data with infinite variance.

Estimates error for robust M-estimators with convex penalties.

The paper analyzes the risk of bagging regularized M-estimators under proportional asymptotics.

We study theoretical properties of regularized robust M-estimators, applicable when data are drawn from a sparse high-dimensional linear model and contaminated by heavy-tailed distributions and/or outliers in the additive errors and covariates. We first establish a form of local statistical consistency for the penalize…

Study proposes an active subsampling method for estimating individualized thresholds in high-dimensional data.

New method approximates M-estimator and predictions without solving fixed-point equations.

Improved statistical inference for expensive data using machine learning predictions.

Proves equations for high-dimensional gradient-based methods from Gaussian data.

We consider the class of optimization problems arising from computationally intensive L1-regularized M-estimators, where the function or gradient values are very expensive to compute. A particular instance of interest is the L1-regularized MLE for learning Conditional Random Fields (CRFs), which are a popular class of …

Paper improves parameter estimation of continuous distributions using preference feedback.

Improved federated learning methods for privacy and efficiency.

The kernel least mean squares (KLMS) algorithm is a computationally efficient nonlinear adaptive filtering method that "kernelizes" the celebrated (linear) least mean squares algorithm. We demonstrate that the least mean squares algorithm is closely related to the Kalman filtering, and thus, the KLMS can be interpreted…

Many statistical $M$-estimators are based on convex optimization problems formed by the combination of a data-dependent loss function with a norm-based regularizer. We analyze the convergence rates of projected gradient and composite gradient methods for solving such problems, working within a high-dimensional framewor…

We consider the problem of sparsity-constrained $M$-estimation when both explanatory and response variables have heavy tails (bounded 4-th moments), or a fraction of arbitrary corruptions. We focus on the $k$-sparse, high-dimensional regime where the number of variables $d$ and the sample size $n$ are related through $…

Develop a comprehensive theory for regularized M-estimation in reproducing kernel Hilbert spaces.

The paper develops methods to handle missing data using regularized M-estimation in reproducing kernel Hilbert space.

Efficiently estimates shrinkage coefficient for RTME using LOOCV approximation.