Improved algorithms solve ℓp-norm regression problems efficiently.
problem Efficiently solving ℓp-norm regression problems for p∈(1,2)∪(2,∞). method Iterative refinement scheme using smoothed ℓp-norms to improve solutions. result Solves ℓp-norm regression to 1/extpoly(n) accuracy in ildeOp(m31) iterations. New algorithm solves ℓ0-norm constrained multilinear logistic regression for tensor data.
problem Non-convex and nonsmooth ℓ0-norm constraints in multilinear logistic regression. method APALM+ method for globally convergent optimization. result APALM+ ensures convergence to a first-order critical point. Paper develops algorithms for sparse linear regression with generalized elastic net penalty.
problem Sparse linear regression with robust penalty for high-dimensional data.
method Iterative Reweighted Framework based on ADMM and PMM with SNN.
result Efficient algorithms provide superior performance in both simulated and real data.
The paper studies the minimum ℓ₁-norm interpolator's risk behavior in over-parameterized settings.
problem Understanding the risk behavior of minimum ℓ₁-norm interpolators in high-dimensional settings.
method Exact characterization of the risk behavior through a system of two non-linear equations.
result Observation of a multi-descent phenomenon in the generalization risk of the minimum ℓ₁-norm interpolator.
Study on Transfer Elastic Net error bounds and grouping effect.
problem Estimation error and grouping effect in Transfer Elastic Net.
method Derives non-asymptotic error bound and examines grouping effect scenarios.
result Effective error bounds and grouping effect observed in Transfer Elastic Net.
New model leads to optimal test loss in sparse linear regression.
problem Sparse linear regression with low test loss despite interpolating training data.
method Developed a new parametrization of the model that combines benefits of ℓ1 and ℓ2 norms.
result Training via gradient descent leads to an interpolator with near-optimal test loss.
Unified analysis of parameter norms in overparameterized linear models, revealing scaling laws and thresholds.
problem Understanding the scaling of parameter norms in overparameterized linear models.
method Simple dual-ray analysis revealing competition between signal spike and bulk of null coordinates.
result Unified closed-form predictions for parameter norm scaling, including elbow and threshold laws.
We study the robustness properties of ℓ1 norm minimization for the classical linear regression problem with a given design matrix and contamination restricted to the dependent variable. We perform a fine error analysis of the ℓ1 estimator for measurements errors consisting of outliers coupled with noise. We…
A new method for multi-target regression robust to outliers.
problem Outliers in regression data affect the performance of ELM-based methods.
method Generalized Outlier Robust ELM (GOR-ELM) using ℓ2,1 norm and Elastic Net theory. result GOR-ELM outperforms other ELM-based techniques in datasets with outliers.
New method uses nuclear and ℓ1 penalties for matrix regression, improving brain disorder detection.
problem Modeling high-dimensional matrix predictors with binary responses.
method Convex optimization with ADMM for low-rank and sparse structures.
result Effective in identifying brain disorder-related connectivity patterns.
Optimal sketching bounds for sparse linear regression under various loss functions are established.
problem Sparse linear regression under different loss functions.
method Distribution over oblivious sketches for sparse ℓ2 norm regression and hinge-like loss functions. result Optimal sketching bounds with O(klog(d/k)/ε2) rows for sparse ℓ2 norm regression and O(μ2klog(μnd/ε)/ε2) rows for hinge-like loss functions. This work provides efficient algorithms for approximating ℓ_p sensitivities and related statistics.
problem Estimating the importance of datapoints in high-dimensional datasets.
method Efficient algorithms for computing α-approximation of ℓ_1 sensitivities and total sensitivity using importance sampling and sensitivity computations.
result Real-world datasets have significantly lower intrinsic effective dimensionality than theoretical predictions.
Active sampling algorithm for linear regression with various norms and improved query complexity.
problem Efficiently querying a few entries of a target vector for near optimal minimizers of linear regression.
method Lewis weight sampling and active sampling algorithms for different p norms. result Optimal query complexity for p∈(0,1), 1<p<2, and 2<p<∞. New approach to adaptively select bandwidths in nonparametric regression.
problem Adaptive bandwidth selection in nonparametric regression.
method Inspired by ℓ2-norms of interval projections, introduces a new bandwidth selection procedure. result Obtains non-asymptotic risk bounds for local polynomial regression methods that adapt to local Hölder exponent.
A neural network solves logistic regression with ℓ1 regularization efficiently.
problem Efficiently solving logistic regression with ℓ1 regularization due to non-differentiability of ℓ1 norm. method A simple projection neural network that avoids auxiliary variables and smooth approximations.
result The neural network converges to a solution of the problem with any initial value and outperforms existing methods.
New algorithm samples matrix rows proportional to their ℓ_p norm in a turnstile data stream.
problem Sampling rows of a dynamic matrix efficiently in a turnstile data stream.
method Develops a novel algorithm for sampling rows proportional to their ℓ_p norm in a turnstile data stream, returning sampled row indexes and approximated sampling probabilities.
result Achieves (1+ε) approximation for logistic regression in a turnstile data stream with polynomial sketch size. Early stopping improves logistic regression's calibration and consistency in high dimensions.
problem Improving the statistical performance of gradient descent in overparameterized logistic regression.
method Investigates the effects of early stopping on gradient descent in logistic regression.
result Early-stopped gradient descent is well-calibrated and statistically consistent, while asymptotic gradient descent is not.
Lower bounds show OLS outperforms basis pursuit in overparameterized linear regression.
problem Excess risk of sparse interpolating procedures in overparameterized linear regression.
method Proved lower bounds on excess risk for OLS and basis pursuit.
result Excess risk of basis pursuit can converge at an exponentially slower rate than OLS.
We consider the empirical risk minimization problem for linear supervised learning, with regularization by structured sparsity-inducing norms. These are defined as sums of Euclidean norms on certain subsets of variables, extending the usual ℓ1-norm and the group ℓ1-norm by allowing the subsets to overlap. T…
BP, a method for sparse recovery, shows generalization error decreases with more features.
problem Understanding the generalization error of overfitting solutions in linear regression.
method Study of Basis Pursuit (BP) for sparse recovery of linear regression models.
result BP's model error decreases with more features, showing double-descent behavior.
Study on linear regression robustness to adversarial attacks.
problem Adversarial attacks on linear regression models.
method Analysis of prediction error bounds, asymptotic results, convex optimization.
result Adversarial error can grow to infinity with more features, while test error goes to zero.
Study differentially private linear regression with heavy-tailed data.
problem Differentially private ℓ1-norm linear regression with heavy-tailed data. method Exponential mechanism for ℓ2-norm bounded second moment; relaxation to ℓ2-norm bounded θ-th moment; coordinate-wise bounded moments. result Achieved upper bounds for privacy-preserving linear regression under various moment conditions.
In presence of sparse noise we propose kernel regression for predicting output vectors which are smooth over a given graph. Sparse noise models the training outputs being corrupted either with missing samples or large perturbations. The presence of sparse noise is handled using appropriate use of ℓ1-norm along-wi…
A new PCA method using Tℓ1-norm outperforms existing methods.
problem Outliers and noise sensitivity in classical PCA.
method PCA based on Tℓ1-norm maximization. result The method outperforms PCA-ℓp, ℓpSPCA, and PCA in numerical experiments. Guarantees recovery of compressible signals from adversarial noise.
problem Recovering compressible signals from noise and adversarial attacks.
method Extends adversarial defense framework to ℓ0, ℓ2, and ℓ∞ norms. result Recovery guarantees for various signal recovery methods under different noise types.
Undirected graphs are often used to describe high dimensional distributions. Under sparsity conditions, the graph can be estimated using ℓ1-penalization methods. We propose and study the following method. We combine a multiple regression approach with ideas of thresholding and refitting: first we infer a sparse u…
In this work we propose to fit a sparse logistic regression model by a weakly convex regularized nonconvex optimization problem. The idea is based on the finding that a weakly convex function as an approximation of the ℓ0 pseudo norm is able to better induce sparsity than the commonly used ℓ1 norm. For a cl…
We develop an approximate formula for evaluating a cross-validation estimator of predictive likelihood for multinomial logistic regression regularized by an ℓ1-norm. This allows us to avoid repeated optimizations required for literally conducting cross-validation; hence, the computational time can be significantl…
This paper uses quantum computing to solve sparse linear regression problems efficiently.
problem Sparse linear regression to identify important features from a large set of variables.
method Formulates the ℓ0 optimization problem as a QUBO problem and solves it using the D-Wave adiabatic quantum computer. result The QUBO solution matches the optimal solution for a wide range of sparsity penalty values across datasets.
We characterize the effectiveness of a classical algorithm for recovering the Markov graph of a general discrete pairwise graphical model from i.i.d. samples. The algorithm is (appropriately regularized) maximum conditional log-likelihood, which involves solving a convex program for each node; for Ising models this is …
We apply the OSCAR (octagonal selection and clustering algorithms for regression) in recovering group-sparse matrices (two-dimensional---2D---arrays) from compressive measurements. We propose a 2D version of OSCAR (2OSCAR) consisting of the ℓ1 norm and the pair-wise ℓ∞ norm, which is convex but non-d…
Adam's bias shifts from full-batch to max-margin of different norms for separable data.
problem Understanding Adam's implicit bias in the incremental batch setting.
method Analyzing incremental Adam on linearly separable data, constructing datasets, and using a proxy algorithm.
result Incremental Adam can converge to different max-margin classifiers depending on the dataset and batching scheme.
Sparse linear regression -- finding an unknown vector from linear measurements -- is now known to be possible with fewer samples than variables, via methods like the LASSO. We consider the multiple sparse linear regression problem, where several related vectors -- with partially shared support sets -- have to be recove…
We analyze coresets for regularized regression problems and propose a modified lasso that yields smaller coresets.
problem Analyzing coresets for regularized regression problems.
method Examined coresets for ridge regression and proposed a modified lasso problem.
result No coreset for regularized regression can be smaller than the unregularized version when reqs. Forward stagewise regression follows a very simple strategy for constructing a sequence of sparse regression estimates: it starts with all coefficients equal to zero, and iteratively updates the coefficient (by a small amount ε) of the variable that achieves the maximal absolute inner product with the current residua…
Statistical analysis of regularization in continual learning tasks.
problem Understanding how regularization affects model performance in sequential learning.
method Derivation of convergence rates, iterative update formula, and optimal hyperparameters for generalized ℓ2-regularization.
result Optimal hyperparameters balance forward and backward knowledge transfer, improving model performance.
Transfer learning improves MNI's performance in high-dimensional linear regression.
problem Improving model performance in high-dimensional linear regression with diverse data.
method Proposes a Transfer MNI approach, analyzing its excess risk and conditions for outperformance.
result Identifies free-lunch covariate shift regimes where knowledge transfer benefits.
In recent years, stochastic gradient descent (SGD) methods and randomized linear algebra (RLA) algorithms have been applied to many large-scale problems in machine learning and data analysis. We aim to bridge the gap between these two methods in solving constrained overdetermined linear regression problems---e.g., $\el…
We describe two nonconventional algorithms for linear regression, called GAME and CLASH. The salient characteristics of these approaches is that they exploit the convex ℓ1-ball and non-convex ℓ0-sparsity constraints jointly in sparse recovery. To establish the theoretical approximation guarantees of GAME an…
Study robust linear regression with outliers, providing exact asymptotics for ERM performance.
problem Robust linear regression in high-dimension with outliers.
method Analyzes ℓ2, ℓ1, and Huber losses, providing asymptotic performance metrics. result Optimally-regularised ERM is asymptotically consistent with simple calibration, but Huber loss requires norm calibration.
In this paper, we propose ℓp-norm regularized models to seek near-optimal sparse portfolios. These sparse solutions reduce the complexity of portfolio implementation and management. Theoretical results are established to guarantee the sparsity of the second-order KKT points of the ℓp-norm regularized models…
High dimensional regression benefits from sparsity promoting regularizations. Screening rules leverage the known sparsity of the solution by ignoring some variables in the optimization, hence speeding up solvers. When the procedure is proven not to discard features wrongly the rules are said to be \emph{safe}. In this …
Many state-of-the-art machine learning models such as deep neural networks have recently shown to be vulnerable to adversarial perturbations, especially in classification tasks. Motivated by adversarial machine learning, in this paper we investigate the robustness of sparse regression models with strongly correlated co…
Paper finds a fast method for a matrix norm proximal operator.
problem Optimizing mixed ℓ1,∞ matrix norms efficiently. method Closed-form computation using soft-thresholding, iterative algorithm for thresholds.
result Mixed ℓ1,∞ prox can be computed in closed form. Signal estimation problems with smoothness and sparsity priors can be naturally modeled as quadratic optimization with ℓ0-"norm" constraints. Since such problems are non-convex and hard-to-solve, the standard approach is, instead, to tackle their convex surrogates based on ℓ1-norm relaxations. In this paper…
We study the problem of learning a sparse linear regression vector under additional conditions on the structure of its sparsity pattern. This problem is relevant in machine learning, statistics and signal processing. It is well known that a linear regression can benefit from knowledge that the underlying regression vec…
New robust estimator for high-dimensional data with outliers and leverage points.
problem Robust regression in high-dimensional datasets with gross contamination.
method Adaptive τ-Lasso estimator with an adaptive ℓ1-norm penalty.
result Adaptive τ-Lasso has the oracle property and robustness to outliers and high-leverage points.
In multivariate regression, a K-dimensional response vector is regressed upon a common set of p covariates, with a matrix B∗∈Rp×K of regression coefficients. We study the behavior of the multivariate group Lasso, in which block regularization based on the ℓ1/ℓ2 norm is used for supp…