This paper solves parameter estimation with ordered ℓ2 regularization using ADMM.
problem Scaling up ordered ℓ2 regularization for large-scale data. method Alternating Direction Method of Multipliers (ADMM) for ordered ℓ2 regularization. result ADMM-Oℓ2 outperforms or matches state-of-the-art methods in parameter estimation. In a recent work (arXiv:0910.2517), for nonlinear models with sparse underlying linear structures, we studied the error bounds of ℓ0-regularized estimation. In this note, we show that ℓ1-regularized estimation in some important cases can achieve the same order of error bounds as those in the aforementioned …
In this paper, we propose ℓp-norm regularized models to seek near-optimal sparse portfolios. These sparse solutions reduce the complexity of portfolio implementation and management. Theoretical results are established to guarantee the sparsity of the second-order KKT points of the ℓp-norm regularized models…
The paper analyzes ℓ1-LinR for Ising model selection using statistical mechanics.
problem Model selection consistency of ℓ1-LinR for Ising models. method Replica method from statistical mechanics, ℓ1-regularized linear regression (ℓ1-LinR). result Model selection consistency with sample complexity $M=\mathcal{O}\left(\log N
ight)$.
Efficiently performs robust and sparse kernel regression.
problem Robust and sparse kernel regression.
method Sign gradient descent and early stopping.
result Sign gradient descent achieves robust and sparse kernel regression efficiently.
New algorithms solve complex minimax problems efficiently.
problem Nonconvex-strongly concave minimax problems in machine learning.
method Gradient norm regularized trust-region (GRTR) and Levenberg-Marquardt (LMNegCur) algorithms.
result Proved iteration complexities matching best known results.
SOAR improves deep networks' robustness against adversarial examples.
problem Improving deep neural networks' robustness against adversarial examples.
method Formulated adversarial robustness problem under robust optimization framework, approximated loss function using second-order Taylor series expansion.
result SOAR significantly improves robustness of networks against adversarial perturbations.
New MIP framework solves high-dimensional ℓ0ℓ2-regularized regression problems.
problem Exact computation of ℓ0ℓ2-regularized regression estimators is challenging for large p. method Specialized nonlinear branch-and-bound (BnB) framework with first-order optimization.
result Achieves speedups of at least 5000x compared to state-of-the-art exact methods.
New neural network training method uses bi-fidelity data to reduce errors.
problem Training neural networks with limited high-fidelity data.
method Bi-fidelity ℓ1-regularization strategies. result Bi-fidelity ℓ1-regularization reduces errors by one order of magnitude. Regularized linear regression improves binary classification performance, especially with ridge and ℓ1 regularization.
problem Improving binary classification accuracy with noisy labels.
method Systematic study of regularization strengths on linear classifiers trained on noisy binary classification data.
result Ridge regression consistently improves classification error, while ℓ1 regularization can induce sparsity and ℓ∞ regularization can concentrate weights to two values. We analyze the effect of quantizing weights and activations of neural networks on their loss and derive a simple regularization scheme that improves robustness against post-training quantization. By training quantization-ready networks, our approach enables storing a single set of weights that can be quantized on-deman…
We investigate the learning rate of multiple kernel learning (MKL) with ℓ1 and elastic-net regularizations. The elastic-net regularization is a composition of an ℓ1-regularizer for inducing the sparsity and an ℓ2-regularizer for controlling the smoothness. We focus on a sparse setting where the total …
New algorithmic view of ℓ2 regularization using ODEs and path-following methods.
problem Optimizing convex loss functions with ℓ2 regularization.
method Established an equivalence between ℓ2-regularized solution paths and ODEs, proposing path-following algorithms based on homotopy methods and numerical ODE solvers.
result The solution path can be viewed as a hybrid of gradient descent and Newton method, providing novel schemes to choose grid points and reducing computational cost.
Advances robust principal component analysis with transformed ℓ1 regularization.
problem Recovering low-rank structures from noisy, partially observed data corrupted by sparse outliers.
method Proposes transformed ℓ1 (TL1) regularization to improve approximations of rank and ℓ0 functional.
result Achieves higher accuracy in estimating low-rank and sparse components compared to classical convex models, especially under non-uniform sampling schemes.
State-of-the-art subspace clustering methods are based on expressing each data point as a linear combination of other data points while regularizing the matrix of coefficients with ℓ1, ℓ2 or nuclear norms. ℓ1 regularization is guaranteed to give a subspace-preserving affinity (i.e., there are no conne…
Develops minibatch stochastic proximal gradient for large-scale learning models.
problem Finding optimal predictors with complex regularizers in large-scale learning models.
method Minibatch variants of stochastic proximal gradient algorithm for composite objective functions.
result Minibatch size N after O(Nε1) iterations achieves ε−suboptimality in expected quadratic distance. New algorithms optimize convex functions with high-order derivatives.
problem Optimizing convex functions with high-order derivatives under various norms.
method Developed a non-Euclidean inexact accelerated proximal point method using an inexact uniformly convex regularizer.
result Showed nearly optimal algorithms for high dimensions in the black-box oracle model for ℓp-settings and all q≥1. Safe screening rules reduce computation time in logistic regression with ℓ0−ℓ2 regularization.
problem Efficiently solving logistic regression with many features and regularization.
method Screening rules based on Fenchel dual lower bounds of strong conic relaxations.
result A high percentage of features can be safely removed before solving, leading to substantial speed-up.
Statistical analysis of regularization in continual learning tasks.
problem Understanding how regularization affects model performance in sequential learning.
method Derivation of convergence rates, iterative update formula, and optimal hyperparameters for generalized ℓ2-regularization.
result Optimal hyperparameters balance forward and backward knowledge transfer, improving model performance.
Many state-of-the-art machine learning models such as deep neural networks have recently shown to be vulnerable to adversarial perturbations, especially in classification tasks. Motivated by adversarial machine learning, in this paper we investigate the robustness of sparse regression models with strongly correlated co…
A new algorithm speeds up EEG source localization using ℓ1 regularization.
problem Challenging inverse problem in mapping EEG readings to brain activity.
method Formulated as a graphical generalized elastic net inverse problem, solved with a variable projected algorithm (VPAL).
result VPAL provides faster and more accurate EEG source localization compared to existing methods.
The paper analyzes ℓq optimization methods for high-dimensional linear regression.
problem Estimating sparse parameters from noisy observations in high-dimensional settings.
method Introduces and analyzes ℓq optimization methods for sparse estimation. result Shows stable recovery properties and bounds for ℓq minimization and regularization methods. Study on sections of line bundles vanishing along subvarieties in complex spaces.
problem Conditions for the dimension of sections vanishing along subvarieties.
method Analyzes necessary and sufficient conditions for the dimension of sections vanishing along subvarieties.
result Find necessary and sufficient conditions for dimH00(X,Lp)∼pn. We provide novel theoretical results regarding local optima of regularized M-estimators, allowing for nonconvexity in both loss and penalty functions. Under restricted strong convexity on the loss and suitable regularity conditions on the penalty, we prove that \emph{any stationary point} of the composite objective f…
A neural network solves logistic regression with ℓ1 regularization efficiently.
problem Efficiently solving logistic regression with ℓ1 regularization due to non-differentiability of ℓ1 norm. method A simple projection neural network that avoids auxiliary variables and smooth approximations.
result The neural network converges to a solution of the problem with any initial value and outperforms existing methods.
Iteratively reweighted ℓ1 algorithm is a popular algorithm for solving a large class of optimization problems whose objective is the sum of a Lipschitz differentiable loss function and a possibly nonconvex sparsity inducing regularizer. In this paper, motivated by the success of extrapolation techniques in accele…
Paper optimizes ES estimation under an ℓ1 constraint, reducing estimation errors.
problem High instability and infeasibility of ES estimation above a critical ratio r=N/T. method Analytical approach using the method of replicas from statistical physics.
result Regularization with ℓ1 constraint renormalizes the aspect ratio r=N/T. The paper establishes prediction bounds for trend filtering with higher order total variation penalties.
problem Estimating signals with jumps of varying orders using total variation regularization.
method Combining oracle inequalities and interpolating vectors to bound effective sparsity.
result The ℓ1-penalty on (k−1)extth order differences allows adaptive estimation for k∈{1,2,3,4}. New solver SR2 tackles deep neural network training with nonsmooth regularization.
problem Training deep neural networks with nonsmooth regularization to achieve sparsity and efficiency.
method Combines adaptive quadratic regularization with proximal stochastic gradient principles.
result Established worst-case iteration complexity of O(ε^−2) for SR2.
In this paper we consider the problem of grouped variable selection in high-dimensional regression using ℓ1−ℓq regularization (1≤q≤∞), which can be viewed as a natural generalization of the ℓ1−ℓ2 regularization (the group Lasso). The key condition is that the dimensionality pn can…
NA0CT2 improves tensor regression predictions with ℓ0 regularization.
problem Improving tensor regression predictions with structural information.
method Noise-Augmented ℓ0 regularization on Tucker decomposition. result Achieves exact ℓ0 regularization on core tensor in linear and generalized linear tensor regression. In many applications, high-dimensional data points can be well represented by low-dimensional subspaces. To identify the subspaces, it is important to capture a global and local structure of the data which is achieved by imposing low-rank and sparseness constraints on the data representation matrix. In low-rank sparse …
Study examines stability of image-reconstruction algorithms using variational regularization.
problem Stability and robustness of image-reconstruction algorithms in medical imaging.
method Review and novel stability results for ℓp-regularized linear inverse problems, focusing on p∈(1,∞). result Guarantees Lipschitz continuity for small p and Hölder continuity for larger p in Lp(Ω) function spaces. The paper examines how to protect LASSO-based feature selection from adversarial attacks.
problem Adversarial attacks on LASSO-based feature selection.
method Formulated as a bi-level optimization problem, reformulated LASSO with linear inequality constraints, solved using interior-point method, and modified using projected gradient descent.
result Demonstrated the effectiveness of the proposed method in protecting LASSO-based feature selection from adversarial attacks.
New harmonic functions show nodal sets can be topologically complex despite frequency and regularity constraints.
problem Understanding the topology of nodal sets of harmonic functions with bounded frequency and regularity.
method Constructing harmonic functions on the unit ball with specific properties.
result The Betti numbers of the nodal set can be arbitrarily large, contradicting previous topological bounds.
Paper tackles low-rank matrix recovery with column ℓ2,0-norm regularization.
problem Low-rank matrix recovery problems with column sparsity constraints.
method Developed alternating majorization-minimization (AMM) methods with extrapolation and hybrid AMM.
result Global convergence analysis and superior performance in matrix completion problems.
Sharp bounds for approximating Sobolev functions by ridge functions and networks.
problem Approximating Sobolev functions with multivariate ridge functions and networks.
method Proving sharp upper and lower bounds for approximation order.
result Order of approximation asymptotically behaves as n−r/(d−ℓ). Proposes a method to adapt models in nonstationary environments using ℓ1 regularization.
problem Adapting models to nonstationary environments in machine learning.
method Integrates ℓ1 regularization of differences between source and target parameters.
result Effective balance of stability and plasticity in model adaptation.
We study the robustness properties of ℓ1 norm minimization for the classical linear regression problem with a given design matrix and contamination restricted to the dependent variable. We perform a fine error analysis of the ℓ1 estimator for measurements errors consisting of outliers coupled with noise. We…
This study connects Jacobian regularization to adversarial robustness and improves generalization.
problem Adversarial attacks make deep neural networks vulnerable.
method Developed a connection between Jacobian regularization and adversarial training, and established robust generalization gaps.
result Jacobian norms are related to both standard and robust generalization.
Compressed Sensing using ℓ1 regularization is among the most powerful and popular sparsification technique in many applications, but why has it not been used to obtain sparse deep learning model such as convolutional neural network (CNN)? This paper is aimed to provide an answer to this question and to show how t…
AdamW optimizes a constrained loss with ℓ∞ norm constraint.
problem Understanding the optimization behavior of AdamW with ℓ∞ norm constraint. method Analyzing AdamW as a smoothed version of SignGD and connecting it to Frank-Wolfe optimization.
result AdamW implicitly performs constrained optimization with ℓ∞ norm constraint. We consider the problem of estimating the parameters of a linear univariate autoregressive model with sub-Gaussian innovations from a limited sequence of consecutive observations. Assuming that the parameters are compressible, we analyze the performance of the ℓ1-regularized least squares as well as a greedy esti…
We investigate the learning rate of multiple kernel leaning (MKL) with elastic-net regularization, which consists of an ℓ1-regularizer for inducing the sparsity and an ℓ2-regularizer for controlling the smoothness. We focus on a sparse setting where the total number of kernels is large but the number of non…
The optimization of the variance supplemented by a budget constraint and an asymmetric ℓ1 regularizer is carried out analytically by the replica method borrowed from the theory of disordered systems. The asymmetric regularizer allows us to penalize short and long positions differently, so the present treatment in…
The paper improves ALO for ℓ1-regularized models.
problem Estimating out-of-sample error for ℓ1-regularized models. method Developed a novel theory for ℓ1-regularized problems, bounding ALO error. result For ℓ1-regularized problems, ALO error goes to zero as p goes to infinity. Improved non-smooth optimization methods achieve faster convergence rates.
problem Non-smooth optimization problems, especially in ℓ∞ and ℓ1-SVM. method Higher-order accelerated methods, leveraging recent advances in smooth convex optimization.
result Achieved O(ε−4/5) iteration complexity for ℓ∞ regression, breaking previous barriers. The paper analyzes the trade-off between smoothness and sparsity in GCN using lp-regularized learning.
problem Quantifying the trade-off between smoothness and sparsity in GCN.
method Proposes a novel SGD proximal algorithm for GCNs with an inexact operator to analyze the stability of the ℓp-regularized stochastic learning. result Establishes an explicit theoretical understanding of GCN with ℓp-regularized stochastic learning.