We investigate the learning rate of multiple kernel learning (MKL) with and elastic-net regularizations. The elastic-net regularization is a composition of an -regularizer for inducing the sparsity and an -regularizer for controlling the smoothness. We focus on a sparse setting where the total …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “$\ell_1$ regularization”
State-of-the-art subspace clustering methods are based on expressing each data point as a linear combination of other data points while regularizing the matrix of coefficients with , or nuclear norms. regularization is guaranteed to give a subspace-preserving affinity (i.e., there are no conne…
The paper analyzes -LinR for Ising model selection using statistical mechanics.
problem Model selection consistency of -LinR for Ising models.
method Replica method from statistical mechanics, -regularized linear regression (-LinR).
result Model selection consistency with sample complexity $M=\mathcal{O}\left(\log N
ight)$.
Safe screening rules reduce computation time in logistic regression with regularization.
problem Efficiently solving logistic regression with many features and regularization.
method Screening rules based on Fenchel dual lower bounds of strong conic relaxations.
result A high percentage of features can be safely removed before solving, leading to substantial speed-up.
Efficiently performs robust and sparse kernel regression.
problem Robust and sparse kernel regression.
method Sign gradient descent and early stopping.
result Sign gradient descent achieves robust and sparse kernel regression efficiently.
A new algorithm speeds up EEG source localization using regularization.
problem Challenging inverse problem in mapping EEG readings to brain activity.
method Formulated as a graphical generalized elastic net inverse problem, solved with a variable projected algorithm (VPAL).
result VPAL provides faster and more accurate EEG source localization compared to existing methods.
Regularization is a popular technique in machine learning for model estimation and avoiding overfitting. Prior studies have found that modern ordered regularization can be more effective in handling highly correlated, high-dimensional data than traditional regularization. The reason stems from the fact that the ordered…
In this paper, we propose -norm regularized models to seek near-optimal sparse portfolios. These sparse solutions reduce the complexity of portfolio implementation and management. Theoretical results are established to guarantee the sparsity of the second-order KKT points of the -norm regularized models…
In this paper, we discuss the statistical properties of the optimization methods , including the minimization method and the regularization method, for estimating a sparse parameter from noisy observations in high-dimensional linear regression with either a deterministic or rando…
On -regularized estimation for nonlinear models that have sparse underlying linear structuresmath.ST
In a recent work (arXiv:0910.2517), for nonlinear models with sparse underlying linear structures, we studied the error bounds of -regularized estimation. In this note, we show that -regularized estimation in some important cases can achieve the same order of error bounds as those in the aforementioned …
A neural network solves logistic regression with regularization efficiently.
problem Efficiently solving logistic regression with regularization due to non-differentiability of norm.
method A simple projection neural network that avoids auxiliary variables and smooth approximations.
result The neural network converges to a solution of the problem with any initial value and outperforms existing methods.
New neural network training method uses bi-fidelity data to reduce errors.
problem Training neural networks with limited high-fidelity data.
method Bi-fidelity -regularization strategies.
result Bi-fidelity -regularization reduces errors by one order of magnitude.
In this paper we consider the problem of grouped variable selection in high-dimensional regression using regularization (), which can be viewed as a natural generalization of the regularization (the group Lasso). The key condition is that the dimensionality can…
NACT improves tensor regression predictions with regularization.
problem Improving tensor regression predictions with structural information.
method Noise-Augmented regularization on Tucker decomposition.
result Achieves exact regularization on core tensor in linear and generalized linear tensor regression.
Study examines stability of image-reconstruction algorithms using variational regularization.
problem Stability and robustness of image-reconstruction algorithms in medical imaging.
method Review and novel stability results for -regularized linear inverse problems, focusing on .
result Guarantees Lipschitz continuity for small and Hölder continuity for larger in function spaces.
In many applications, high-dimensional data points can be well represented by low-dimensional subspaces. To identify the subspaces, it is important to capture a global and local structure of the data which is achieved by imposing low-rank and sparseness constraints on the data representation matrix. In low-rank sparse …
New harmonic functions show nodal sets can be topologically complex despite frequency and regularity constraints.
problem Understanding the topology of nodal sets of harmonic functions with bounded frequency and regularity.
method Constructing harmonic functions on the unit ball with specific properties.
result The Betti numbers of the nodal set can be arbitrarily large, contradicting previous topological bounds.
Paper tackles low-rank matrix recovery with column -norm regularization.
problem Low-rank matrix recovery problems with column sparsity constraints.
method Developed alternating majorization-minimization (AMM) methods with extrapolation and hybrid AMM.
result Global convergence analysis and superior performance in matrix completion problems.
Regularized linear regression improves binary classification performance, especially with ridge and regularization.
problem Improving binary classification accuracy with noisy labels.
method Systematic study of regularization strengths on linear classifiers trained on noisy binary classification data.
result Ridge regression consistently improves classification error, while regularization can induce sparsity and regularization can concentrate weights to two values.
Proposes a method to adapt models in nonstationary environments using ℓ1 regularization.
problem Adapting models to nonstationary environments in machine learning.
method Integrates ℓ1 regularization of differences between source and target parameters.
result Effective balance of stability and plasticity in model adaptation.
Regularization of minimization for dealing with outliers and noise in Statistics and Signal Recoverymath.OC
We study the robustness properties of norm minimization for the classical linear regression problem with a given design matrix and contamination restricted to the dependent variable. We perform a fine error analysis of the estimator for measurements errors consisting of outliers coupled with noise. We…
Adversarially robust generalization theory via Jacobian regularization for deep neural networksstat.ML
This study connects Jacobian regularization to adversarial robustness and improves generalization.
problem Adversarial attacks make deep neural networks vulnerable.
method Developed a connection between Jacobian regularization and adversarial training, and established robust generalization gaps.
result Jacobian norms are related to both standard and robust generalization.
Compressed Sensing using regularization is among the most powerful and popular sparsification technique in many applications, but why has it not been used to obtain sparse deep learning model such as convolutional neural network (CNN)? This paper is aimed to provide an answer to this question and to show how t…
AdamW optimizes a constrained loss with norm constraint.
problem Understanding the optimization behavior of AdamW with norm constraint.
method Analyzing AdamW as a smoothed version of SignGD and connecting it to Frank-Wolfe optimization.
result AdamW implicitly performs constrained optimization with norm constraint.
We investigate the learning rate of multiple kernel leaning (MKL) with elastic-net regularization, which consists of an -regularizer for inducing the sparsity and an -regularizer for controlling the smoothness. We focus on a sparse setting where the total number of kernels is large but the number of non…
The optimization of the variance supplemented by a budget constraint and an asymmetric regularizer is carried out analytically by the replica method borrowed from the theory of disordered systems. The asymmetric regularizer allows us to penalize short and long positions differently, so the present treatment in…
Approximate Leave-one-out Cross Validation for Regression with Regularizers (extended version)math.ST
The paper improves ALO for -regularized models.
problem Estimating out-of-sample error for -regularized models.
method Developed a novel theory for -regularized problems, bounding ALO error.
result For -regularized problems, ALO error goes to zero as p goes to infinity.
The paper analyzes the trade-off between smoothness and sparsity in GCN using lp-regularized learning.
problem Quantifying the trade-off between smoothness and sparsity in GCN.
method Proposes a novel SGD proximal algorithm for GCNs with an inexact operator to analyze the stability of the -regularized stochastic learning.
result Establishes an explicit theoretical understanding of GCN with -regularized stochastic learning.
Despite its nonconvex nature, sparse approximation is desirable in many theoretical and application cases. We study the sparse approximation problem with the tool of deep learning, by proposing Deep Encoders. Two typical forms, the regularized problem and the -sparse problem, are …
Unified framework for accurate coresets in latent variable models and regularized regression.
problem Efficiently training models on large datasets.
method Unified framework for constructing accurate coresets for latent variable models and -regularized regression.
result Unified framework reduces coreset size for latent variable models and -regularized regression.
New regularization techniques improve stability of deep neural networks.
problem Improving stability of deep neural networks in high-dimensional data.
method Apply manifold regularization to develop new regularizers based on graph Laplacian sparsification.
result Empirically, networks achieve high stability in various perturbation models, including adversarial attacks.
Multi-task feature learning aims to identity the shared features among tasks to improve generalization. It has been shown that by minimizing non-convex learning models, a better solution than the convex alternatives can be obtained. Therefore, a non-convex model based on the capped- regularization wa…
Sparse Identification of Nonlinear Dynamical Systems via Reweighted -regularized Least Squaresstat.ML
Method improves SINDy for noisy nonlinear systems.
problem Recover nonlinear dynamical systems from noisy data.
method Reweighted -regularized least squares.
result Improved accuracy and robustness in noisy conditions.
Study supports recovery of PDEs from noisy data using a specific regularization method.
problem Support recovery of PDEs from a single noisy trajectory.
method Applying ℓ1-regularized Pseudo-Least Squares model to a given data set.
result Support of ℓ1-c coefficients asymptotically converges to the true signed-support of the PDE.
Two new regularization methods improve neural network performance and complexity control.
problem Improving neural network performance and complexity control with correlated or high-dimensional features.
method Two regularization strategies: covariance-aware ridge and covariance-aware lasso.
result Improves predictive performance and complexity control over standard penalties.
New model leads to optimal test loss in sparse linear regression.
problem Sparse linear regression with low test loss despite interpolating training data.
method Developed a new parametrization of the model that combines benefits of ℓ1 and ℓ2 norms.
result Training via gradient descent leads to an interpolator with near-optimal test loss.
Advances robust principal component analysis with transformed ℓ1 regularization.
problem Recovering low-rank structures from noisy, partially observed data corrupted by sparse outliers.
method Proposes transformed ℓ1 (TL1) regularization to improve approximations of rank and ℓ0 functional.
result Achieves higher accuracy in estimating low-rank and sparse components compared to classical convex models, especially under non-uniform sampling schemes.
We analyze the effect of quantizing weights and activations of neural networks on their loss and derive a simple regularization scheme that improves robustness against post-training quantization. By training quantization-ready networks, our approach enables storing a single set of weights that can be quantized on-deman…
We consider a discrete optimization formulation for learning sparse classifiers, where the outcome depends upon a linear combination of a small subset of features. Recent work has shown that mixed integer programming (MIP) can be used to solve (to optimality) -regularized regression problems at scales much larg…
Subspace clustering methods based on , or nuclear norm regularization have become very popular due to their simplicity, theoretical guarantees and empirical success. However, the choice of the regularizer can greatly impact both theory and practice. For instance, regularization is guaranteed t…
New algorithmic view of ℓ2 regularization using ODEs and path-following methods.
problem Optimizing convex loss functions with ℓ2 regularization.
method Established an equivalence between ℓ2-regularized solution paths and ODEs, proposing path-following algorithms based on homotopy methods and numerical ODE solvers.
result The solution path can be viewed as a hybrid of gradient descent and Newton method, providing novel schemes to choose grid points and reducing computational cost.
This paper proposes a method to select relevant features for multi-label learning.
problem Feature selection in multi-label learning to retain important information with minimal features.
method Random manifold sampling and joint sparse regularization to solve multicollinearity and obtain sparse feature sets.
result The proposed method outperforms other methods in selecting relevant features for multi-label learning.
FedElasticNet reduces communication costs and handles client drift in FL.
problem Expensive communication costs and client drift issues in federated learning.
method Leverages elastic net regularizers to sparsify local updates and limit client drift.
result FedElasticNet effectively resolves communication cost and client drift problems.
We analyze coresets for regularized regression problems and propose a modified lasso that yields smaller coresets.
problem Analyzing coresets for regularized regression problems.
method Examined coresets for ridge regression and proposed a modified lasso problem.
result No coreset for regularized regression can be smaller than the unregularized version when .
Recently, Mahoney and Orecchia demonstrated that popular diffusion-based procedures to compute a quick \emph{approximation} to the first nontrivial eigenvector of a data graph Laplacian \emph{exactly} solve certain regularized Semi-Definite Programs (SDPs). In this paper, we extend that result by providing a statistica…
This paper addresses how well we can recover a data matrix when only given a few of its elements. We present a randomized algorithm that element-wise sparsifies the data, retaining only a few its elements. Our new algorithm independently samples the data using sampling probabilities that depend on both the squares ($\e…
In this paper, we discuss how a suitable family of tensor kernels can be used to efficiently solve nonparametric extensions of regularized learning methods. Our main contribution is proposing a fast dual algorithm, and showing that it allows to solve the problem efficiently. Our results contrast recent finding…
In this paper, we made an extension to the convergence analysis of the dynamics of two-layered bias-free networks with one output. We took into consideration two popular regularization terms: the and norm of the parameter vector , and added it to the square loss function with coefficient $λ/…