Spectral norm regularization improves deep learning models' generalizability.
problem High sensitivity to input perturbation degrades deep learning model performance.
method Spectral norm regularization, penalizing high spectral norm of weight matrices.
result Models trained with spectral norm regularization show better generalizability.
Efficiently regularizes deep learning models using Jacobian nuclear norm.
problem Regularizing deep learning models to prevent overfitting and improve generalization.
method Proposes a denoising-style approximation to penalize the Jacobian nuclear norm without computing the Jacobian matrix.
result Demonstrates that penalizing the average squared Frobenius norm of Jg and Jh is equivalent to penalizing the Jacobian nuclear norm for function compositions. New proof shows norms can't explain deep learning's implicit regularization.
problem Understanding the implicit regularization in deep learning.
method Mathematical proof on matrix factorization problems.
result Implicit regularization drives norms towards infinity, suggesting rank minimization is key.
Exact spectral norm regularization improves neural network generalization.
problem Improving neural network generalization while protecting against noise.
method Exact spectral norm regularization of the Jacobian.
result Improved generalization performance compared to previous methods.
Adversarial training linked to operator norm regularization, proving network sensitivity to attacks.
problem Robustifying neural networks against adversarial attacks.
method Theoretical link established between adversarial training and operator norm regularization.
result Adversarial training is equivalent to data-dependent operator norm regularization.
Study shows how networks converge to minimum norm solutions with regularization.
problem Interpolating between known regions in shallow ReLU networks.
method Investigates empirical risk minimizers and weight decay regularizers.
result Empirical risk minimizers converge to minimum norm interpolants under specific conditions.
New norms derived from box-norm improve multitask learning performance.
problem Improving multitask learning performance in matrix completion and prediction.
method Derived new norms (box-norm, spectral k-support, spectral box-norm) and improved algorithms to compute them.
result New norms provide state-of-the-art performance in matrix completion and multitask learning.
Alpha-norm regularization simplifies marketing demand forecasting.
problem Ultra high-dimensional problems in demand estimation and forecasting.
method Nonconvex alpha-norm objective with coordinate descent and proximal operators.
result Alpha-norm regularization provides accurate out-of-sample estimates for promotion effects.
A new perspective on deep neural network regularization using RKHS norms.
problem Improving deep neural network performance and robustness.
method Using the norm of a reproducing kernel Hilbert space (RKHS) for regularization, with practical approximations.
result Effective regularization strategies for deep neural networks, including new penalties and hybrid approaches.
New nonconvex regularizers improve low-rank matrix recovery efficiency and accuracy.
problem Efficiently recover low-rank matrices from incomplete data.
method Factor group-sparse regularization, related to Schatten-p norms.
result Improved generalization error bounds for Schatten-p norms as p decreases.
The k-support norm is a regularizer which has been successfully applied to sparse vector prediction problems. We show that it belongs to a general class of norms which can be formulated as a parameterized infimum over quadratics. We further extend the k-support norm to matrices, and we observe that it is a special …
This work improves trace norm regularization for multi-task learning with limited data.
problem Learning from few samples across multiple tasks.
method Trace norm regularization for a linear shared representation model.
result First estimation error bound for trace norm regularized estimator with scarce data.
New method for regularizing deep networks using function norms.
problem Difficulty in measuring function complexity in deep networks.
method Sampling-based approximations to weighted function norms.
result Improved performance on classification and image segmentation tasks.
Enhances KLR for indefinite kernels with L1-norm regularization.
problem Classifying with indefinite kernels captures more domain-specific information.
method Introduces L1-norm regularization to induce sparsity and a proximal linearized algorithm. result Superior performance in accuracy and sparsity on multiple datasets.
Characterizes dropout's regularizer in deep linear networks.
problem Understanding dropout's regularization effect in deep learning.
method Formal characterization of dropout's regularizer, showing it is composed of an ℓ2-path regularizer and the squared nuclear norm. result For large dropout rates, the global optima of the dropout objective can be characterized.
Proposes a new sparse graph-regularized SVD for biclustering.
problem Learning blocking structure in high-dimensional data.
method Imposes graph-regularized norm and L0-norm penalties on singular vectors.
result Efficiently captures natural blocking structures in real and simulated data.
Deep networks with path norm regularization can approximate analytic functions.
problem Approximating analytic functions with neural networks.
method Path norm regularized deep networks with activation function.
result Deep networks can approximate analytic functions with logarithmic dependence on approximation error.
Characterizes inductive bias in multi-channel linear CNNs with bounded weight norm.
problem Understanding the inductive bias in multi-channel linear convolutional networks.
method Function space characterization and empirical testing of gradient descent.
result The inductive bias depends on the number of output channels for multi-channel inputs but not for single-channel inputs.
Gradient descent on matrix factorization leads to minimum nuclear norm solution.
problem Optimizing underdetermined quadratic objectives over matrices.
method Gradient descent on a full dimensional factorization of the matrix.
result Gradient descent converges to the minimum nuclear norm solution.
The paper analyzes error bounds and KL properties for noisy matrix recovery problems.
problem Noisy low-rank matrix recovery problems.
method Squared F-norm regularization, accelerated alternating minimization method.
result Established error bounds and KL properties for critical points and global minimizers.
Paper tackles low-rank matrix recovery with column ℓ2,0-norm regularization.
problem Low-rank matrix recovery problems with column sparsity constraints.
method Developed alternating majorization-minimization (AMM) methods with extrapolation and hybrid AMM.
result Global convergence analysis and superior performance in matrix completion problems.
New insights into network generalization show learning rate affects both norm and sharpness.
problem Understanding the generalization of overparameterized networks.
method Empirical analysis and theoretical proof of the trade-off between norm and sharpness.
result Learning rate influences both norm and sharpness, neither alone minimizes generalization error.
IRKSN algorithm achieves sparse recovery with wider applicability conditions.
problem Sparse recovery challenges due to NP-hard nature and restrictive conditions.
method IRKSN algorithm based on k-support norm regularizer. result Achieves sparse recovery with explicit constants and standard linear rate.
Adam's hyperparameters implicitly regularize solutions, penalizing or impeding loss gradients' norms.
problem Implicit regularization in Adam's hyperparameters and training stage.
method Backward error analysis and ODE approximations to study Adam's behavior.
result Adam's implicit regularization depends on hyperparameters and training stage, involving different norms.
Proposes a filtering method for cluster analysis using ℓ0-norm regularization.
problem Improving cluster analysis by filtering data.
method Minimizes a least squares function with a weighted ℓ0-norm penalty, approximated by smooth non-convex functions. result The proposed method can enhance existing clustering techniques.
The study analyzes robustness of estimators in linear models with adversarial errors.
problem Analyzing robustness of estimators in linear models with adversarial errors.
method Develops a general theory for minimum norm interpolating estimators and RERM in linear models without conditions on errors.
result Quantitative bound for the prediction error relating it to Rademacher complexity, norm of minimum norm interpolator of errors, and subdifferential size.
Spectral regularization simplifies sequence models by focusing on grammatical simplicity.
problem Sequence modeling challenges in learning tasks.
method Introduces spectral regularization based on Hankel matrices and trace norm, addressing bi-infinite matrices with an unbiased estimator.
result Demonstrates spectral regularization's potential benefits on Tomita grammars.
PSiLON Net uses L1 weight normalization and 1-path-norm regularization for efficient learning and sparsity.
problem Efficient learning and sparsity in neural networks with limited data.
method PSiLON Net employs L1 weight normalization and 1-path-norm regularization to simplify the 1-path-norm and achieve efficient learning and near-sparse parameters. result PSiLON Net achieves reliable optimization and strong performance in the small data regime.
Unified framework for rare feature selection and aggregation in high-dimensional statistics.
problem Challenges in high-dimensional statistics due to rare features.
method Developed a unified computational framework for norms promoting discrete structures, using orthogonal projection oracle.
result Proposed estimation procedure for automatic feature selection and aggregation with statistical bounds.
Paper proposes a new matrix recovery method relaxing uniform sampling assumptions.
problem Matrix completion under arbitrary sampling schemes.
method Max-norm and nuclear-norm regularization, alternating direction method of multipliers.
result The proposed method achieves fast rates of convergence and is computationally efficient.
A new screening rule 'dynamic Sasvi' improves sparse optimization speed.
problem Sparse optimization problem identification.
method Flexible framework based on Fenchel-Rockafellar duality for norm-regularized least squares.
result Dynamic Sasvi can eliminate more features and increase solver speed.
Noise injection before gradient steps helps in regularization for neural networks.
problem Improving generalization in overparametrized neural networks.
method Injecting small noise perturbations before computing gradient steps, especially in layer-wise fashion.
result Small noise perturbations can explicitly regularize neural networks without variance explosion.
Proposes a new SVM model for binary classification with theoretical and practical advantages.
problem Binary classification in supervised learning.
method Quadratic surface support vector machine with L1 norm regularization.
result The model can detect true sparsity patterns and is efficient for both synthetic and real data.
New framework assesses regularization norms in ill-posed problems, revealing L2 instability and proposing adaptive fractional RKHS solutions.
problem Comparative analysis of regularization norms in ill-posed problems.
method Small noise analysis framework for Tikhonov and RKHS regularizations.
result Optimal convergence rates achieved with adaptive fractional RKHS, but hyper-parameters decay too fast.
Method introduces topological regularization using information filtering networks.
problem Sparse probabilistic modeling and multicollinear regression.
method Topological regularization via information filtering network.
result Direct application to L0-norm regularized problems. New capacity measure for ReLU networks reduces generalization error.
problem Improper influence of path norm on ReLU network capacity.
method Basis-path Norm, optimization algorithms.
result Basis-path Norm better explains ReLU network generalization.
This work analyzes how to choose regularization norms for adversarial training in high dimensions.
problem Choosing the right regularization norm for adversarial training in high-dimensional settings.
method Derives asymptotic descriptions and uniform convergence bounds for robust, regularized empirical risk minimizers.
result Characterizes the relationship between perturbation size and optimal regularization choice.
Using the ℓ1-norm to regularize the estimation of the parameter vector of a linear model leads to an unstable estimator when covariates are highly correlated. In this paper, we introduce a new penalty function which takes into account the correlation of the design matrix to stabilize the estimation. This norm, ca…
Optimal estimates derived for residual networks' generalization error.
problem Estimating the generalization error of residual networks.
method Derives optimal a priori estimates using a weighted path norm.
result Optimal error estimates are comparable to Monte Carlo error rates.
New method uses function norm for deep network regularization, improving performance on small datasets.
problem Deep networks overfit small datasets.
method Directly uses L2 function norm for regularization in stochastic backpropagation. result New methods outperform state-of-the-art methods on benchmark datasets.
Regularization can induce grokking in neural networks, improving generalization.
problem Delayed generalization following overfitting in neural networks.
method Demonstrates that gradient descent with small regularization of model properties induces grokking.
result Regularization can induce grokking, extending previous work on weight decay.
SAM improves generalization in overparameterized models, but its behavior in tensorized models is less understood.
problem Understanding the implicit regularization of SAM in tensorized models.
method Scale-invariance analysis and gradient flow analysis to derive Norm Deviation as a measure of core norm imbalance, and propose Deviation-Aware Scaling (DAS).
result DAS achieves competitive or improved performance over SAM, while offering reduced computational overhead.
Sparse learning has recently received increasing attention in many areas including machine learning, statistics, and applied mathematics. The mixed-norm regularization based on the l1q norm with q>1 is attractive in many applications of regression and classification in that it facilitates group sparsity in the model. T…
In this paper, we propose ℓp-norm regularized models to seek near-optimal sparse portfolios. These sparse solutions reduce the complexity of portfolio implementation and management. Theoretical results are established to guarantee the sparsity of the second-order KKT points of the ℓp-norm regularized models…
The study solves the isoperimetric problem for Heisenberg group norms.
problem Solving the isoperimetric problem for anisotropic norms in the Heisenberg group.
method Representation formula for perimeter, foliation property, differential equation characterization, approximation procedure.
result Characterization of isoperimetric sets as sub-Finsler analogues of Pansu's bubbles.
Paper introduces new norms for rank-constrained optimization problems.
problem Rank-constrained optimization problems in various fields.
method Introduces a family of low-rank inducing norms and regularizers.
result Other low-rank inducing norms outperform nuclear norm in matrix completion problems.
Two new regularization methods improve neural network performance and complexity control.
problem Improving neural network performance and complexity control with correlated or high-dimensional features.
method Two regularization strategies: covariance-aware ridge and covariance-aware lasso.
result Improves predictive performance and complexity control over standard penalties.
The paper tackles system identification via Hankel nuclear norm regularization, improving estimation rates and singular value gaps.
problem Identifying low-order linear systems from limited data.
method Hankel nuclear norm regularization to encourage low-rankness of the Hankel matrix.
result Hankel regularization enables optimal system recovery with fewer observations and better estimation rates.