Spectral norm regularization improves deep learning models' generalizability.
problem High sensitivity to input perturbation degrades deep learning model performance.
method Spectral norm regularization, penalizing high spectral norm of weight matrices.
result Models trained with spectral norm regularization show better generalizability.
We introduce a new family of matrix norms, the "local max" norms, generalizing existing methods such as the max norm, the trace norm (nuclear norm), and the weighted or smoothed weighted trace norms, which have been extensively used in the literature as regularizers for matrix reconstruction problems. We show that this…
Study improves image classifier robustness to random p-norm corruptions.
problem Improving robustness of image classifiers to real-world imperceptible corruptions.
method Training and testing with random p-norm corruptions, evaluating robustness against different p-norms.
result Training with a combination of p-norm corruptions significantly improves robustness.
ILM-Norm normalizes instances individually for better performance.
problem Learning to normalize parameters for improved model performance.
method ILM-Norm learns normalization parameters via feature feed-forward and gradient back-propagation.
result ILM-Norm consistently improves model performance across different architectures and tasks.
Improved LDA with capped l_{2,1}-norm reduces outlier sensitivity.
problem Outliers and noise sensitivity in classical LDA.
method Introducing capped l_{2,1}-norm and proposing CLDA.
result CLDA effectively removes outliers and suppresses noise.
Exact spectral norm regularization improves neural network generalization.
problem Improving neural network generalization while protecting against noise.
method Exact spectral norm regularization of the Jacobian.
result Improved generalization performance compared to previous methods.
SAM improves generalization in overparameterized models, but its behavior in tensorized models is less understood.
problem Understanding the implicit regularization of SAM in tensorized models.
method Scale-invariance analysis and gradient flow analysis to derive Norm Deviation as a measure of core norm imbalance, and propose Deviation-Aware Scaling (DAS).
result DAS achieves competitive or improved performance over SAM, while offering reduced computational overhead.
Improved bounds for discrete probability distribution estimation under the ℓ∞ norm.
problem Estimating discrete probability distributions under the ℓ∞ norm with improved bounds.
method Minimax bounds in expectation and high-probability tail bounds.
result Resolved open questions posed in Kontorovich and Painsky (JMLR, 2025), including a fully empirical tightest risk bound and identifying the worst-case extremal distribution.
Norm-range partition improves MIPS search efficiency by reducing query complexity.
problem Efficiently searching for maximum inner product in large datasets.
method Norm-range partition technique that divides datasets into sub-datasets with similar norms and builds independent hash indexes.
result Significantly reduces the number of probed buckets for LSH-based MIPS algorithms.
Norm-ranging LSH improves MIPS performance by addressing 2-norm distribution issues.
problem Long tails in 2-norm distribution of real datasets affect Simple-LSH performance.
method Norm-ranging LSH partitions datasets into sub-datasets and builds independent hash indexes.
result Norm-ranging LSH achieves an order of magnitude speedup over Simple-LSH.
The paper improves tensor completion by introducing incoherent tensor norms.
problem Higher order tensor completion with varying sample sizes.
method Introducing incoherent tensor norms to improve nuclear norm minimization.
result Higher order tensors can be recovered with fewer samples than traditional methods.
The k-support norm is a regularizer which has been successfully applied to sparse vector prediction problems. We show that it belongs to a general class of norms which can be formulated as a parameterized infimum over quadratics. We further extend the k-support norm to matrices, and we observe that it is a special …
Study improves bounds on p-covectors and proves stable systolic inequalities.
problem Improving bounds on p-covectors and proving stable systolic inequalities.
method Analyzing Euclidean norms and fundamental cohomology classes.
result Improved upper bounds and stable systolic inequalities proved.
We study a regularizer which is defined as a parameterized infimum of quadratics, and which we call the box-norm. We show that the k-support norm, a regularizer proposed by [Argyriou et al, 2012] for sparse vector prediction problems, belongs to this family, and the box-norm can be generated as a perturbation of the fo…
The study finds conditions for improved Lp norms of eigenfunctions on compact manifolds.
problem Finding necessary and sufficient conditions for improved Lp norms of eigenfunctions on compact Riemannian manifolds. method Analyzes eigenfunctions on compact Riemannian manifolds and uses properties of half-wave operators to determine conditions for improved norms.
result Conditions for improved Lpc(M) norms are necessary and sufficient for improved norms of eigenfunctions. FedNNNN improves FL by adjusting model update vector norms.
problem Slow convergence and low prediction accuracy in FL.
method Norm-Normalized Neural Network Aggregation (FedNNNN) with momentum control.
result Up to 5.4% accuracy improvement on multiple datasets.
The spectral k-support norm enjoys good estimation properties in low rank matrix learning problems, empirically outperforming the trace norm. Its unit ball is the convex hull of rank k matrices with unit Frobenius norm. In this paper we generalize the norm to the spectral (k,p)-support norm, whose additional para…
Estimates for geodesics on hyperbolic tori improve previous bounds.
problem Counting simple closed geodesics on hyperbolic tori.
method McShane-Rivin norm balls and Markoff numbers.
result The number of simple closed geodesics of length exactly L≥2 is at most CX(logL)2. Convex norms improve coupled matrix and tensor completion.
problem Efficiently complete coupled matrices and tensors with shared information.
method Proposed convex norms and completion algorithm.
result Excess risk bounds show improved performance compared to uncoupled norms.
Improved algorithms solve ℓp-norm regression problems efficiently.
problem Efficiently solving ℓp-norm regression problems for p∈(1,2)∪(2,∞). method Iterative refinement scheme using smoothed ℓp-norms to improve solutions. result Solves ℓp-norm regression to 1/extpoly(n) accuracy in ildeOp(m31) iterations. New pivoting strategy improves trace norm contraction in low-rank approximation.
problem Finding good low-rank approximations of symmetric, positive-definite matrices.
method Choosing rows with likelihood proportional to Aii2 for randomly pivoted partial Cholesky algorithm. result Same trace norm contraction result in Frobenius norm for improved pivoting strategy.
CNN layers with large norms are still robust to adversarial attacks.
problem Understanding the relationship between layer norms and adversarial robustness in CNNs.
method Theoretical analysis of ℓ1 and ℓ∞ norms, norm decay method, adversarial training frameworks. result Adversarially robust CNNs can have comparable or larger layer norms than non-adversarially robust ones.
New bounds improve deep learning performance efficiently.
problem Improving generalization and robustness of deep learning models.
method Deriving four provable upper bounds on spectral norm of convolution layers, differentiable and efficient.
result Minimum of four bounds is a tight, differentiable and efficient upper bound on spectral norm.
This study reveals the critical role of scale vectors in large language models, improving optimization and expressivity.
problem Understanding and optimizing the scale vectors in large language models.
method Systematic study of scale vectors from expressivity, optimization, and architectural perspectives; theoretical and empirical analysis of weight decay; proposing and evaluating improvements.
result Scale vectors improve optimization through a self-amplifying preconditioning effect and are beneficial for expressivity in certain architectures.
Improves logistic regression performance by reducing dependence on predictor norm.
problem Improper learning in logistic regression with exponential dependence on predictor norm.
method Designing an efficient improper learning algorithm for online logistic regression with doubly-exponential improvement in predictor norm dependence.
result Improves regret bound for online logistic regression with doubly-exponential improvement in dependence on predictor norm.
New method for regularizing deep networks using function norms.
problem Difficulty in measuring function complexity in deep networks.
method Sampling-based approximations to weighted function norms.
result Improved performance on classification and image segmentation tasks.
Improved online PCA algorithm learns from evolving norm of parameter vector.
problem Discarding evolving norm in online PCA leads to suboptimal learning.
method Implicitly Normalized Online PCA (INO-PCA) removes unit-norm constraint.
result Parameter norm evolution leads to improved learning behavior.
A toolkit for path-norms enhances neural network generalization bounds.
problem Establishing generalization bounds for modern neural networks.
method Introducing a comprehensive toolkit for path-norms in ReLU networks with various operations.
result Established generalization bounds for modern neural networks that are the most widely applicable and recover/beat the sharpest known bounds.
The paper improves least-squares regression learning rates for stronger norms.
problem Improving least-squares regression learning rates for stronger norms.
method Combining integral operator techniques with embedding properties.
result Learning rates for Sobolev norms without requiring the function to be in the hypothesis space.
Study shows various pixel p-norm measures do not match human perception of adversarial attacks.
problem Understanding human perception of adversarial attacks on image classification systems.
method Performed a behavioral study comparing different p-norm measures and alternative metrics.
result Human perception of adversarial attacks does not align with pixel p-norm measures and other metrics.
Improved Frank-Wolfe algorithm solves convex trace-norm ball problems.
problem Optimizing convex functions over trace-norm balls.
method Rank-k variant of Frank-Wolfe algorithm using top-k singular-vector computation.
result Linear convergence rate for smooth and strongly convex objectives with rank-limited solutions.
New nonconvex regularizers improve low-rank matrix recovery efficiency and accuracy.
problem Efficiently recover low-rank matrices from incomplete data.
method Factor group-sparse regularization, related to Schatten-p norms.
result Improved generalization error bounds for Schatten-p norms as p decreases.
S-MTGPR improves normative modeling of neuroimaging data.
problem Normative modeling of neuroimaging data without spatial covariance structure.
method Scalable multi-task Gaussian process regression (S-MTGPR) with low-rank approximation and Kronecker product.
result S-MTGPR provides higher sensitivity in novelty detection scenarios.
Recently, l2,1 matrix norm has been widely applied to many areas such as computer vision, pattern recognition, biological study and etc. As an extension of l1 vector norm, the mixed l2,1 matrix norm is often used to find jointly sparse solutions. Moreover, an efficient iterative algorithm has been designed…
Improved sample complexity for ReLU networks with norm constraints.
problem Estimating sample complexity for ReLU networks under norm constraints.
method Refined Rademacher complexity analysis for function class.
result Often no explicit depth-dependence in sample complexity bound.
New algorithm improves reinforcement learning policies without degrading performance.
problem Policy updates may degrade performance in reinforcement learning with general function approximators.
method Derives a new policy improvement bound with an average divergence instead of sup norm, leading to Easy Monotonic Policy Iteration.
result Generates sequences of policies with guaranteed non-decreasing returns.
A new algorithm optimizes L1-norm error fitting problems efficiently.
problem Optimizing L1-norm error fitting models with data aggregation.
method Data aggregation-based algorithm with monotonic convergence to global optimum.
result The proposed algorithm optimally solves any L1-norm error fitting model.
AdaGrad-Norm achieves linear convergence for certain functions.
problem Proving linear convergence for specific types of functions.
method Introducing RUIG, a measure of gradient balance; developing a two-stage framework.
result AdaGrad-Norm achieves linear convergence for certain functions.
HBR improves normative modeling of neuroimaging data across multiple sites.
problem Dealing with nuisance variation in neuroimaging data across different sites.
method Hierarchical Bayesian regression (HBR) for multi-site normative modeling.
result HBR provides more accurate normative ranges compared to existing methods.
A new clustering algorithm REFCMFS improves K-Means efficiency and robustness.
problem Efficiently clustering data with outliers and L0-norm constraints. method REFCMFS uses L2,1-norm robust loss and L0-norm constraint on membership matrix. result REFCMFS achieves more promising performance and efficient optimization.
This work improves normalization methods in deep networks, enhancing stability and performance.
problem Shortcomings of Batch-Normalization hinder its use for certain tasks.
method Presented a novel view on normalization methods and weight-decay, suggesting alternatives like L1 and L∞. result Improved normalization methods enable first batch-norm alternative for half-precision implementations.
Deep normative modeling of clinical neuroimaging data improves diagnostic performance.
problem Modeling variation of neuroimaging measures across individuals for psychiatric disorders.
method Proposes a deep normative modeling framework based on neural processes (NPs) for spatially structured mixed-effect modeling of neuroimaging data.
result Substantial improvements in novelty detection performance for certain diagnostic problems.
Novel L1-norm and L2-norm LDA methods improve discriminant analysis.
problem Improving linear discriminant analysis for robustness and adaptability.
method Proposes L1BLDA and L2BLDA using Bhattacharyya error bound, maximizing between-class scatters and minimizing within-class scatters.
result Proposed methods avoid SSS and have no rank limit, demonstrating robust performance and effectiveness.
Improved 2-bit covariance estimator with reduced operator norm error and no tuning needed.
problem Improving 2-bit covariance estimation with reduced operator norm error and no tuning needed.
method Proposed a new 2-bit covariance matrix estimator using triangular dithering scales.
result Improved operator norm error rate that depends on effective rank of covariance matrix, closing theoretical gap.
Unified scalable equivalent formulations for Schatten quasi-norms improve efficiency.
problem Efficiently solving Schatten quasi-norm minimization problems for large-scale matrices.
method Proved equivalence between Schatten-p quasi-norm and product/sum of Schatten-p1 and p2 norms of factor matrices.
result Transformed SQNM problems into simpler, more efficient algorithms for p>1/2.
This work improves trace norm regularization for multi-task learning with limited data.
problem Learning from few samples across multiple tasks.
method Trace norm regularization for a linear shared representation model.
result First estimation error bound for trace norm regularized estimator with scarce data.
Proposes a new method for hyperspectral image dimensionality reduction.
problem Highly correlated noisy hyperspectral images.
method Trace Lasso-L1 Graph Cut method using L1-norm for robustness and sparsity.
result Optimal projection matrix maximizing between-class dispersion to within-class dispersion.
New method improves brain activity analysis with better amplitude and source selection.
problem Improving brain activity analysis with high temporal and spatial resolution.
method Iterative reweighted Mixed-Norm Estimate (irMxNE) for solving non-convex optimization problems.
result Improves on standard Mixed Norm Estimate (MxNE) in amplitude bias, support recovery, and stability.