Diagonal linear networks converge to lasso regularization path during training.
problem Understanding the regularization behavior of diagonal linear networks.
method Analyzing the training trajectory of diagonal linear networks and comparing it to the lasso regularization path.
result The training trajectory of diagonal linear networks is closely related to the lasso regularization path.
SGD on diagonal linear networks approximates to SDE in high dimensions.
problem Understanding optimization and generalization in neural models.
method High-dimensional analysis of SGD on diagonal linear networks, approximated by SDE.
result SGD dynamics in high dimensions converge exponentially to zero risk.
Noise in linear networks minimizes sharpness and leads to shrinkage-thresholding.
problem Minimizing sharpness in diagonal linear networks.
method Stochastic sharpness-aware minimization (SAM) with isotropic noise.
result Noise forces shrinkage-thresholding of true parameters.
This study uses continuous-time analysis to understand how momentum affects the optimisation of diagonal linear networks.
problem The effect of momentum on the optimisation trajectory of gradient descent.
method Leveraging a continuous-time approach to analyze momentum gradient descent with step size γ and momentum parameter β.
result Small values of λ help recover sparse solutions in overparametrised regression settings.
Study on GD and SGD over diagonal networks, focusing on stepsizes and regularisation.
problem Understanding the impact of stochasticity and large stepsizes on gradient descent and SGD solutions.
method Investigation of GD and SGD over diagonal linear networks with macroscopic stepsizes, proving convergence and characterizing solutions.
result Large stepsizes consistently benefit SGD for sparse regression problems, but can hinder GD recovery of sparse solutions, especially in the edge of stability regime.
In this paper, we study deep diagonal circulant neural networks, that is deep neural networks in which weight matrices are the product of diagonal and circulant ones. Besides making a theoretical analysis of their expressivity, we introduced principled techniques for training these models: we devise an initialization s…
Gradient methods work well on overparameterized diagonal linear networks.
problem Understanding why gradient-based methods work well in overparameterized models.
method Study of Deep Diagonal Linear Networks with gradient flow analysis.
result Gradient flow on layer parameters induces a mirror-flow dynamic in the effective parameter space, leading to explicit convergence guarantees.
Unified analysis of DLNs using DMFT reveals dynamics of loss convergence and generalization trade-offs.
problem Understanding the overall dynamics of diagonal linear networks (DLNs) in neural network training.
method Dynamical Mean-Field Theory (DMFT) applied to DLNs.
result Derives low-dimensional effective process capturing high-dimensional gradient flow dynamics.
Two Fisher information matrix estimators are analyzed for neural networks, focusing on their variances and trade-offs.
problem Estimating the Fisher information matrix in neural networks due to its high computational cost.
method Examined two popular diagonal Fisher information matrix estimators and their variances in neural networks for regression and classification.
result The variances of the estimators depend on the non-linearity with respect to different parameter groups and should not be neglected.
New insights into Hessian structure of neural networks reveal two forces.
problem Understanding the Hessian structure of neural networks.
method Analyzing the static and dynamic forces, comparing limit distributions using random matrix theory.
result The Hessian structure arises from a combination of static and dynamic forces, with C being a primary driver. Gradient descent with large momentum finds flatter minima.
problem Understanding the effects of momentum in gradient descent.
method Empirical and theoretical analysis of gradient descent with large momentum.
result Large momentum leads to flatter minima than gradient descent.
Gradient descent implicitly favors group sparsity in neural networks.
problem Understanding implicit regularization in neural networks for structured sparsity.
method Novel neural reparameterization for diagonally grouped linear networks.
result Gradient descent without explicit regularization biases towards group sparsity.
A susceptibility propagation that is constructed by combining a belief propagation and a linear response method is used for approximate computation for Markov random fields. Herein, we formulate a new, improved susceptibility propagation by using the concept of a diagonal matching method that is based on mean-field app…
Unified analysis of reweighted least-squares algorithms for linear models.
problem Recovering unknown signals from linear measurements using reweighted least squares.
method Unified asymptotic analysis of IRLS, lin-RFM, and alternating minimization algorithms.
result The algorithms can achieve favorable performance in a few iterations with appropriate reweighting.
Sharp results link DLN gradient flow to basis pursuit optimization and GHA phase transitions.
problem Understanding implicit regularization in Diagonal Linear Networks.
method Sharp convergence bounds and characterization of ℓ1 minimizers. result Gradient flow of DLNs with tiny initialization approximates minimizers of basis pursuit optimization problem.
The paper analyzes how gradient descent implicitly regularizes solutions in overparameterized neural networks, revealing depth-dependent regularization effects.
problem Understanding implicit regularization in overparameterized linear neural networks for regression problems.
method Analyzing the approximation error between gradient flow limit points and ℓ1-minimization solutions, deriving tight upper and lower bounds. result The approximation error decreases linearly for D≥3 and at a slower rate for D=2, linked to null space property constants. New Spencer complexes for Lie groupoids developed.
problem Developing Spencer complexes for Lie groupoids.
method Extending Malgrange's diagonal calculus to IimesG. result Construction of non-linear and linear Spencer complexes.
Study reveals how initialization scale affects training accuracy in linear networks.
problem Understanding implicit bias in linear classification models.
method Asymptotic analysis of gradient flow trajectories and training loss minimization.
result Implicit bias is more complex at reasonable initialization scales and training accuracies.
The covariance of a stationary process X is diagonalized by a Fourier transform. It does not take into account the complex Fourier phase and defines Gaussian maximum entropy models. We introduce a general family of phase harmonic covariance moments, which rely on complex phases to capture non-Gaussian properties. The…
New tensor formulation reveals gradient flow's bias in linear neural networks.
problem Understanding implicit bias in linear neural network training.
method Tensor formulation of neural networks, including fully-connected, diagonal, and convolutional networks.
result Gradient flow on linear tensor networks converges to solutions of specific optimization problems.
New findings show GD converges to a linear interpolator even with quadratic loss function under certain conditions.
problem Understanding convergence of Gradient Descent with quadratic loss functions.
method Parameterized linear regression with quadratic loss function, empirical and theoretical analysis.
result Gradient Descent converges to a linear interpolator even with quadratic loss function under the Edge of Stability regime.
We propose a fast general projection-free metric learning framework, where the minimization objective minM∈SQ(M) is a convex differentiable function of the metric matrix M, and M resides in the set S of generalized graph Laplacian matrices for con…
Dynamic pricing learns demand model from sparse product networks.
problem Minimizing revenue loss in a large network of products with unknown demand parameters.
method Combines optimism-in-the-face-of-uncertainty and PAC-Bayesian approaches.
result Achieves asymptotically optimal performance in terms of network size and time horizon.
This paper analyzes implicit bias in Deep Linear Discriminant Analysis.
problem The implicit bias of Deep Linear Discriminant Analysis.
method Analyzing gradient flow on a L-layer diagonal linear network.
result Under balanced initialization, the network transforms additive updates into multiplicative updates, conserving the (2/L) quasi-norm.
New approach uses hinge loss for iterative regularization in classification.
problem Improving classification accuracy through regularization.
method Develops an iterative regularization approach based on hinge loss.
result Proves convergence and rates of convergence for classification.
Unified analysis of parameter norms in overparameterized linear models, revealing scaling laws and thresholds.
problem Understanding the scaling of parameter norms in overparameterized linear models.
method Simple dual-ray analysis revealing competition between signal spike and bulk of null coordinates.
result Unified closed-form predictions for parameter norm scaling, including elbow and threshold laws.
In this paper, we propose a new Recurrent Neural Network (RNN) architecture. The novelty is simple: We use diagonal recurrent matrices instead of full. This results in better test likelihood and faster convergence compared to regular full RNNs in most of our experiments. We show the benefits of using diagonal recurrent…
We provide the first experimental results on non-synthetic datasets for the quasi-diagonal Riemannian gradient descents for neural networks introduced in [Ollivier, 2015]. These include the MNIST, SVHN, and FACE datasets as well as a previously unpublished electroencephalogram dataset. The quasi-diagonal Riemannian alg…
This paper improves linear system solving by optimizing matrix diagonal scaling.
problem Improving the condition number of a matrix for faster iterative methods.
method Left or right diagonal rescaling of the matrix A, with new bounds and algorithms.
result Jacobi preconditioning reduces A's condition number to within a quadratic factor of the best possible scaling.
Adaptive gradient approaches that automatically adjust the learning rate on a per-feature basis have been very popular for training deep networks. This rich class of algorithms includes Adagrad, RMSprop, Adam, and recent extensions. All these algorithms have adopted diagonal matrix adaptation, due to the prohibitive co…
Three methods for tuning HMC diagonal scale matrices compared.
problem Improving Hamiltonian Monte Carlo efficiency with diagonal scale matrices.
method Three approaches: ISG, median crossing frequency, and estimated marginal standard deviations.
result ISG method leads to more efficient sampling in many cases.
We define a second-order neural network stochastic gradient training algorithm whose block-diagonal structure effectively amounts to normalizing the unit activations. Investigating why this algorithm lacks in robustness then reveals two interesting insights. The first insight suggests a new way to scale the stepsizes, …
New MCMC method learns sparse preconditioner for high-dimensional problems.
problem High-dimensional sampling with complex correlation structures.
method Adaptive MCMC with sparse preconditioner using online PCA.
result Significant reduction in computational complexity and improved performance.
A number of recent papers have provided evidence that practical design questions about neural networks may be tackled theoretically by studying the behavior of random networks. However, until now the tools available for analyzing random neural networks have been relatively ad-hoc. In this work, we show that the distrib…
Paper proposes a new framework to improve stability-based bounds in deep learning.
problem Explaining generalization in overparameterized neural networks.
method Decomposes excess risk dynamics into signal and noise components, applying stability-based bounds only to the noise.
result The decomposition framework improves stability-based bounds and explains generalization in neural networks.
A new method solves diagonally constrained SDPs quickly and accurately.
problem Solving large-scale diagonally constrained SDPs efficiently.
method Combines momentum from convex optimization with coordinate descent and matrix factorization.
result Local linear convergence and first-order critical point convergence proved.
Paper develops methods for solving complex stochastic equations using Malliavin calculus.
problem Existence, uniqueness, and regularity of solutions to BSVIEs.
method Malliavin calculus for tackling diagonal processes and nonlinear dependence.
result Developed well-posedness results for BSVIEs, including probabilistic interpretation of PDEs and portfolio optimization.
A leading hypothesis for the surprising generalization of neural networks is that the dynamics of gradient descent bias the model towards simple solutions, by searching through the solution space in an incremental order of complexity. We formally define the notion of incremental learning dynamics and derive the conditi…
We propose a modular extension of backpropagation for the computation of block-diagonal approximations to various curvature matrices of the training objective (in particular, the Hessian, generalized Gauss-Newton, and positive-curvature Hessian). The approach reduces the otherwise tedious manual derivation of these mat…
Bayesian approach learns linear operators from noisy data.
problem Learning linear operators from noisy data in infinite-dimensional spaces.
method Bayesian approach with Gaussian priors.
result Establishes posterior contraction rates and generalization error guarantees.
This paper improves neural network generalization by dynamically learning kernel parameters.
problem Improving neural network generalization and adaptability.
method Diagonal adaptive kernel model that learns kernel eigenvalues and output coefficients during training.
result The diagonal adaptive kernel model significantly improves generalization over fixed-kernel methods.
New model handles complex non-linear relationships with hidden graph structures.
problem Modeling non-linear relationships with hidden graph-structured interactions.
method Block-diagonal localized mixture of polynomial experts (BLoMPE) regression model with penalized maximum likelihood selection criterion.
result Strong theoretical guarantee for finite-sample oracle inequality.
The paper explores how sinks and diagonal patterns prevent attention oversmoothing.
problem Preventing attention oversmoothing in neural networks.
method Analyzing geometric conditions and conditions for dense vs. sparse attention, proving equivalence between sinks and hard attention switch, and comparing the costs of sinks vs. diagonal patterns.
result Sinks and diagonal patterns effectively prevent attention oversmoothing, and diagonal patterns provide a more flexible approach.
We propose a novel approach to addressing the vanishing (or exploding) gradient problem in deep neural networks. We construct a new architecture for deep neural networks where all layers (except the output layer) of the network are a combination of rotation, permutation, diagonal, and activation sublayers which are all…
Study on nilpotent Lie algebras with specific metrics.
problem Classifying nilsolitons in nilpotent Lie algebras.
method Classification up to dimension 9, reduction to linear and polynomial equations.
result Complete classification of nice nilsolitons in various dimensions and signatures.
This paper explains double descent in linear neural networks, identifying new factors.
problem Understanding double descent in linear neural networks.
method Gradient flow derivation and necessary conditions for double descent.
result Singular values of input-output covariance matrix are important for double descent in two-layer models.
The adaptive gradient online learning method known as AdaGrad has seen widespread use in the machine learning community in stochastic and adversarial online learning problems and more recently in deep learning methods. The method's full-matrix incarnation offers much better theoretical guarantees and potentially better…
Apollo improves nonconvex stochastic optimization efficiency.
problem Nonconvex stochastic optimization challenges.
method Adaptive parameter-wise diagonal quasi-Newton method approximating Hessian.
result Significant improvements in convergence speed and generalization over SGD and Adam.