Gradient flow in ReLU networks biases towards generalization but makes them vulnerable to adversarial attacks.
problem Generalization vs. Adversarial Robustness in ReLU Networks
method Analysis of gradient flow in two-layer ReLU networks with clustered data.
result Gradient flow biases towards generalization but also makes networks vulnerable to adversarial attacks.
Mirror flow in shallow neural networks shows similar implicit bias to gradient flow, with key differences in curvature penalties.
problem Analyzing implicit bias in shallow neural networks with mirror flow.
method Characterization through variational problems and scaled potentials.
result Mirror flow with scaled potentials induces a rich class of biases not captured by RKHS norms.
EnCF improves data assimilation for implicit, non-smooth observations.
problem Data assimilation challenges with implicit, many-to-one observations.
method EnCF uses a stochastic controlled flow to update forecast distributions.
result EnCF outperforms Kalman filters for non-Gaussian, implicit observations.
New method for modeling densities on Riemannian manifolds with symmetries.
problem Modeling densities on Riemannian manifolds with known symmetry groups.
method Combining implicit neural layers and optimal transport theory to propose IRCPMs.
result IRCPMs are simpler to incorporate symmetries and less expensive than ODE-flows.
This paper analyzes implicit bias in Deep Linear Discriminant Analysis.
problem The implicit bias of Deep Linear Discriminant Analysis.
method Analyzing gradient flow on a L-layer diagonal linear network.
result Under balanced initialization, the network transforms additive updates into multiplicative updates, conserving the (2/L) quasi-norm.
ImpFlows generalize normalizing flows by implicitly defining transformations.
problem Creating flexible and tractable probability distributions.
method Implicitly defined invertible transformations using roots of equations.
result ImpFlows can represent functions that ResFlows cannot, with comparable parameters.
Novel approach finds implicit regularisation in two-player games using BEA.
problem Understanding implicit regularisation in two-player games.
method Using backward error analysis to construct continuous-time flows with gradient-eligible vector fields.
result Identifies new implicit regularisation effects in two-player games.
Gradient-trained shallow networks can generalize well but are vulnerable to small-radius adversarial attacks.
problem Adversarial robustness of gradient-trained shallow networks.
method Analysis of neuron alignment and polynomial ReLU activation.
result Gradient-trained shallow networks with polynomial ReLU activation are robust to small-radius adversarial attacks.
Study reveals how initialization scale affects training accuracy in linear networks.
problem Understanding implicit bias in linear classification models.
method Asymptotic analysis of gradient flow trajectories and training loss minimization.
result Implicit bias is more complex at reasonable initialization scales and training accuracies.
SGD with large learning rates can achieve better test accuracy than expected.
problem SGD with large learning rates often outperforms expected convergence bounds.
method Proved that SGD with small learning rates stays close to gradient flow path on modified loss.
result Explicitly adding an implicit regularizer to the loss improves test accuracy.
This study investigates how gradient-based methods bias neural networks trained on high-dimensional data.
problem The implicit biases of gradient-based optimization algorithms in neural networks trained on high-dimensional data.
method Investigation of gradient flow and gradient descent in two-layer fully-connected neural networks with leaky ReLU activations.
result Gradient flow and gradient descent lead to neural networks with low-rank solutions and linear decision boundaries.
JKO scheme adds deceleration in rapidly changing metric curvature directions.
problem Understanding the implicit bias of the JKO scheme in Wasserstein gradient flow.
method Characterized the implicit bias of the JKO scheme at second order in η, modifying the energy functional.
result JKO scheme adds deceleration in directions where metric curvature of J is rapidly changing.
SNPLA uses normalizing flows for efficient inference in implicit models.
problem Efficient inference in implicit models with complex likelihood and posterior learning.
method Sequential Neural Posterior and Likelihood Approximation (SNPLA) algorithm using normalizing flows.
result SNPLA achieves competitive performance with faster posterior draws compared to MCMC methods.
Sharp results link DLN gradient flow to basis pursuit optimization and GHA phase transitions.
problem Understanding implicit regularization in Diagonal Linear Networks.
method Sharp convergence bounds and characterization of ℓ 1 \ell_1 ℓ 1 minimizers. result Gradient flow of DLNs with tiny initialization approximates minimizers of basis pursuit optimization problem.
Study shows how SGD's implicit regularization relates to ridge regression.
problem Least squares regression optimization with mini-batch SGD.
method Analyzes stochastic gradient flow as a continuous-time model of SGD.
result Bound on excess risk of SGD flow over ridge regression, revealing how parameters drive risk.
New tensor formulation reveals gradient flow's bias in linear neural networks.
problem Understanding implicit bias in linear neural network training.
method Tensor formulation of neural networks, including fully-connected, diagonal, and convolutional networks.
result Gradient flow on linear tensor networks converges to solutions of specific optimization problems.
Gradient flow on softmax attention minimizes nuclear norm of weight matrices.
problem Classification with separate key and query weight matrices.
method Gradient flow on exponential loss, separability assumption, reparameterization, approximate KKT conditions.
result Gradient flow implicitly minimizes nuclear norm of weight matrices, contrasting with Frobenius norm minimization.
Gradient flow on ReLU networks converges to a simple model with few regions.
problem Understanding the dynamics of gradient flow in shallow ReLU networks.
method Analysis of gradient flow dynamics on univariate ReLU neural networks.
result Gradient flow converges to a network with at most O(r) linear regions.
Gradient descent in tensor factorization favors low-rank solutions.
problem Tackling implicit regularization in tensor factorization problems.
method Gradient descent with small random initialization for overparametrized tensor factorization.
result Gradient descent leads to implicit regularization towards low tubal rank solutions.
Gradient flow with infinitesimal initialization converges to Greedy Low-Rank Learning for matrix factorization.
problem Understanding implicit regularization in gradient descent for matrix factorization.
method Theoretical and empirical analysis of gradient flow with infinitesimal initialization and Greedy Low-Rank Learning.
result Gradient flow with infinitesimal initialization is mathematically equivalent to Greedy Low-Rank Learning for depth-2 matrix factorization under reasonable assumptions.
PVI improves SIVI by directly optimizing ELBO without parametric assumptions.
problem Intractable variational densities in SIVI methods.
method Particle Variational Inference (PVI) using empirical measures to approximate optimal mixing distributions.
result PVI directly optimizes the ELBO and performs favorably compared to other SIVI methods.
FTIP uses normalizing flows to improve posterior inference in function space.
problem Challenges in posterior inference with implicit-process priors.
method FTIP uses normalizing flows to define a richer variational distribution over combination weights.
result FTIP captures asymmetric and multimodal posterior structure better than Gaussian coefficient approximations.
MGD with early stopping tends to ridge regularization in least squares regression.
problem Characterizing the implicit regularization of MGD with early stopping.
method Continuous-time view of MGD (momentum gradient flow) and comparison with explicit ridge regularization.
result Under optimal tuning, the risk of MGF is no more than 1.54 times that of ridge.
Single ReLU neuron's gradient dynamics reveal support vectors as key to generalization.
problem Understanding the generalization capability of ReLU networks.
method Examined gradient flow dynamics and support vectors in single ReLU neuron training.
result Support vectors play a crucial role in the generalization of ReLU networks.
Deep residual networks implicitly converge to neural ODEs.
problem Link between discrete and continuous deep learning models.
method Establishing implicit regularization for residual networks towards neural ODEs.
result Deep residual networks initialized as discretizations of neural ODEs converge to such ODEs during training.
This paper shows how to train only the implicit layer of overparameterized implicit neural networks.
problem Understanding how the implicit layer contributes to the training of overparameterized implicit neural networks.
method Restricting training to only the implicit layer and analyzing the generalization error for ReLU-activated networks.
result Global convergence is guaranteed even if only the implicit layer is trained, and gradient flow with proper random initialization can achieve small generalization errors.
Gradient descent converges to a global minimum in nonlinear ReLU implicit networks with linear width.
problem Understanding convergence of gradient methods in nonlinear, infinitely deep ReLU networks.
method Introduced a scaling constant to ensure well-posedness of the equilibrium equation, proving convergence to a global minimum for linear width networks.
result Gradient descent converges to a global minimum at a linear rate for nonlinear ReLU implicit networks with linear width.
A geometric flow on ( 2 , 2 ) (2,2) ( 2 , 2 ) -forms is introduced which preserves the balanced condition of metrics, and whose stationary points satisfy the anomaly equation in Strominger systems. The existence of solutions for a short time is established, using Hamilton's version of the Nash-Moser implicit function theorem.
Study Ricci flow on torus bundles and related manifolds.
problem Compute Ricci flow formulas for torus bundles.
method Use invariant metrics compatible with connections on principal G-bundles.
result Solutions to Ricci flows on Heisenberg groups and Berger 3-spheres.
This paper proves SGD converges to global minimum for over-parameterized ReLU networks.
problem Theoretical understanding of implicit neural networks is limited.
method Gradient flow analysis of ReLU activated implicit neural networks.
result Randomly initialized gradient descent converges to global minimum at a linear rate for square loss function in over-parameterized ReLU networks.
A new flow optimizes synthetic data alignment for generative modeling.
problem Optimizing synthetic data to match target distributions.
method Score difference (SD) flow to reduce KL divergence.
result SD flow provides a theoretical link between generative models.
ReLU networks implicitly favor low-rank solutions, but not as strongly as linear networks.
problem Understanding implicit regularization in ReLU networks for rank minimization.
method Analysis of gradient flow on ReLU networks, empirical testing.
result Gradient flow on ReLU networks does not necessarily minimize ranks, unlike in linear networks.
SIFG uses noisy particles to efficiently sample from complex distributions.
problem Efficient sampling from complex distributions using particle-based methods.
method SIFG introduces a semi-implicit functional gradient flow with Gaussian noise to improve sampling efficiency and accuracy.
result SIFG achieves strong theoretical convergence guarantees and efficient sampling.
GD converges faster to flatter minima than gradient flow in shallow networks.
problem Understanding the dynamics of gradient descent in shallow linear networks.
method Analyzing the convergence rate and solution of gradient descent in depth-2 linear neural networks.
result GD converges linearly to flatter minima than gradient flow, even with large step sizes.
A new method improves flow matching by dynamically weighting density estimates.
problem High-dimensional integration inefficiency in flow matching.
method Density-weighted Dynamic Stein operators.
result Significant improvement in vector field smoothness and sampling efficiency.
Gradient descent biases towards stable rank networks for nearly-orthogonal data.
problem Understanding implicit bias in non-smooth neural networks trained by gradient descent.
method Analysis of two-layer ReLU and leaky ReLU networks trained by gradient descent on nearly-orthogonal data.
result Gradient descent biases towards networks with stable rank and uniform margin for nearly-orthogonal data.
Study accelerates gradient methods in machine learning, revealing risk and stability connections.
problem Understanding the statistical risk of accelerated gradient methods in machine learning.
method Continuous-time analysis of Nesterov's accelerated gradient method and Polyak's heavy ball method for least squares regression.
result Connections between early stopping, stability, and curvature of loss function are revealed.
Neural networks learn incrementally from orthogonal data, interpolating with minimal complexity.
problem Understanding the learning dynamics and implicit bias in ReLU networks with orthogonal data.
method Gradient flow analysis of two-layer ReLU networks from small initialization with orthogonal training data.
result The learned interpolator has a squared ℓ 2 \ell_2 ℓ 2 -norm scaling as n \sqrt{n} n , close to the minimal interpolator's complexity. AgFlow speeds up model selection in penalized PCA.
problem Efficient model selection in penalized PCA for HDLSS settings.
method Implicit regularization effect of gradient flow to reduce computation complexity.
result AgFlow achieves the complete solution path of L2-penalized PCA.
New method estimates HMM hidden states efficiently.
problem Inaccurate posterior predictive distribution in HMMs.
method Autoregressive-flow for estimating hidden states.
result Estimates comparable to SMC algorithm.
Study integrable geodesic flows on 2-surfaces with high-degree polynomial first integrals.
problem Integrable geodesic flows on 2-surfaces with high-degree polynomial first integrals.
method Semi-Hamiltonian systems of PDEs and generalized hodograph method.
result Construction of many local explicit and implicit integrable examples with polynomial first integrals of degrees 3, 4, 5.
Proposes LDIDPs for efficient sequential data generation from latent dynamical models.
problem Challenges in generating high-fidelity sequential samples from latent dynamical models.
method Utilizes implicit diffusion processes to sample from latent dynamical processes.
result Demonstrates accurate learning of dynamics and efficient generation of high-quality sequential data.
Local gluing connects flow lines in finite time intervals.
problem Connecting flow lines in finite time intervals.
method Functional analytic approach to define local gluing map.
result Explicit construction of local gluing map in Euclidean case; intricate construction in non-Euclidean case.
Stable neural flows ensure robustness and efficiency in deep learning.
problem Ensuring robustness and stability in deep learning models.
method Introducing a stable variant of neural ODEs with a neural network parametrizing an energy functional, solving as an optimal control problem with adjoint sensitivity analysis.
result The proposed model provides robustness against input perturbations and low computational burden.
MDNs offer a data-efficient alternative to diffusion and flow models for multimodal scientific learning.
problem Capturing multimodal conditional uncertainty in scientific inverse problems.
method Mixture Density Networks (MDNs) as explicit parametric density estimators.
result MDNs achieve superior generalization, interpretability, and sample efficiency in scientific tasks.
The paper explores how linear neural networks can overfit without bias when data is well-behaved.
problem Understanding why linear neural networks can generalize well despite fitting noisy data.
method Analyzing two-layer linear neural networks trained with gradient flow, deriving bounds on excess risk.
result The excess risk depends on initialization quality and data covariance matrix properties.
New error bounds for flow matching methods using deterministic sampling.
problem Improving the accuracy of flow matching methods for generating probability distributions.
method Derived error bounds for flow matching methods under deterministic sampling conditions.
result Presented error bounds for flow matching methods using L 2 L^2 L 2 loss and regularity conditions. A tuning-free method recovers jointly sparse signals in MMV using implicit regularization.
problem Recovering jointly sparse signals in MMV with minimal tuning or prior knowledge.
method Reparameterizes MMV estimation matrix into decoupled factors and applies gradient descent to a least-squares objective.
result Gradient descent dynamics exhibit a momentum-like effect, converging towards an idealized row-sparse solution.