New path-gradient estimator for continuous normalizing flows.
problem Limitation of simple Gaussian variational distributions in complex applications.
method Proposed a path-gradient estimator for continuous normalizing flows.
result Empirical evidence of superior performance of the new estimator.
Improved KL divergence estimators for normalizing flows lead to faster convergence and better approximations.
problem Estimating KL divergences for normalizing flows efficiently and accurately.
method Path-gradient estimators for reverse and forward KL divergences.
result Path-gradient estimators lead to faster convergence and better approximation results.
Researchers compare different gradient methods for ridge regression, finding conjugate gradients have similar performance.
problem Comparing statistical properties of different gradient methods in ridge regression.
method Explicit non-standard error decomposition to bound prediction error of conjugate gradient iterates.
result Conjugate gradient iterates share optimality properties with gradient flow and ridge regression up to a constant factor.
Proposes Geodesic Integrated Gradients (GIG) for more accurate feature attributions in deep networks.
problem Flawed attributions using straight paths from Integrated Gradients (IG).
method Introduces a model-induced Riemannian metric and computes attributions along geodesics.
result GIG produces more faithful attributions than IG on benchmarks.
New algorithmic view of ℓ2 regularization using ODEs and path-following methods.
problem Optimizing convex loss functions with ℓ2 regularization.
method Established an equivalence between ℓ2-regularized solution paths and ODEs, proposing path-following algorithms based on homotopy methods and numerical ODE solvers.
result The solution path can be viewed as a hybrid of gradient descent and Newton method, providing novel schemes to choose grid points and reducing computational cost.
PathBoost boosts graph-level predictions using path-based features.
problem Graph-level classification and regression challenges.
method Gradient tree boosting method for graph-level prediction.
result PathBoost outperforms graph neural networks and graph kernel approaches in many cases.
Improved sampling efficiency for molecular systems using path gradients after Flow Matching.
problem Improving sampling efficiency for complex molecular systems.
method Hybrid approach combining Flow Matching and path gradients.
result Up to a threefold increase in sampling efficiency for molecular systems.
A new path gradient estimator speeds up normalizing flows without sacrificing accuracy.
problem High computational cost and limited scalability of path gradient estimators for normalizing flows.
method Proposed a fast path gradient estimator that improves computational efficiency and scalability.
result The new estimator achieves superior performance and reduced variance across various applications.
PS-IG improves feature attribution by reducing noise and variance.
problem Improving feature attribution in machine learning models.
method Path-sampled integrated gradients (PS-IG) computes expected value over sampled baselines.
result PS-IG reduces attribution variance by a factor of 1/3 under uniform sampling.
Solves complex Monge-Ampère equation for Kähler-Ricci solitons.
problem Complex Monge-Ampère equation for shrinking gradient Kähler-Ricci solitons.
method Aubin continuity path, implementing another continuity method.
result Shows existence of solution for the equation.
Paper introduces a new gradient estimator for SNNs.
problem High variance in score function gradient estimator impedes SNNs training.
method Developed a differentiable point process to derive path-wise gradient estimator.
result Demonstrated effectiveness of path-wise gradient estimator through simulations.
Framework for training stochastic spiking neural networks with rough signals.
problem Training stochastic spiking neural networks with noisy spike timing and dynamics.
method Rough path theory and signature kernels for gradient computation.
result Pathwise gradients of SSNNs' trajectories and event times exist and satisfy a recursive relation.
For sub-Riemannian manifolds with a chosen complement, we first establish the derivative formula and integration by parts formula on path space with respect to a natural gradient operator. By using these formulae, we then show that upper and lower bounds of the horizontal Ricci curvature correspond to functional inequa…
URGE improves diffusion model quality without gradients or Hessian.
problem Improving sample quality in diffusion models without gradient evaluations.
method Path-wise importance reweighting via Girsanov change of measure.
result URGE achieves better generation quality than existing methods.
Paper tackles NAS problem by modeling it as a sparse supernet.
problem Neural Architecture Search (NAS) problem, particularly Mixed-Path Search.
method Model NAS as a sparse supernet with sparsity constraints. Use hierarchical accelerated proximal gradient algorithm for optimization.
result Proposed method finds compact, general, and powerful neural architectures.
Solves complex equation for specific geometric solitons.
problem Solving complex Monge-Ampère equation for specific geometric solitons.
method Aubin continuity path and continuity method.
result Initial value of the path parameter has a solution and is open to all.
As adversarial attacks pose a serious threat to the security of AI system in practice, such attacks have been extensively studied in the context of computer vision applications. However, few attentions have been paid to the adversarial research on automatic path finding. In this paper, we show dominant adversarial exam…
A regularized optimization problem over a large unstructured graph is studied, where the regularization term is tied to the graph geometry. Typical regularization examples include the total variation and the Laplacian regularizations over the graph. When applying the proximal gradient algorithm to solve this problem, t…
Study on test risk dynamics in learning theory with stochastic gradient flow.
problem Understanding test risk in stochastic gradient flow dynamics.
method Path integral formulation for small learning rates, explicit computation for weak features.
result Explicit corrections due to stochastic term in dynamics, good agreement with simulations.
We study a simplification of GAN training: the problem of transporting particles from a source to a target distribution. Starting from the Sobolev GAN critic, part of the gradient regularized GAN family, we show a strong relation with Optimal Transport (OT). Specifically with the less popular dynamic formulation of OT …
Transformers learn multi-step reasoning through gradient descent.
problem Understanding how transformers solve symbolic multi-step reasoning tasks.
method Theoretical analysis of gradient descent dynamics and multi-phase training.
result Trained one-layer transformers can solve both backward and forward reasoning tasks with generalization guarantees.
We develop a normative framework for hierarchical model-based policy optimization based on applying second-order methods in the space of all possible state-action paths. The resulting natural path gradient performs policy updates in a manner which is sensitive to the long-range correlational structure of the induced st…
Improves BED scalability for implicit models.
problem Designing experiments for implicit models with intractable data distributions.
method Hybrid gradient approach combining variational MI estimator, ES, and SGA.
result Significantly improves scalability of BED for implicit models.
Generalizes Li-Yau Harnack inequality to path space of manifolds.
problem Extending classical Harnack inequalities to infinite-dimensional path space.
method Defines finite-dimensional gradients and Laplacians on path space, proving a generalized Harnack inequality.
result Established a new Harnack inequality on path space of manifolds.
The study identifies volatility models from path geometry using signature-based methods.
problem Identifying different stochastic volatility models from observed data.
method Mapping volatility trajectories into a feature space via truncated path signatures and applying a gradient boosting classifier.
result The method achieves high classification accuracy across various volatility dynamics and parameter settings.
The study compares different game-theoretic attribution methods and finds that interventional Shapley values yield less consistent results than Aumann-Shapley due to path symmetry.
problem Investigating the influence of path choice on game-theoretic attribution algorithms.
method Comparative analysis of interventional Shapley values and Generalized Integrated Gradients (GIG) methods.
result Interventional Shapley values yield less consistent attributions than Aumann-Shapley due to path symmetry and extended away from the training data manifold.
Noether's theorem clarifies how symmetries in neural networks influence learning.
problem Understanding how symmetries in neural networks affect learning.
method Systematic study of symmetry interactions with learning algorithms using Noether's theorem.
result Symmetries impose restrictions on the optimization path, leading to conserved quantities.
This paper bridges variational inference and Wasserstein gradient flows.
problem Combining variational inference and Wasserstein gradient flows for more efficient approximations.
method Recasting Bures-Wasserstein gradient flow as a Euclidean gradient flow and using path-derivative gradient estimator.
result A new gradient estimator for f-divergences that can be implemented using machine learning libraries. Foundation for robust finance using rough path theory.
problem Mathematical models of financial markets under Knightian uncertainty.
method Introducing Property (RIE) for càdlàg paths, proving existence of rough integrals, verifying admissibility of trading strategies.
result Existence and stability of rough path integrals for non-gradient integrands.
For a 3-manifold M with b1(M)=1 fibered over S1 and the fiberwise gradient ξ of a fiberwise Morse function on M, we introduce the notion of amidakuji-like path (AL-path) on M. An AL-path is a piecewise smooth path on M consisting of edges each of which is either a part of a critical locus of ξ or a fl…
Gradient descent implicitly follows regularization for general losses.
problem The implicit bias of gradient descent methods in machine learning.
method Empirical risk minimization over linear predictors with arbitrary convex, strictly decreasing losses.
result Gradient descent and regularization paths converge to the same direction for non-attained risks.
This paper shows how path spaces on two-level manifolds can be Hilbert manifold structures.
problem Addressing the structure of path spaces on two-level manifolds.
method Introducing the notion of tameness and constructing charts on path spaces of two-level manifolds.
result Path spaces on tame two-level manifolds have the structure of a Hilbert manifold.
SGD generalization bounds derived from information theory.
problem Understanding generalization of SGD for non-convex functions.
method Combining information-theoretic bounds with perturbation analysis.
result Upper bounds on SGD's generalization error based on gradient variance and function smoothness.
New estimator for SDEs is shown to be an adjoint state method.
problem Estimating gradients for overparameterized SDEs efficiently.
method Demonstrates generator gradient estimator as an adjoint state method.
result Generator gradient estimator is an adjoint state method for SDEs.
Paper analyzes regret bounds for unconstrained online optimization.
problem Minimizing regret in dynamic online learning for strongly convex and smooth functions.
method Preconditioned OGD, Online Optimistic Newton (OON), multiple gradient queries.
result Achieves O(C2,T∗) regret bound with one gradient query per round. We generalize the classical Bochner formula for the heat flow on M to martingales on the path space PM, and develop a formalism to compute evolution equations for martingales on path space. We see that our Bochner formula on PM is related to two sided bounds on Ricci curvature in much the same manner that the classical…
Recurrent neural networks are known for their notorious exploding and vanishing gradient problem (EVGP). This problem becomes more evident in tasks where the information needed to correctly solve them exist over long time scales, because EVGP prevents important gradient components from being back-propagated adequately …
By using Hsu's multiplicative functional for the Neumann heat equation, a natural damped gradient operator is defined for the reflecting Brownian motion on compact manifolds with boundary. This operator is linked to quasi-invariant flows in terms of a integration by parts formula, which leads to the standard log-Sobole…
The behavior of the gradient descent (GD) algorithm is analyzed for a deep neural network model with skip-connections. It is proved that in the over-parametrized regime, for a suitable initialization, with high probability GD can find a global minimum exponentially fast. Generalization error estimates along the GD path…
Adapts IG for better feature attributions and robustness.
problem Reliability concerns in feature attributions for deep learning models.
method Adaptation of path-based feature attribution to Riemannian geometry of data manifolds.
result IG along geodesics generates more intuitive and robust explanations.
SGD with large learning rates can achieve better test accuracy than expected.
problem SGD with large learning rates often outperforms expected convergence bounds.
method Proved that SGD with small learning rates stays close to gradient flow path on modified loss.
result Explicitly adding an implicit regularizer to the loss improves test accuracy.
Variance-reduced algorithms, although achieve great theoretical performance, can run slowly in practice due to the periodic gradient estimation with a large batch of data. Batch-size adaptation thus arises as a promising approach to accelerate such algorithms. However, existing schemes either apply prescribed batch-siz…
New Banach spaces for ReLU networks enable better function approximation and gradient dynamics analysis.
problem Function approximation and gradient dynamics in multi-layer ReLU networks.
method Developed Banach spaces for ReLU networks, defined new function representations, and analyzed gradient flow dynamics.
result Gradient flow dynamics of the new representation is the continuous analog of gradient descent for ReLU networks.
In this paper, we aim to understand Residual Network (ResNet) in a scientifically sound way by providing a bridge between ResNet and Feynman path integral. In particular, we prove that the effect of residual block is equivalent to partial differential equation, and the ResNet transforming process can be equivalently co…
Gradient filters track moving parameters under noisy data and misspecification.
problem Tracking multidimensional time-varying parameters under noisy observations and model misspecification.
method Gradient-based filters update parameters using the gradient of a postulated objective function, evaluated at either the predicted or updated parameters.
result Novel sufficient conditions for exponential stability of the filtered parameter path, and finite-sample and asymptotic mean squared error bounds.
Greedy PIG adapts integrated gradients for better feature attribution.
problem Interpreting deep learning model predictions.
method Unified discrete optimization framework for feature attribution and selection.
result Greedy PIG improves feature attribution on various tasks.
It is well known that neural networks with rectified linear units (ReLU) activation functions are positively scale-invariant. Conventional algorithms like stochastic gradient descent optimize the neural networks in the vector space of weights, which is, however, not positively scale-invariant. This mismatch may lead to…
Proposes a new method to learn entire solution paths without discretization.
problem Optimizing a family of problems indexed by hyperparameters.
method Parameterizes the solution path with basis functions and solves a single stochastic optimization problem.
result Uniform error of learned path converges linearly to a constant related to basis expressiveness.