New path-gradient estimator for continuous normalizing flows.
problem Limitation of simple Gaussian variational distributions in complex applications.
method Proposed a path-gradient estimator for continuous normalizing flows.
result Empirical evidence of superior performance of the new estimator.
Improved KL divergence estimators for normalizing flows lead to faster convergence and better approximations.
problem Estimating KL divergences for normalizing flows efficiently and accurately.
method Path-gradient estimators for reverse and forward KL divergences.
result Path-gradient estimators lead to faster convergence and better approximation results.
A new path gradient estimator speeds up normalizing flows without sacrificing accuracy.
problem High computational cost and limited scalability of path gradient estimators for normalizing flows.
method Proposed a fast path gradient estimator that improves computational efficiency and scalability.
result The new estimator achieves superior performance and reduced variance across various applications.
Paper introduces a new gradient estimator for SNNs.
problem High variance in score function gradient estimator impedes SNNs training.
method Developed a differentiable point process to derive path-wise gradient estimator.
result Demonstrated effectiveness of path-wise gradient estimator through simulations.
Researchers compare different gradient methods for ridge regression, finding conjugate gradients have similar performance.
problem Comparing statistical properties of different gradient methods in ridge regression.
method Explicit non-standard error decomposition to bound prediction error of conjugate gradient iterates.
result Conjugate gradient iterates share optimality properties with gradient flow and ridge regression up to a constant factor.
New estimator for SDEs is shown to be an adjoint state method.
problem Estimating gradients for overparameterized SDEs efficiently.
method Demonstrates generator gradient estimator as an adjoint state method.
result Generator gradient estimator is an adjoint state method for SDEs.
This paper bridges variational inference and Wasserstein gradient flows.
problem Combining variational inference and Wasserstein gradient flows for more efficient approximations.
method Recasting Bures-Wasserstein gradient flow as a Euclidean gradient flow and using path-derivative gradient estimator.
result A new gradient estimator for f-divergences that can be implemented using machine learning libraries. Improves BED scalability for implicit models.
problem Designing experiments for implicit models with intractable data distributions.
method Hybrid gradient approach combining variational MI estimator, ES, and SGA.
result Significantly improves scalability of BED for implicit models.
We generalize the classical Bochner formula for the heat flow on M to martingales on the path space PM, and develop a formalism to compute evolution equations for martingales on path space. We see that our Bochner formula on PM is related to two sided bounds on Ricci curvature in much the same manner that the classical…
URGE improves diffusion model quality without gradients or Hessian.
problem Improving sample quality in diffusion models without gradient evaluations.
method Path-wise importance reweighting via Girsanov change of measure.
result URGE achieves better generation quality than existing methods.
Generalizes Li-Yau Harnack inequality to path space of manifolds.
problem Extending classical Harnack inequalities to infinite-dimensional path space.
method Defines finite-dimensional gradients and Laplacians on path space, proving a generalized Harnack inequality.
result Established a new Harnack inequality on path space of manifolds.
A new slicing method speeds up sliced Wasserstein estimation.
problem Efficiently estimating sliced Wasserstein distance.
method Random-Path Projecting Direction (RPD) for fast sampling.
result RPSW and IWRPSW show favorable performance in training generative models.
We generalize stochastic smoothing for gradient estimation of non-differentiable functions.
problem Gradient estimation for non-differentiable functions.
method Developed a general framework for relaxation and gradient estimation of non-differentiable black-box functions using stochastic smoothing with reduced assumptions.
result Empirically validated the effectiveness of variance reduction strategies for various non-differentiable tasks.
Foundation for robust finance using rough path theory.
problem Mathematical models of financial markets under Knightian uncertainty.
method Introducing Property (RIE) for càdlàg paths, proving existence of rough integrals, verifying admissibility of trading strategies.
result Existence and stability of rough path integrals for non-gradient integrands.
Proposes Geodesic Integrated Gradients (GIG) for more accurate feature attributions in deep networks.
problem Flawed attributions using straight paths from Integrated Gradients (IG).
method Introduces a model-induced Riemannian metric and computes attributions along geodesics.
result GIG produces more faithful attributions than IG on benchmarks.
New method for estimating gradients in stochastic binary networks.
problem Challenges in training neural networks with binary activations and weights.
method Combines sampling and analytic approximation steps to estimate gradients accurately.
result Significantly reduced variance at the cost of small bias, leading to practical tradeoffs.
New algorithmic view of ℓ2 regularization using ODEs and path-following methods.
problem Optimizing convex loss functions with ℓ2 regularization.
method Established an equivalence between ℓ2-regularized solution paths and ODEs, proposing path-following algorithms based on homotopy methods and numerical ODE solvers.
result The solution path can be viewed as a hybrid of gradient descent and Newton method, providing novel schemes to choose grid points and reducing computational cost.
The behavior of the gradient descent (GD) algorithm is analyzed for a deep neural network model with skip-connections. It is proved that in the over-parametrized regime, for a suitable initialization, with high probability GD can find a global minimum exponentially fast. Generalization error estimates along the GD path…
PathBoost boosts graph-level predictions using path-based features.
problem Graph-level classification and regression challenges.
method Gradient tree boosting method for graph-level prediction.
result PathBoost outperforms graph neural networks and graph kernel approaches in many cases.
Improved sampling efficiency for molecular systems using path gradients after Flow Matching.
problem Improving sampling efficiency for complex molecular systems.
method Hybrid approach combining Flow Matching and path gradients.
result Up to a threefold increase in sampling efficiency for molecular systems.
PS-IG improves feature attribution by reducing noise and variance.
problem Improving feature attribution in machine learning models.
method Path-sampled integrated gradients (PS-IG) computes expected value over sampled baselines.
result PS-IG reduces attribution variance by a factor of 1/3 under uniform sampling.
Variance-reduced algorithms, although achieve great theoretical performance, can run slowly in practice due to the periodic gradient estimation with a large batch of data. Batch-size adaptation thus arises as a promising approach to accelerate such algorithms. However, existing schemes either apply prescribed batch-siz…
We generalize the classical Bochner formula for the heat flow on evolving manifolds (M,gt)t∈[0,T] to an infinite-dimensional Bochner formula for martingales on parabolic path space PM of space-time M=M×[0,T]. Our new Bochner formula and the inequalities that follow from it a…
New method infers population dynamics from snapshots using path space optimization.
problem Recover dynamics of a population from its temporal marginals.
method Grid-free algorithm using Schrödinger bridges coupled via noisy gradient descent in mean-field limit.
result Global convergence to min-entropy estimator with end-to-end theoretical guarantees.
The paper develops statistical inference for gradient flows in optimization.
problem Uncertainty quantification along the entire optimization path.
method Uniform central limit theorem and algorithm-aware covariance estimator.
result Asymptotically valid confidence intervals for target parameter.
Solves complex Monge-Ampère equation for Kähler-Ricci solitons.
problem Complex Monge-Ampère equation for shrinking gradient Kähler-Ricci solitons.
method Aubin continuity path, implementing another continuity method.
result Shows existence of solution for the equation.
Framework for training stochastic spiking neural networks with rough signals.
problem Training stochastic spiking neural networks with noisy spike timing and dynamics.
method Rough path theory and signature kernels for gradient computation.
result Pathwise gradients of SSNNs' trajectories and event times exist and satisfy a recursive relation.
We study a risk-constrained version of the stochastic shortest path (SSP) problem, where the risk measure considered is Conditional Value-at-Risk (CVaR). We propose two algorithms that obtain a locally risk-optimal policy by employing four tools: stochastic approximation, mini batches, policy gradients and importance s…
Introduces a neural network-based method for efficient state and parameter estimation in complex systems.
problem Efficiently estimating state paths and parameters from noisy measurements in high-dimensional nonlinear systems.
method Bayesian Information Field Theory with neural network parameterization and optimization algorithms.
result Proposes a method to simplify and enrich state path parameterizations using neural networks, improving inference accuracy.
In this paper, we propose a new technique named \textit{Stochastic Path-Integrated Differential EstimatoR} (SPIDER), which can be used to track many deterministic quantities of interest with significantly reduced computational cost. We apply SPIDER to two tasks, namely the stochastic first-order and zeroth-order method…
For sub-Riemannian manifolds with a chosen complement, we first establish the derivative formula and integration by parts formula on path space with respect to a natural gradient operator. By using these formulae, we then show that upper and lower bounds of the horizontal Ricci curvature correspond to functional inequa…
This study examines biases in flow matching samplers using finite-sample estimation.
problem Biases in flow matching samplers when using finite-sample surrogates.
method Finite-sample plug-in estimation and hierarchy of empirical FM models.
result Exact empirical minimizer and smoothed plug-in regime identified for affine conditional flows.
Local gluing connects flow lines in finite time intervals.
problem Connecting flow lines in finite time intervals.
method Functional analytic approach to define local gluing map.
result Explicit construction of local gluing map in Euclidean case; intricate construction in non-Euclidean case.
Paper tackles NAS problem by modeling it as a sparse supernet.
problem Neural Architecture Search (NAS) problem, particularly Mixed-Path Search.
method Model NAS as a sparse supernet with sparsity constraints. Use hierarchical accelerated proximal gradient algorithm for optimization.
result Proposed method finds compact, general, and powerful neural architectures.
Solves complex equation for specific geometric solitons.
problem Solving complex Monge-Ampère equation for specific geometric solitons.
method Aubin continuity path and continuity method.
result Initial value of the path parameter has a solution and is open to all.
AgFlow speeds up model selection in penalized PCA.
problem Efficient model selection in penalized PCA for HDLSS settings.
method Implicit regularization effect of gradient flow to reduce computation complexity.
result AgFlow achieves the complete solution path of L2-penalized PCA.
New modifiers improve noisy RNN replay in hippocampal networks.
problem Improving noisy RNN replay in hippocampal networks.
method Three approaches: hidden state leakage, adaptation, and momentum.
result Hidden state leakage, adaptation, and momentum improve noisy RNN replay.
The paper optimizes bridge-type estimators for sparse models using pathwise methods.
problem Sparse parametric models with adaptive coefficients and multiple penalties.
method Pathwise optimization with accelerated proximal gradient descent and blockwise alternating optimization.
result Efficient computation of the full solution path for adaptive bridge estimators.
New estimator for digital options using path splitting and MLMC.
problem Estimating digital options with stochastic differential equations.
method Repeated path splitting, Multilevel Monte Carlo (MLMC).
result Estimator complexity similar to MLMC for Lipschitz payoffs.
New methods improve Monte Carlo estimation of partition functions.
problem Estimating the normalization constant of complex distributions.
method Annealing through paths of distributions to estimate partition functions.
result Optimal path for estimation is arithmetic, improving efficiency.
New method estimates velocity fields for minimizing f-divergences without overfitting.
problem Minimizing statistical discrepancies between target and particle distributions.
method Directly estimate velocity fields using interpolation techniques, proving consistency under mild conditions.
result Consistent estimators of velocity fields improve accuracy in applications like domain adaptation and missing data imputation.
As adversarial attacks pose a serious threat to the security of AI system in practice, such attacks have been extensively studied in the context of computer vision applications. However, few attentions have been paid to the adversarial research on automatic path finding. In this paper, we show dominant adversarial exam…
A scalable method for BED with implicit models using approximate gradients.
problem Efficiently estimating posterior distribution and maximizing MI for implicit models.
method Stochastic approximate gradient ascent with smoothed variational MI estimator.
result Significantly improves scalability of BED in high-dimensional problems.
A regularized optimization problem over a large unstructured graph is studied, where the regularization term is tied to the graph geometry. Typical regularization examples include the total variation and the Laplacian regularizations over the graph. When applying the proximal gradient algorithm to solve this problem, t…
Study on test risk dynamics in learning theory with stochastic gradient flow.
problem Understanding test risk in stochastic gradient flow dynamics.
method Path integral formulation for small learning rates, explicit computation for weak features.
result Explicit corrections due to stochastic term in dynamics, good agreement with simulations.
We study a fully nonlinear PDE involving a linear combination of symmetric polynomials of the Kähler form on a Kähler manifold. A C0 \emph{a priori} estimate is proven in general and a gradient estimate is proven in certain cases. Independently, we also provide a method-of-continuity proof via a path of Kähler metri…
We study a simplification of GAN training: the problem of transporting particles from a source to a target distribution. Starting from the Sobolev GAN critic, part of the gradient regularized GAN family, we show a strong relation with Optimal Transport (OT). Specifically with the less popular dynamic formulation of OT …
Transformers learn multi-step reasoning through gradient descent.
problem Understanding how transformers solve symbolic multi-step reasoning tasks.
method Theoretical analysis of gradient descent dynamics and multi-phase training.
result Trained one-layer transformers can solve both backward and forward reasoning tasks with generalization guarantees.