A new method computes gradients without backpropagation.
problem Optimization of machine learning models.
method Forward mode automatic differentiation to compute gradients.
result Forward gradient is an unbiased estimate of the gradient, eliminating the need for backpropagation.
Forward gradients improve neural network training without backpropagation issues.
problem Training neural networks without backpropagation's locking and memorization problems.
method Using directional derivatives in forward differentiation mode, with biased guesses based on feedback from small auxiliary networks.
result Using gradients from a local loss as a candidate direction improves Forward Gradient methods.
Forward-Euler fails for simulating Wasserstein gradient flows with KL divergence.
problem Simulating Wasserstein gradient flows with forward-Euler discretization fails for KL divergence.
method Forward-Euler discretization for Wasserstein gradient flows with KL divergence.
result Forward-Euler discretization can be incorrect for Wasserstein gradient flows with KL divergence.
Backpropagation-free trunk training improves model performance on various benchmarks.
problem Memory inefficiency and noisy gradient estimates in deep network training.
method Split Forward Gradient (Split-FG) method that splits network into trunk and head, estimating only trunk gradient.
result Split-FG achieves better performance than pure forward-gradient training and backpropagation on various benchmarks.
SGD converges with perturbed forward-backward passes, explained by geometric amplification.
problem Analyzing convergence of SGD with perturbed forward-backward passes in composite optimization.
method Characterized propagation and amplification of perturbations, derived convergence guarantees for non-convex and PL objectives.
result Perturbations cascade through the computational graph, affecting convergence order under specific conditions.
We study two procedures (reverse-mode and forward-mode) for computing the gradient of the validation error with respect to the hyperparameters of any iterative learning algorithm such as stochastic gradient descent. These procedures mirror two methods of computing gradients for recurrent neural networks and have differ…
FDS tackles long horizon hyperparameter optimization issues.
problem Memory scaling and gradient degradation in long horizon tasks.
method Forward-mode differentiation with sharing (FDS).
result Significantly outperforms greedy gradient-based alternatives.
FP uses random projections to train networks without feedback, achieving comparable performance to backpropagation.
problem Training neural networks without feedback from downstream layers.
method Forward Projection (FP) method that uses randomised nonlinear projections and closed-form regression.
result FP achieves comparable generalisation to backpropagation methods with a single forward pass, offering significant speedup.
Improved KL divergence estimators for normalizing flows lead to faster convergence and better approximations.
problem Estimating KL divergences for normalizing flows efficiently and accurately.
method Path-gradient estimators for reverse and forward KL divergences.
result Path-gradient estimators lead to faster convergence and better approximation results.
In this paper, we further study the forward-backward envelope first introduced in [28] and [30] for problems whose objective is the sum of a proper closed convex function and a twice continuously differentiable possibly nonconvex function with Lipschitz continuous gradient. We derive sufficient conditions on the origin…
CPS solves inverse problems using forward passes and constrained particle seeking.
problem Solving inverse problems with limited forward observation information.
method Gradient-free approach that reformulates inverse problem as constrained optimization.
result CPS achieves results comparable to gradient-based methods while outperforming alternatives.
New algorithm optimizes nonlinear SDEs online with convergence guarantees.
problem Optimizing nonlinear stochastic differential equations (SDEs) is computationally challenging.
method Forward propagation algorithm that solves an SDE derived using forward differentiation.
result Convergence theorem for nonlinear dissipative SDEs with bounds on stochastic fluctuations.
Paper analyzes PSGLD for adaptive IRL with finite-sample bounds.
problem Estimating cost function of a forward learner using noisy gradients.
method Passive stochastic gradient Langevin dynamics (PSGLD) algorithm.
result Explicit bounds on 2-Wasserstein distance between PSGLD sample measure and stationary measure.
We study sampling as optimization in the space of measures. We focus on gradient flow-based optimization with the Langevin dynamics as a case study. We investigate the source of the bias of the unadjusted Langevin algorithm (ULA) in discrete time, and consider how to remove or reduce the bias. We point out the difficul…
Backpropagation-free RL method trains layers using local signals.
problem Vanishing or exploding gradients in backpropagation-based RL.
method Local pairwise distance matching for layer-wise training without backpropagation.
result Backpropagation-free method achieves competitive performance and stability.
GANs can learn hierarchical distributions in real-world images efficiently.
problem Understanding and efficiently learning complex, real-world distributions with GANs.
method Formally studying how GANs can learn hierarchically generated distributions close to real-life image distributions using SGDA.
result Training GANs via SGDA can efficiently learn distributions with a 'forward super-resolution' structure, both in sample and time complexities.
DG improves policy gradient efficiency by selectively backpropagating only valuable samples.
problem Expensive backward passes in policy gradient methods reduce efficiency.
method Introduces 'delight' as a forward-pass signal of learning value and a Kondo gate to selectively backpropagate.
result Selective backpropagation reduces backward pass costs without sacrificing learning quality.
LSTM and gradient boosting models fail to outperform random chance in predicting MNQ futures.
problem Predicting intraday direction in MNQ futures using LSTM and gradient boosting.
method Comparing LSTM and gradient boosting models on 944 trading days of MNQ futures data.
result No model achieves statistically significant accuracy above random chance.
Gradient flossing stabilizes RNN training by controlling Lyapunov exponents.
problem Gradient instability in RNNs leading to exploding and vanishing gradients.
method Regularizing Lyapunov exponents through backpropagation using differentiable linear algebra.
result Gradient flossing improves RNN training success rate and convergence speed.
Method solves optimisation problems on non-Riemannian surfaces with bilateral curvature bounds.
problem Optimisation problems on non-Riemannian surfaces with sharp edges.
method Forward-backward splitting in Alexandrov spaces with bilateral curvature bounds.
result Convergence of the forward-backward method in Alexandrov spaces with bilateral curvature bounds.
SHINE uses forward pass quasi-Newton matrices to approximate Jacobian inverses for faster bi-level optimization.
problem Efficiently solving bi-level optimization problems with large Jacobian matrices.
method Proposes using quasi-Newton matrices from the forward pass to approximate the inverse Jacobian matrix.
result Empirically shows SHINE reduces computational cost of the backward pass for various problems.
DDEQs extend DEQs to discrete measure inputs using Wasserstein gradient flows.
problem Applying DEQs to discrete measure inputs like sets or point clouds.
method Wasserstein gradient flows for finding fixed points of discrete measures under permutation-invariance.
result DDEQs can compete with state-of-the-art models in tasks like point cloud classification and completion.
In this introductory paper, we discuss how quantitative finance problems under some common risk factor dynamics for some common instruments and approaches can be formulated as time-continuous or time-discrete forward-backward stochastic differential equations (FBSDE) final-value or control problems, how these final val…
Transformers learn multi-step reasoning through gradient descent.
problem Understanding how transformers solve symbolic multi-step reasoning tasks.
method Theoretical analysis of gradient descent dynamics and multi-phase training.
result Trained one-layer transformers can solve both backward and forward reasoning tasks with generalization guarantees.
SDPA is shown to be an optimal transport problem in deep learning.
problem The mathematical foundation and optimization perspective of SDPA.
method SDPA is shown to be the exact solution to a degenerate, one-sided Entropic Optimal Transport (EOT) problem.
result The SDPA mechanism is a principled mechanism where the forward pass performs optimal inference and the backward pass implements a rational, manifold-aware learning update.
A method makes particle filters differentiable without altering their forward pass.
problem Compatibility issues between particle filters and automatic differentiation.
method Introduces a correction to particle weights using the stop-gradient operator.
result Automatic differentiation produces good estimators for gradients and second-order derivatives.
Deep Neural Networks(DNNs) require huge GPU memory when training on modern image/video databases. Unfortunately, the GPU memory is physically finite, which limits the image resolutions and batch sizes that could be used in training for better DNN performance. Unlike solutions that require physically upgrade GPUs, the G…
The study compares feed-forward and attention layers in language models.
problem Understanding the role of feed-forward and attention layers in language models.
method Empirical and theoretical analysis in a synthetic setting.
result Feed-forward layers learn simple distributional associations, while attention layers focus on in-context reasoning.
Paper introduces FoMoH for optimization without backpropagation.
problem Optimizing machine learning models without backpropagation.
method Second-order hyperplane search, forward-mode stochastic gradient method, hyper-dual numbers, FoMoH.
result Developed a novel optimization algorithm that avoids backpropagation.
Innovative method solves nonconvex optimization on manifolds.
problem Nonconvex optimization problems on Riemannian manifolds.
method Intrinsic Riemannian proximal gradient method.
result Converges for nonconvex or nonembedded problems.
Unified framework for forward and inverse PDE problems in multiphase media.
problem Non-differentiable inverse problems in discrete-valued material fields.
method GenPANIS: Latent-variable generative framework preserving discrete microstructures.
result Unified bidirectional inference with minimal labeled pairs and physics-aware decoder.
Study inverse problems with measure samples, improving estimator calibration and recovery.
problem Inverse problems with unknown potentials observed through measure samples.
method Introduced convex empirical objectives and sharpened Fenchel--Young losses for finite-dimensional potential classes.
result High-probability parameter recovery bounds for inverse entropic unbalanced optimal transport and inverse JKO learning.
We conduct mathematical analysis on the effect of batch normalization (BN) on gradient backpropogation in residual network training, which is believed to play a critical role in addressing the gradient vanishing/explosion problem, in this work. By analyzing the mean and variance behavior of the input and the gradient i…
Recently mean field theory has been successfully used to analyze properties of wide, random neural networks. It gave rise to a prescriptive theory for initializing feed-forward neural networks with orthogonal weights, which ensures that both the forward propagated activations and the backpropagated gradients are near $…
Derives Mirror Descent from gradient flow on a Riemannian manifold.
problem No specific problem stated; focuses on derivation.
method Derives Mirror Descent from gradient flow on a Riemannian manifold with a natural discretization.
result Generalizes Mirror Descent to non-Hessian metrics.
Dropout and RaM become equivalent in large ResNets as depth and width increase.
problem Improving performance in deep learning models.
method Comparing Dropout and Random Gradient Masking in ResNets.
result Dropout and RaM converge to the same large-scale limiting dynamics in ResNets.
We propose a new optimization method for training feed-forward neural networks. By rewriting the activation function as an equivalent proximal operator, we approximate a feed-forward neural network by adding the proximal operators to the objective function as penalties, hence we call the lifted proximal operator machin…
This paper provides a unifying theoretical framework for stochastic optimization algorithms by means of a latent stochastic variational problem. Using techniques from stochastic control, the solution to the variational problem is shown to be equivalent to that of a Forward Backward Stochastic Differential Equation (FBS…
In recent years, the mean field theory has been applied to the study of neural networks and has achieved a great deal of success. The theory has been applied to various neural network structures, including CNNs, RNNs, Residual networks, and Batch normalization. Inevitably, recent work has also covered the use of dropou…
Gradient descent optimization improved by circuit perspective.
problem Efficient training of large AI systems.
method Utilizing circuit perspective for mechanistic interpretability.
result Designing a curriculum for efficient learning.
Single deep model detects out-of-distribution data with single forward pass.
problem Detecting out-of-distribution data points in neural networks.
method Deterministic uncertainty quantification (DUQ) using gradient penalty for reliable detection.
result Single model outperforms or matches ensemble methods in out-of-distribution detection.
New method shows how order of gradient updates impacts stability and convergence in deep learning.
problem Training deep learning models can be unstable and computationally expensive.
method Theoretical analysis and experiments with backward-SGD.
result The order of gradient updates affects stability and convergence, leading to improved performance.
A new family of momentum coefficients improves the convergence rate of accelerated algorithms.
problem Improving the convergence rate of accelerated gradient methods for strongly convex functions.
method Introducing a family of controllable momentum coefficients for forward-backward accelerated methods.
result Established a controllable $O\left(1/k^{2α}
ight)$ convergence rate for the NAG-α method. In this paper, we introduce a new form of amortized variational inference by using the forward KL divergence in a joint-contrastive variational loss. The resulting forward amortized variational inference is a likelihood-free method as its gradient can be sampled without bias and without requiring any evaluation of eith…
This paper introduces Selective-Backprop, a technique that accelerates the training of deep neural networks (DNNs) by prioritizing examples with high loss at each iteration. Selective-Backprop uses the output of a training example's forward pass to decide whether to use that example to compute gradients and update para…
In this work, we present direction-of-arrival (DoA) estimation algorithms based on the Krylov subspace that effectively exploit prior knowledge of the signals that impinge on a sensor array. The proposed multi-step knowledge-aided iterative conjugate gradient (CG) (MS-KAI-CG) algorithms perform subtraction of the unwan…
We consider forward-backward greedy algorithms for solving sparse feature selection problems with general convex smooth functions. A state-of-the-art greedy method, the Forward-Backward greedy algorithm (FoBa-obj) requires to solve a large number of optimization problems, thus it is not scalable for large-size problems…
Study on feature learning dynamics in infinite-depth neural networks, focusing on ResNets.
problem Understanding how features evolve during training in deep neural networks, especially in the large-depth limit.
method Conditional Gaussian representations and SDE system with decoupled backward weights.
result Depth-induced suppression of forward-backward coupling in infinite-depth networks, leading to a decoupled forward-backward SDE system.