The reparameterization gradient has become a widely used method to obtain Monte Carlo gradients to optimize the variational objective. However, this technique does not easily apply to commonly used distributions such as beta or gamma without further approximations, and most practical applications of the reparameterizat…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,051 papers · 148 categories
Trend · papers per month
48 results for “generalized gradients”
Study on gradient solitons on specific manifolds.
problem Existence and properties of gradient solitons on warped product manifolds.
method Analyzing necessary and sufficient conditions for the existence of generalized quasi Yamabe gradient solitons.
result Existence of non-trivial gradient Yamabe solitons on specific spacetimes.
Gradient penalty improves GAN performance by inducing a large-margin classifier.
problem Improving GAN performance and addressing vanishing gradients.
method A unifying framework of expected margin maximization, showing gradient penalties induce large-margin classifiers.
result Gradient penalties reduce vanishing gradients and produce better generated outputs.
We prove that a nontrivial complete generalized quasi Yamabe gradient soliton (M; g) must be a quasi Yamabe gradient soliton on each connected component of M and that a nontrivial complete locally conformally at generalized quasi Yamabe gradient soliton has a special warped product structure.
New findings show noisy gradient descent can generalize well, even with non-SGD noise.
problem The role of noise in gradient descent's generalization ability.
method Analyzed the structure of SGD noise and proposed a new noisy gradient descent algorithm.
result Noises in classes different from SGD can also effectively regularize gradient descent.
SGD reduces test error by decorrelating updates.
problem Improving generalization error in machine learning models.
method Derive a formula for generalization gap change due to SGD updates, compare to GD, and show decorrelation effect.
result SGD implicitly regularizes generalization error by decorrelating updates.
New approach improves model generalization through distributionally robust learning.
problem Improving model generalization in machine learning.
method Stochastic gradient descent applied to the outer minimization problem, with gradient estimation through multi-level Monte Carlo randomization.
result Our approach yields significant benefits over previous work in numerical experiments.
Proposes Coherent Gradients to explain and reduce overfitting in neural networks.
problem Why neural networks generalize well despite fitting random data.
method Hypothesis about gradient dynamics and a modification to gradient descent.
result Supports hypothesis with heuristic arguments and perturbative experiments.
New findings on optimal transport gradient for generative models, addressing numerical instabilities.
problem Numerical instabilities in training Wasserstein Generative Adversarial Networks (WGAN).
method Valid differentiation theorem for entropic regularized transport, semi-discrete gradient formulation, and optimization algorithm.
result Existence of optimal transport gradient for generative models under specified conditions.
Generalizes NTK for surrogate gradient learning in neural networks.
problem Lack of theoretical foundation for surrogate gradient learning.
method Generalizes neural tangent kernel (NTK) for surrogate gradient learning (SGL).
result Surrogate gradient NTK provides a good characterization of SGL.
New method for bidirectional generative modeling using adversarial gradient estimation.
problem Bidirectional generative modeling with various -divergences.
method Adversarial gradient estimation for -divergence optimization.
result Similar algorithms for different -divergences with varying scaling.
Optimizes CNNs by directing gradients along output channels.
problem Improving generalization error in CNNs.
method Output-channel directed re-weighted L2 or Sobolev metrics.
result Improves generalization error by optimizing gradients.
Generalization in deep neural networks can be analyzed using minimax rates for gradient methods.
problem Generalization performance of over-parameterized neural networks
method Establishing a connection between gradient-based methods and kernel methods
result Deriving minimax-optimal rates for GD and SGD under polynomial network width scaling
GENIE balances domain-invariant feature learning and gradient alignment for improved DG performance.
problem Domain Generalization (DG) overfitting to domain-specific features
method GENIE (Generalization-ENhancing Iterative Equalizer) optimizer
result Prevents a small subset of parameters from dominating optimization, promoting domain-invariant feature learning
Generalizes smoothness conditions for optimization methods.
problem Optimization under non-uniform smoothness conditions.
method Develops a new analysis technique for bounding gradients.
result Obtains convergence rates for gradient descent and Nesterov's method.
Gradient descent methods for deep ReLU networks achieve optimal generalization rates.
problem Generalization of gradient descent methods for deep neural networks
method Establishing minimax-optimal rates for GD and SGD with deep ReLU networks
result Gradient descent methods for deep ReLU networks achieve optimal generalization rates
Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problemcond-mat.dis-nn
Gradient descent variants improve phase retrieval accuracy.
problem Phase retrieval problem in high-dimensional spaces.
method Gradient descent, stochastic gradient descent, Langevin algorithm, dynamical mean-field theory.
result Stochastic variants of gradient descent achieve better generalization in phase retrieval.
Gradient descent on MMD GAN parameter space converges globally to target distribution.
problem Convergence of gradient descent in Maximum Mean Discrepancy (MMD) GANs.
method Proposes a parametric kernelized gradient flow that mimics the min-max game in gradient regularized MMD GAN.
result Gradient descent on the generator's parameter space in gradient regularized MMD GAN is globally convergent to the target distribution under certain conditions.
Paper improves convergence rates and step sizes for gradient algorithms.
problem Limited convergence rates and step sizes in existing algorithms.
method General proximal incremental aggregated gradient algorithm.
result Better and novel convergence results under general scheme.
New method for natural policy gradients converges linearly.
problem Improving natural policy gradient methods for better convergence.
method Fisher-Rao gradient flow applied to state-action distributions.
result Linear convergence rate with geometry-dependent factor.
A new ParVI framework improves particle-based variational inference methods.
problem Non-trivial kernel design in particle-based variational inference methods.
method Proposes a generalized Wasserstein gradient descent (GWG) framework with broader regularizers.
result Demonstrates strong convergence guarantees and effectiveness on simulated and real data.
Kernel-Gradient Drifting improves generative modeling for non-Euclidean data.
problem Challenges in generative modeling for non-Euclidean data.
method Replaces Euclidean displacement with kernel-induced directions, exposing score-based structure.
result Kernel-gradient drifting enables state-of-the-art one-step generation for non-Euclidean data.
SGD improves generalization by using gradient variability as a proxy for data randomness.
problem Improving generalization in machine learning models trained with stochastic gradient descent.
method Bootstrap perspective on SGD, analyzing gradient variability and algorithmic variability.
result SGD avoids spurious solutions and improves generalization by implicitly regularizing the trace of the gradient covariance matrix.
Communication-Efficient Adaptive Batch Size Strategies for Distributed Local Gradient Methodsstat.ML
Adaptive batch sizes improve local gradient methods in distributed training.
problem Communication bottlenecks in distributed deep learning.
method Adaptive batch size strategies for local gradient methods.
result Adaptive batch sizes reduce minibatch gradient variance and improve training efficiency.
A framework for natural gradient with arbitrary similarity measures.
problem Unclear metric for natural gradient in non-Euclidean spaces.
method Derive a metric for natural gradient given an arbitrary similarity measure.
result General framework for natural gradient in non-Euclidean spaces.
Study on generalization in gradient-based meta-learning, showing flatter solutions and coherence between adaptation trajectories.
problem Understanding generalization in gradient-based meta-learning.
method Analysis of objective landscapes, experimental demonstration of solution properties, and empirical evidence on coherence between adaptation trajectories.
result Meta-test solutions become flatter, lower in loss, and further away from the meta-train solution as meta-training progresses, even as generalization starts to degrade.
Gradient descent dynamics in nonconvex models explained with universality.
problem Understanding long-time behavior of nonconvex gradient descent.
method Developed a state evolution system for tracking gradient descent iterates.
result Gradient descent iterates are approximately independent of data and strongly incoherent with feature vectors.
EAGC boosts GCD by regulating gradient entanglement, improving known and novel category separability.
problem Gradient entanglement distorts supervised gradients and overlaps known and novel class representations.
method EAGC uses AGA and EEP to align and project gradients, reducing entanglement and overlap.
result EAGC consistently boosts GCD performance, setting new state-of-the-art results.
Paper establishes a generalization bound for gradient flow using a data-dependent kernel.
problem Understanding the generalization properties of gradient-based optimization methods.
method Establishes a generalization bound for gradient flow through a data-dependent kernel called the loss path kernel (LPK).
result The LPK captures the entire training trajectory and leads to tighter generalization guarantees.
The paper connects optimization and generalization using a new gradient inequality.
problem Connecting optimization dynamics to generalization bounds in machine learning.
method The approach uses the Łojasiewicz gradient inequality to derive convergence rates and generalization bounds.
result The framework provides generalization estimates matching or extending previous results for various models.
Natural gradient descent avoids the magic of model parametrization, leading to different optimization outcomes.
problem Understanding the impact of model parametrization on optimization and generalization in deep learning.
method Characterization of natural gradient flow in deep linear networks and nonlinear neural networks.
result Natural gradient descent fails to generalize in some cases, while gradient descent with the right architecture performs well.
The paper analyzes stability and generalization of shallow neural networks using gradient methods.
problem Understanding the generalization of overparameterized shallow neural networks.
method The paper uses gradient descent and stochastic gradient descent to study shallow neural networks, developing consistent excess risk bounds.
result The analysis improves on existing methods by providing a refined estimation of iterates and Hessian eigenvalues, leading to better excess risk bounds.
The paper analyzes stability and generalization of decentralized SGD.
problem Stability and generalization of decentralized stochastic gradient descent.
method Novel formulation of decentralized stochastic gradient descent combined with non/convex optimization theory.
result First stability and generalization guarantees for decentralized stochastic gradient descent.
This paper shows equivalence between SVGD and BBVI using kernel gradient flows.
problem Bayesian inference methods and their equivalence.
method Formalizes equivalence between SVGD and BBVI using kernel gradient flows.
result BBVI corresponds precisely to SVGD when using the neural tangent kernel.
Introduces gradient decay in Softmax for better generalization.
problem Improving generalization performance in neural networks.
method Gradient decay hyperparameter in Softmax for varying gradient rates based on probability.
result Gradient decay rate affects generalization performance and can be tuned for better optimization.
New method corrects bias in stochastic gradient samplers.
problem Bias in stochastic gradient samplers.
method Gradient-Guided Monte Carlo with stochastic gradients.
result Corrected sampler yields nonzero acceptance probabilities.
AutoClip automatically adjusts gradient clipping for better audio separation.
problem Improving generalization in audio source separation networks.
method Adaptive gradient clipping based on historical gradient norms.
result Improves generalization performance in audio source separation networks.
Paper introduces a differentially private generative model using gradient flow and sliced Wasserstein distance.
problem Protecting privacy in sensitive training data for generative models.
method Gradient flow in the space of probability measures, Gaussian-smoothed Sliced Wasserstein Distance, and numerical scheme for SDE.
result Demonstrates higher-fidelity data generation at low privacy budget compared to existing methods.
Single ReLU neuron's gradient dynamics reveal support vectors as key to generalization.
problem Understanding the generalization capability of ReLU networks.
method Examined gradient flow dynamics and support vectors in single ReLU neuron training.
result Support vectors play a crucial role in the generalization of ReLU networks.
Generalized Exponentiated Gradient Algorithms and Their Application to On-Line Portfolio Selectioncs.LG
EGAB algorithms improve online portfolio selection.
problem Online portfolio selection problem.
method Generalized exponentiated gradient (EG) updates with Alpha-Beta divergence regularization.
result EGAB algorithms enhance portfolio performance, especially with transaction costs.
GNC smooths loss function for large-batch SGD, improving generalization.
problem Extremely large-batch SGD leads to poor generalization and converges to sharp minima.
method Gradient noise convolution (GNC) smooths loss function by convolving gradient noise with the loss function.
result GNC achieves state-of-the-art generalization performance for large-scale deep neural networks.
New method for RL with general utilities using variational policy gradient.
problem Optimizing policies with general concave utility functions in RL.
method Derives Variational Policy Gradient Theorem, develops variational Monte Carlo gradient estimation algorithm.
result Global convergence to optimal policy for general objectives, exponential convergence under strong convexity.
Adaptive correlated MC improves sequence generation stability.
problem High gradient variance in reinforcement learning for sequence generation.
method Adapts policy gradient estimator using correlated Monte Carlo rollouts.
result Reduces gradient variance and improves model performance.
New study shows neural networks can generalize without gradient descent, especially in deep settings.
problem Whether neural networks need gradient descent for generalization.
method Theoretical study of matrix factorization with linear and non-linear activation, comparing gradient descent to Guess & Check.
result Generalization under Guess & Check deteriorates with increasing width but improves with depth, challenging conventional wisdom.
New bounds for model generalization under deterministic gradient descent.
problem Establishing generalization bounds for models trained with gradient descent methods.
method PAC-Bayesian bounds for deterministic optimisation algorithms.
result Fully computable bounds that depend on initial distribution and Hessian.
This research explains why SGD generalizes better than ADAM in deep learning.
problem Understanding the generalization gap between SGD and ADAM in deep learning.
method Analyzing local convergence behaviors through Levy-driven stochastic differential equations (SDEs).
result SGD is more locally unstable and better escapes from sharp minima to flatter ones, leading to better generalization.
New algorithm tackles nonconvex machine learning problems with adaptive normalization and independent sampling.
problem Nonconvex machine learning problems with generalized-smoothness.
method Adaptive gradient normalization, independent sampling, and gradient clipping.
result Achieves an O(ε^(-4)) sample complexity for fast convergence.
GraN-GAN normalizes gradients for better GAN performance.
problem Improving image generation in GANs with piecewise linear discriminators.
method Piecewise Gradient Normalization (GraN) for input-dependent normalization.
result Significant performance gains in image generation across various datasets.