First-order methods play a central role in large-scale machine learning. Even though many variations exist, each suited to a particular problem, almost all such methods fundamentally rely on two types of algorithmic steps: gradient descent, which yields primal progress, and mirror descent, which yields dual progress. W…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “gradient coupling”
In this sequel to arXiv:1510.03817, we apply our abstract Lojasiewicz-Simon gradient inequality to prove Lojasiewicz-Simon gradient inequalities for coupled Yang-Mills energy functions using Sobolev spaces which impose minimal regularity requirements on pairs of connections and sections. The Lojasiewicz-Simon gradient …
A new RNN model based on coupled oscillators mitigates gradient issues.
problem Gradient vanishing and exploding issues in RNNs.
method Time-discretization of a system of second-order ODEs modeling coupled oscillators.
result The model maintains bounded gradients, leading to stable learning of long-term dependencies.
New gradient estimators for discrete variables improve model training.
problem Training models with discrete latent variables is challenging due to high gradient variance.
method Introduced novel gradient estimators based on importance sampling and statistical couplings, extending to categorical variables.
result Proposed gradient estimators outperform previous methods in systematic experiments.
Sharp pseudospectral bounds prevent transient amplification in coupled gradient descent.
problem Transient amplification in coupled gradient descent systems.
method Developed a sharp pseudospectral theory for block-triangular Jacobians, proving Kreiss constant bounds and matching minimax lower bounds.
result Obtained a finite-horizon iteration-complexity bound of for stochastic coupled descent.
BinaryDuo: Reducing Gradient Mismatch in Binary Activation Network by Coupling Binary Activationscs.LG
BinaryDuo improves BNNs by coupling binary activations, outperforming state-of-the-art models.
problem Gradient mismatch in BNNs due to binarizing activations.
method Using gradient of smoothed loss function to estimate gradient mismatch, proposing BinaryDuo scheme with coupled ternary activations.
result BinaryDuo outperforms state-of-the-art BNNs on various benchmarks.
Paper proposes a coupling-based diagnostic for SGD stepsize optimization.
problem Optimizing stepsize for SGD convergence.
method Coupling-based convergence diagnostic for monitoring stationarity.
result Proposed stepsize scheme achieves superior performance across convex and non-convex problems.
SGLD proves geometric ergodicity via reflection coupling for nonconvex log-concave distributions.
problem Proving geometric ergodicity of SGLD in nonconvex, log-concave settings.
method Reflection coupling technique to handle SGLD's time discretization and minibatch issues.
result SGLD has an invariant distribution and geometric ergodicity in distance.
Two algorithms solve nonconvex minimax problems with linear constraints, achieving complexity guarantees.
problem Nonconvex minimax problems with coupled linear constraints.
method Zeroth-order primal-dual alternating projected gradient (ZO-PDAPG) and zeroth-order regularized momentum primal-dual projected gradient (ZO-RMPDPG) algorithms.
result Iteration complexity guarantees for solving nonconvex-(strongly) concave minimax problems with coupled linear constraints.
Efficiently factorizes coupled matrix tensor data for better accuracy and speed.
problem Poor computation efficiency in existing N-CMTF algorithms.
method Column-wise element selection to prevent frequent gradient updates.
result More accurate and computationally efficient factorization.
Weak Convergence of Approximate reflection coupling and its Application to Non-convex Optimizationmath.PR
Paper proposes a weak approximation of reflection coupling for non-convex optimization.
problem Non-convex optimization problems with different drift terms.
method Proposes an approximate reflection coupling (ARC) for stochastic differential equations (SDEs).
result ARC converges weakly to the reflection coupling and can be applied to non-convex optimization.
We consider parallel asynchronous Markov Chain Monte Carlo (MCMC) sampling for problems where we can leverage (stochastic) gradients to define continuous dynamics which explore the target distribution. We outline a solution strategy for this setting based on stochastic gradient Hamiltonian Monte Carlo sampling (SGHMC) …
Coupling by reflection of diffusion processes via discrete approximation under a backward Ricci flowmath.PR
A coupling by reflection of a time-inhomogeneous diffusion process on a manifold are studied. The condition we assume is a natural time-inhomogeneous extension of lower Ricci curvature bounds. In particular, it includes the case of backward Ricci flow. As in time-homogeneous cases, our coupling provides a gradient esti…
The Black Box Variational Inference (Ranganath et al. (2014)) algorithm provides a universal method for Variational Inference, but taking advantage of special properties of the approximation family or of the target can improve the convergence speed significantly. For example, if the approximation family is a transforma…
New algorithm AG-OG optimizes separable convex-concave problems efficiently.
problem Efficiently solving separable convex-concave minimax optimization problems.
method Leverages Nesterov acceleration and optimistic gradient on component and coupling parts of the problem.
result Achieves optimal convergence rate for various settings including bilinearly coupled problems.
Stochastic Schwarz lemma on Kähler manifolds via couplings.
problem Develop a new Schwarz lemma for Kähler manifolds.
method Probabilistic approach using Markovian couplings.
result Improved gradient estimates for harmonic functions.
New method improves sampling efficiency in complex stochastic systems.
problem Sampling efficiency in nonconvex stochastic gradient cases.
method Reflection coupling for unadjusted generalized Hamiltonian Monte Carlo.
result Quantitative Gaussian concentration bounds and convergence rates established.
New algorithm solves minimax games with linear constraints.
problem Nonconvex minimax games with coupled linear constraints.
method Primal-dual alternating proximal gradient (PDAPG) algorithm.
result Achieves ε-stationary solution within O(ε^(-2)) iterations for strongly concave settings.
Paper studies asymmetric matrix sensing, proving gradient descent converges to low-rank solutions.
problem Reconstructing asymmetric low-rank matrices from linear measurements.
method Factorized gradient descent with coupling and regularization properties.
result Gradient descent from small random initialization converges to globally optimal and generalizing solutions.
We study gradient bounds and other functional inequalities for the diffusion semigroup generated by Kolmogorov type operators. The focus is on two different methods: coupling techniques and generalized -calculus techniques. The advantages and drawbacks of each of these methods are discussed.
CFIL uses coupled flows to model state distributions for imitation learning.
problem Lack of explicit modeling of state distributions in reinforcement and imitation learning.
method Coupled normalizing flows for state and state-action distributions.
result CFIL achieves state-of-the-art performance on benchmark tasks.
The paper proposes methods to estimate MCMC quality with couplings, bounding Wasserstein distance.
problem Improving MCMC efficiency without sacrificing asymptotic consistency.
method Estimators based on couplings of Markov chains to assess quality of asymptotically biased sampling methods.
result Empirical upper bounds of Wasserstein distance for assessing MCMC quality.
End-to-end training of DBMs with improved gradient estimation.
problem Biased gradient estimation in DBMs, especially with high-dimensional states.
method Unbiased contrastive divergence using MH coupling and local mode initialization.
result End-to-end training of DBMs without greedy pretraining, achieving FID score of 10.33 for MNIST.
Unbiased gradient estimation improves VAE performance.
problem Training VAEs via maximum likelihood is difficult due to intractable integrals.
method Introduced unbiased estimators of the log-likelihood gradient using coupled Markov chains.
result Unbiased estimators lead to better predictive performance in VAEs.
Develops an algorithm for bilevel optimization with coupled constraints.
problem Challenges in bilevel optimization with coupled constraints.
method Primal-dual-assisted penalty approach and a fully first-order algorithm (BLOCC).
result Established rigorous convergence theory and demonstrated effectiveness on real-world applications.
uHMC achieves fast mixing in high dimensions with gradient evaluations.
problem Quantifying mixing time of uHMC in high dimensions.
method Construction of successful couplings for uHMC.
result uHMC mixes in total variation with logarithmic dependence on dimension.
Bayesian network approach for efficient cooperative MARL.
problem Leveraging inter-agent coupling information for scalable MARL algorithms.
method Modeling cooperative MARL via Bayesian networks, identifying value dependency sets, proposing P-DTDE paradigm.
result P-DTDE policy gradient estimator has lower total variance than CTDE.
fSGLD optimizes deep learning by favoring flat regions in the loss landscape.
problem Understanding and improving the behavior and generalization of deep learning algorithms.
method Flatness-Aware Stochastic Gradient Langevin Dynamics (fSGLD) that biases learning towards flat basins.
result fSGLD targets a flatness-biased Gibbs distribution with explicit excess risk guarantees.
Graph-Coupled Oscillator Networks (GraphCON) tackles graph-based learning problems.
problem The oversmoothing problem in Graph Neural Networks (GNNs).
method GraphCON is a novel framework based on discretizations of ODEs modeling oscillators coupled via graph adjacency.
result GraphCON mitigates the oversmoothing problem and exploding/vanishing gradients issues.
New Langevin dynamics samples from entropy-regularized optimal transport.
problem Sampling from entropy-regularized optimal transport.
method Introduced analogous diffusion dynamics constrained to .
result Long-time limit is the unique solution of an entropic optimal transport problem.
The paper establishes a series of gradient estimates for positive solutions to the heat equation on a manifold evolving under the Ricci flow, coupled with the harmonic map flow between and a second manifold . We prove Li-Yau type Harnack inequalities and we consider the cases when is a complete manifold …
Gradient descent forces neural network eigenvalues to a specific threshold.
problem Understanding why gradient descent drives eigenvalues to a specific threshold.
method Introduced edge coupling, a functional on consecutive iterate pairs, to explain the trajectory towards the eigenvalue threshold.
result Gradient descent forces the Hessian eigenvalue to the threshold from arbitrary initialization.
Artifical Neural Networks are a particular class of learning systems modeled after biological neural functions with an interesting penchant for Hebbian learning, that is "neurons that wire together, fire together". However, unlike their natural counterparts, artificial neural networks have a close and stringent couplin…
Develop a framework for barycentric projections of optimal transport plans on Riemannian manifolds.
problem Optimal transport couplings are probabilistic objects, while many learning pipelines require deterministic maps.
method Develop a framework for barycentric projections of transport couplings on Riemannian manifolds.
result The intrinsic projection maps each source point to the conditional Fréchet mean of its destination law and is shown to be the best deterministic representative under squared geodesic loss.
Combining SchNet and SHARC: The SchNarc machine learning approach for excited-state dynamicsphysics.chem-ph
SchNarc combines SchNet and SHARC for efficient photodynamics simulations.
problem Efficiently simulate excited-state dynamics of complex molecules.
method Combines SchNet for multiple electronic states with SHARC for molecular dynamics, learning energies, forces, and couplings.
result Paves the way for efficient photodynamics simulations of complex systems.
Laprop separates Adam's momentum and adaptivity to improve stability and speed.
problem Unnecessary coupling between Adam's momentum and adaptivity leads to instability and divergence.
method Proposes Laprop, a method that decouples momentum and adaptivity.
result Laprop consistently improves speed and stability over Adam on various tasks.
A new method estimates protein evolutionary fields and couplings from alignments.
problem Estimating evolutionary fields and couplings from protein sequence alignments.
method Boltzmann machine with parallel, persistent Markov chain Monte Carlo method.
result Improved precision in predicting contact residue pairs.
We present a convergence rate analysis for biased stochastic gradient descent (SGD), where individual gradient updates are corrupted by computation errors. We develop stochastic quadratic constraints to formulate a small linear matrix inequality (LMI) whose feasible points lead to convergence bounds of biased SGD. Base…
By using a coupling method, an explicit log-Harnack inequality with local geometry quantities is established for (sub-Markovian) diffusion semigroups on a Riemannian manifold (possibly with boundary). This inequality as well as the consequent -gradient inequality, are proved to be equivalent to the pointwise curva…
Mini-batch stochastic gradient descent and variants thereof have become standard for large-scale empirical risk minimization like the training of neural networks. These methods are usually used with a constant batch size chosen by simple empirical inspection. The batch size significantly influences the behavior of the …
The inverse Potts problem to infer a Boltzmann distribution for homologous protein sequences from their single-site and pairwise amino acid frequencies recently attracts a great deal of attention in the studies of protein structure and evolution. We study regularization and learning methods and how to tune regularizati…
The paper is related to the classification of special manifolds and projective special manifolds. One of the result of this paper is that, if the Weil-Petersson metric on a projective special manifold is complete, then the Hodge metrc and the Weil-Petersson metrc are equivalent.
Equations link metrics with tensors, revealing curvature constraints.
problem Understanding curvature properties of geometric structures.
method Formal analogies to Einstein-Maxwell equations, studying Codazzi and conformal Killing equations.
result Constraints on scalar curvature of metrics in solutions.
Tseytlin has recently proposed that an action functional exists whose gradient generates to all orders in perturbation theory the Renormalization Group (RG) flow of the target space metric in the worldsheet sigma model. The gradient is defined with respect to a metric on the space of coupling constants which is explici…
Conditions for statistical structures on manifolds derived from solitons.
problem Characterizing statistical structures on manifolds from soliton equations.
method Analyzing gradient solitons on statistical manifolds to derive conditions for statistical structures.
result Established necessary and sufficient conditions for statistical structures under various soliton types.
We study the gradient flow of the Yang--Mills functional on the space of connection 1-forms on a principal -bundle over the sphere from the perspective of Morse theory. The resulting Morse homology is compared to the heat flow homology of the space of based loops in the compact Lie group . An iso…
This work presents a partitioned solution procedure to compute shape gradients in fluid-structure interaction (FSI) using black-box adjoint solvers. Special attention is paid to project the gradients onto the undeformed configuration. This is due to the mixed Lagrangian-Eulerian formulation of large-displacement FSI in…
Structured Nonconvex and Nonsmooth Optimization: Algorithms and Iteration Complexity Analysismath.OC
Nonconvex and nonsmooth optimization problems are frequently encountered in much of statistics, business, science and engineering, but they are not yet widely recognized as a technology in the sense of scalability. A reason for this relatively low degree of popularity is the lack of a well developed system of theory an…