TERA method speeds up derivative Gaussian processes in high dimensions.
problem High-dimensional function evaluations and gradient computations are computationally expensive.
method TERA uses exact gradient reduction to decouple n and d from the computational cost. result TERA achieves state-of-the-art predictive accuracy with orders of magnitude faster computation.
Variance reduction (VR) methods boost the performance of stochastic gradient descent (SGD) by enabling the use of larger, constant stepsizes and preserving linear convergence rates. However, current variance reduced SGD methods require either high memory usage or an exact gradient computation (using the entire dataset)…
New method improves convergence and reduces variance in noisy optimization problems.
problem Computing exact minimizers with noisy gradient information.
method Stochastic mirror descent with interacting particles.
result Interaction helps improve convergence and reduce variance.
Two reduction schemes for symplectic manifolds are shown equivalent.
problem Reduction of Hamiltonian systems on exact symplectic manifolds.
method Modified Marsden-Meyer-Weinstein reduction theorem for exact symplectic manifolds and contact manifolds.
result Reduction schemes are equivalent for exact symplectic manifolds and energy hypersurfaces.
Push-SAGA is a decentralized algorithm for directed graphs that converges linearly.
problem Finite-sum minimization over directed graphs with stochastic gradients.
method Combines variance reduction, gradient tracking, and consensus algorithms.
result Achieves linear convergence for smooth and strongly convex problems.
Some aspects of the multidimensional soliton geometry are considered. It is shown that some simples (2+1)-dimensional equations are exact reductions of the Self-Dual Yang-Mills equation or its higher hierarchy.
Improved EXACT strategy reduces GNN memory consumption and runtime.
problem Efficiently training large-scale GNNs with reduced memory usage.
method Block-wise quantization of intermediate activation maps with improved variance minimization.
result Further reduction in memory consumption (>15%) and runtime speedup (5%) with similar performance trade-offs.
This work studies the implicit bias of mini-batch SGD in classification.
problem Understanding the implicit bias of mini-batch SGD in multi-class classification.
method Characterizes how batch size, momentum, and variance reduction affect convergence and max-margin behavior under different norms.
result Momentum enables small-batch convergence to an approximate max-margin solution, while variance reduction recovers the exact full-batch bias.
Unified convergence analysis of alpha-SVRG under strong convexity.
problem Analyzing the convergence of alpha-SVRG in strongly convex environments.
method Unified convergence rate expression for alpha-SVRG under fixed learning rate, demonstrating faster convergence than SGD and SVRG.
result alpha-SVRG has a faster convergence rate compared to SGD and SVRG under suitable choice of alpha.
This paper discusses the exact simulation of the stock price process underlying the 3/2 model. Using a result derived by Craddock and Lennox using Lie Symmetry Analysis, we adapt the Broadie-Kaya algorithm for the simulation of affine processes to the 3/2 model. We also discuss variance reduction techniques and find th…
New method computes affine normal directions efficiently for sparse polynomials.
problem Computing affine normal directions is computationally expensive in high dimensions.
method Reduces third-order tensor contraction to matrix-free formulation using log-determinant gradient.
result Scalable implementations with near-linear scaling in dimension and sparsity.
We study control systems invariant under a Lie group with application to the problem of nonlinear trajectory planning. A theory of symmetry reduction of exterior differential systems is employed to demonstrate how symmetry reduction and reconstruction is effective in the explicit, exact construction of planned system t…
We reduce variance in Bures-Wasserstein variational inference.
problem High variance in Monte Carlo approximations of Bures-Wasserstein gradients.
method Control variates to reduce variance in the forward step.
result Proposed estimator reduces variance by orders of magnitude.
Policy gradient methods are very attractive in reinforcement learning due to their model-free nature and convergence guarantees. These methods, however, suffer from high variance in gradient estimation, resulting in poor sample efficiency. To mitigate this issue, a number of variance-reduction approaches have been prop…
Highly expressive directed latent variable models, such as sigmoid belief networks, are difficult to train on large datasets because exact inference in them is intractable and none of the approximate inference methods that have been applied to them scale well. We propose a fast non-iterative approximate inference metho…
Abstract: Generalized reduction methods for symmetries in graded geometry.
problem Generalized reduction of symmetries in graded geometry.
method Graded symplectic reduction for Courant, Dirac, and generalized complex structures.
result Systematic recovery of reduction schemes for exact cases.
Optimization geometrodynamics simplifies adaptive optimizer dynamics.
problem Hidden states in adaptive optimizers complicate gradient-based learning.
method Develops a variational theory to eliminate hidden states and compose across hierarchies.
result Yields interaction curvature that integrates to finite contrasts.
New sparsity operator reduces variance reduction methods' computational cost.
problem Reduce computational cost of variance reduction methods.
method Introduce random-top-k operator to estimate gradient sparsity and reduce operations per update.
result Our algorithm consistently outperforms SpiderBoost in various tasks.
We obtain the first polynomial-time algorithm for exact tensor completion that improves over the bound implied by reduction to matrix completion. The algorithm recovers an unknown 3-tensor with r incoherent, orthogonal components in Rn from r⋅O~(n1.5) randomly observed entries of the tensor…
We study a class of nonlinear pricing models which involves the feedback effect from the dynamic hedging strategies on the price of asset introduced by Sircar and Papanicolaou. We are first to study the case of a nonlinear demand function involved in the model. Using a Lie group analysis we investigate the symmetry pro…
Families of exact solutions are found to a nonlinear modification of the Black-Scholes equation. This risk-adjusted pricing methodology model (RAPM) incorporates both transaction costs and the risk from a volatile portfolio. Using the Lie group analysis we obtain the Lie algebra admitted by the RAPM equation. It gives …
Unified model for reducing dimensions and clustering high-dimensional data.
problem High-dimensional data clustering and dimensionality reduction.
method Hierarchical mixtures of Gaussians (HMoGs) with closed-form likelihood and inference.
result Efficiently models hundreds of latent dimensions, improving clustering performance.
Gradient descent achieves exact linear convergence rate for symmetric matrix completion.
problem Low-rank symmetric matrix completion using gradient descent.
method Local analysis of gradient descent for symmetric matrices without additional assumptions.
result Closed-form expression of exact linear convergence rate matches practice.
In this paper, we investigate the non-linear Black--Scholes equation: ut+ax2uxx+bx3uxx2+c(xux−u)=0,a,b>0, c≥0. and show that the one can be reduced to the equation ut+(uxx+ux)2=0 by an appropriate point transformation of variables. For the resulting equation, we study the group-theore…
New method extends invariant reduction to rescaled geometric structures.
problem Computing invariant geometric structures under symmetries.
method Extends invariant reduction to rescaled structures using shift rule.
result Emergence and loss of invariance in reductions.
This work proposes using zero-variance control variates to reduce variance in pathwise gradient estimators for variational inference.
problem Pathwise gradient estimators in variational inference have high variance, leading to inefficient optimization.
method Apply zero-variance control variates to pathwise gradient estimators.
result Zero-variance control variates can significantly reduce the variance of pathwise gradient estimators without requiring complex assumptions.
A new method reduces the complexity of decentralized optimization.
problem Decentralized stochastic non-convex optimization over a network.
method GT-HSGD, a hybrid variance-reduced method.
result Achieves an oracle complexity of O(n^(-1)ε^(-3)) for small ε.
We propose Kernel Hamiltonian Monte Carlo (KMC), a gradient-free adaptive MCMC algorithm based on Hamiltonian Monte Carlo (HMC). On target densities where classical HMC is not an option due to intractable gradients, KMC adaptively learns the target's gradient structure by fitting an exponential family model in a Reprod…
Improves gradient estimation for discrete distributions with variance reduction techniques.
problem Excessive variance in gradient estimation for discrete distributions.
method Stein operators for discrete distributions and control variates.
result Substantially lower variance in gradient estimation.
Taking advantage of the recent litterature on exact simulation algorithms (Beskos, Papaspiliopoulos and Roberts) and unbiased estimation of the expectation of certain fonctional integrals (Wagner, Beskos et al. and Fearnhead et al.), we apply an exact simulation based technique for pricing continuous arithmetic average…
RevDEQs improve performance on tasks with exact gradients and fewer function evaluations.
problem Inexact gradient calculation in DEQs leads to unstable training and requires regularisation or many function evaluations.
method Introduce Reversible Deep Equilibrium Models (RevDEQs) that allow for exact gradient calculation, no regularisation, and far fewer function evaluations.
result RevDEQs significantly improve performance on language modelling and image classification tasks.
Proves new inequality linking spectral numbers of Lagrangians and their reductions.
problem Understanding spectral properties of Lagrangian submanifolds.
method Develops inverse reduction inequalities for spectral numbers.
result Proof of inequality between spectral numbers of Lagrangian and its reductions.
We construct symplectic and Kähler ray reduced spaces and discuss their relation with the Marsden-Weinstein (point) reduction. This Kähler reduction is well defined even when the momentum value is not totally isotropic. The compatibility of the ray reduction with the cone construction and the Boothby-Wang fibration is …
We discuss Levi-Civita connections on Courant algebroids. We define an appropriate generalization of the curvature tensor and compute the corresponding scalar curvatures in the exact and heterotic case, leading to generalized (bosonic) Einstein-Hilbert type of actions known from supergravity. In particular, we carefull…
Torsion found in knot homology, challenging augmentation theories.
problem Torsion in linearized contact homology for Legendrian knots.
method Examples of Legendrian knots with non-trivial homology over Z.
result Augmentations not induced by exact fillings, even mod 2.
New method improves scalability of SGD for large datasets.
problem High variance in stochastic gradient descent.
method Adaptive measure reduction with Carathéodory's theorem.
result Improved scalability to high-dimensional spaces.
Proposes variance reduction for optimizing permutation models.
problem High variance in gradient estimates for discrete latent variables.
method Control variates for the Plackett-Luce distribution.
result Optimization of black-box functions over permutations using SGD.
This paper explores the computational hardness of generating latent vectors for generative models.
problem Computational hardness of generating latent vectors for generative models.
method Established lower bounds for exact and approximate model inversion under strong exponential time hypothesis (SETH) and exponential time hypothesis (ETH).
result Lower bounds for computational complexity of exact and approximate model inversion.
TrIM improves gradient-based dimension reduction and regression.
problem Efficiently identifying relevant feature subspace for high-dimensional regression.
method Introduced TrIM forest, an iterative approach using Mondrian forest and EGOP estimate.
result Consistency guarantees and convergence rates for EGOP matrix and random forest estimator.
Identifies a gradient flow to solve kernel learning problems with noise reduction.
problem Kernel learning problem with Gaussian noise.
method Riemannian gradient flow with continuous Lyapunov functionals.
result Flow reduces noise and finds stationary points.
Gradient descent slows significantly in over-parameterized single neuron learning.
problem Learning a single neuron with over-parameterization and square loss.
method Analysis of gradient descent dynamics, proving convergence rates and lower bounds.
result Over-parameterization can exponentially slow down the convergence rate of gradient descent.
Unified framework for stable RL learning with theoretical guarantees.
problem Lack of systematic theoretical principles guiding RL post-training methods.
method Unified theoretical framework for policy-gradient estimators and optimization algorithms.
result Establishes unbiasedness, variance expressions, and convergence guarantees.
Extends dimension reduction to data-driven settings without gradients.
problem Gradient-based dimension reduction limitations in data-driven settings.
method Score ratio matching framework, tailored parameterization, regularization, eigenvalue deflation.
result Outperforms standard score-matching for problems with low-dimensional structure.
Variable selection and dimension reduction are two commonly adopted approaches for high-dimensional data analysis, but have traditionally been treated separately. Here we propose an integrated approach, called sparse gradient learning (SGL), for variable selection and dimension reduction via learning the gradients of t…
We show that certain submanifolds of generalized complex manifolds ("weak branes") admit a natural quotient which inherits a generalized complex structure. This is analog to quotienting coisotropic submanifolds of symplectic manifolds. In particular Gualtieri's generalized complex submanifolds ("branes") quotient to sp…
We generalize stochastic smoothing for gradient estimation of non-differentiable functions.
problem Gradient estimation for non-differentiable functions.
method Developed a general framework for relaxation and gradient estimation of non-differentiable black-box functions using stochastic smoothing with reduced assumptions.
result Empirically validated the effectiveness of variance reduction strategies for various non-differentiable tasks.
Evolution Strategies (ES) are a powerful class of blackbox optimization techniques that recently became a competitive alternative to state-of-the-art policy gradient (PG) algorithms for reinforcement learning (RL). We propose a new method for improving accuracy of the ES algorithms, that as opposed to recent approaches…
Paper tackles gradient-free minimax optimization with variance reduction for faster convergence.
problem Gradient-free minimax optimization problems in machine learning.
method Variance reduction technique to design a novel zeroth-order gradient descent ascent algorithm.
result Achieves the best known query complexity of O(κ(d₁ + d₂)ε⁻³), outperforming previous methods.