This paper analyzes minibatch optimal transport distances and their applications.
problem Optimal transport distances are complex and impractical for large datasets.
method Extended analysis of minibatch optimal transport distances, focusing on various kernels and debiased functions.
result Minibatch optimal transport distances are unbiased estimators and have statistical and optimisation properties.
Unbalanced minibatch Optimal Transport improves domain adaptation performance.
problem Optimal transport distances are computationally expensive for large datasets.
method Use unbalanced minibatch Optimal Transport to estimate distances over subsets of data.
result Unbalanced Optimal Transport leads to better domain adaptation results.
New method reduces discrete flow transitions, improving perplexity estimation.
problem Stochasticity in discrete paths makes rectification strategies ineffective.
method Dynamic-optimal-transport-like minimization objective with minibatch strategies.
result 32 times reduction in transitions for same perplexity.
Optimal transport distances are powerful tools to compare probability distributions and have found many applications in machine learning. Yet their algorithmic complexity prevents their direct use on large scale datasets. To overcome this challenge, practitioners compute these distances on minibatches {\em i.e.} they a…
A mesh-free method solves continuum-marginal optimal transport problems.
problem Recovering minimum-energy velocity fields from time-continuous probability marginals.
method Embeds weak continuity equation in a reproducing kernel Hilbert space, optimizing with mini-batch stochastic methods.
result Accurately recovers drift and maintains marginal consistency in synthetic experiments.
A new method reformulates Optimal Transport Conditional Flow Matching using proximal operators.
problem Optimal Transport Conditional Flow Matching (OT-CFM) for generating models.
method Reformulate OT-CFM using proximal operators and extended Brenier potential.
result OT-CFM dynamics are terminally normally hyperbolic for manifold-supported targets.
Local SGD outperforms minibatch SGD for quadratic objectives.
problem Theoretical foundations of local SGD are lacking.
method Proved local SGD strictly dominates minibatch SGD for quadratic objectives and accelerated local SGD is minimax optimal.
result Local SGD does not dominate minibatch SGD in general convex objectives.
A new method learns straight trajectories in one step for optimal flow matching.
problem Learning flows with straight trajectories for fast inference.
method Optimal Flow Matching (OFM) approach using convex functions for vector fields.
result Recovering straight OT displacements in just one FM step for quadratic transport.
Minibatch SGD outperforms Local SGD in heterogeneous distributed learning.
problem Optimizing a combined convex objective with stochastic gradient estimates from different machines.
method Analysis of Minibatch SGD and Local SGD in a heterogeneous distributed setting.
result Minibatch SGD dominates Local SGD in the heterogeneous distributed setting.
Exact minibatch MH method improves scalability for large datasets.
problem Inexactness in minibatch MH methods causes inference errors.
method TunaMH proposes an exact minibatch MH method with a tunable batch size.
result TunaMH is asymptotically optimal in terms of batch size.
Develops minibatch stochastic proximal gradient for large-scale learning models.
problem Finding optimal predictors with complex regularizers in large-scale learning models.
method Minibatch variants of stochastic proximal gradient algorithm for composite objective functions.
result Minibatch size N N N after O ( 1 N ε ) \mathcal{O}(\frac{1}{Nε}) O ( N ε 1 ) iterations achieves ε − ε- ε − suboptimality in expected quadratic distance. Anytime MiniBatch speeds up online distributed optimization by handling slow nodes.
problem Mitigating the impact of slow nodes (stragglers) in distributed optimization.
method Proposes an online distributed optimization method that averages minibatch gradients via consensus rounds.
result Prevents stragglers from slowing progress without wasting work.
New sampling technique improves KGC model performance.
problem Ignoring entity neighbors in minibatches affects KGC model training.
method Random-walk based minibatch sampling.
result Proposed method achieves state-of-the-art performance on DB100K.
QDSB accelerates Schrödinger bridge learning with quantized approximations.
problem Learning generative models from unpaired samples.
method Quantized diffusion Schrödinger bridges (QDSB) using anchor-quantized distributions and cell-wise sampling.
result QDSB achieves sample quality similar to existing methods but with significantly less computational time.
New method shows stochastic momentum can converge quickly on optimization problems.
problem Improving convergence of stochastic optimization methods.
method Stochastic heavy ball momentum with minibatching.
result Stochastic heavy ball momentum retains fast linear rate on quadratic problems.
New algorithms accelerate model-based optimization for stochastic problems.
problem Optimizing model-based stochastic optimization problems efficiently.
method Proposed new model-based algorithms with acceleration and minibatch techniques.
result Non-asymptotic convergence guarantees with linear speedup in minibatch size.
Machine learning, especially deep neural networks, has been rapidly developed in fields including computer vision, speech recognition and reinforcement learning. Although Mini-batch SGD is one of the most popular stochastic optimization methods in training deep networks, it shows a slow convergence rate due to the larg…
We present novel minibatch stochastic optimization methods for empirical risk minimization problems, the methods efficiently leverage variance reduced first-order and sub-sampled higher-order information to accelerate the convergence speed. For quadratic objectives, we prove improved iteration complexity over state-of-…
Unified analysis of stochastic gradient methods for convex and smooth optimization.
problem Minimizing composite convex and smooth functions.
method Unified convergence analysis of various stochastic gradient methods.
result Unified convergence rates for a variety of methods including proximal SGD, variance reduced methods, quantization, and coordinate descent.
FedProx algorithm improved for non-smooth and heterogeneous data.
problem Theoretical understanding of FedProx for non-convex federated optimization.
method Local dissimilarity invariant convergence theory through algorithmic stability.
result Convergence guarantees for non-smooth FL problems and minibatch size.
It is well known that, for most datasets, the use of large-size minibatches for Stochastic Gradient Descent (SGD) typically leads to slow convergence and poor generalization. On the other hand, large minibatches are of great practical interest as they allow for a better exploitation of modern GPUs. Previous literature …
Gibbs sampling is a Markov chain Monte Carlo method that is often used for learning and inference on graphical models. Minibatching, in which a small random subset of the graph is used at each iteration, can help make Gibbs sampling scale to large graphical models by reducing its computational cost. In this paper, we p…
Stochastic Gradient Descent (SGD) is a popular optimization method which has been applied to many important machine learning tasks such as Support Vector Machines and Deep Neural Networks. In order to parallelize SGD, minibatch training is often employed. The standard approach is to uniformly sample a minibatch at each…
SGD converges to global minimum for structured non-convex functions.
problem Optimizing non-convex functions using SGD with slow convergence rates.
method Convergence theorems for SGD on structured non-convex functions, including Quasar and PL conditions.
result SGD converges to global minimum for specific non-convex functions under certain conditions.
A new method for distributed optimization reduces communication rounds without minibatches.
problem Efficient training in distributed machine learning with different data distributions.
method A primal-dual method (GA-MSGD) applied to the Lagrangian of distributed optimization.
result Achieves linear convergence in communication rounds for strongly convex objectives.
New method improves convergence of SPP for convex optimization problems.
problem Stochastic optimization and robustness to SGD.
method Minibatch Stochastic Proximal Point (M-SPP) method with stability analysis.
result M-SPP achieves faster convergence rates under smoothness and quadratic growth conditions.
This work improves SGD minibatch sampling using determinantal point processes based on orthogonal polynomials.
problem Improving variance reduction in stochastic gradient descent (SGD) for large datasets.
method Orthogonal polynomial-based determinantal point processes for sampling minibatches in SGD.
result DPP minibatches lead to a smaller mean square approximation error than uniform minibatches.
DP-PCA improves privacy in PCA computations with optimal statistical error.
problem Differentially private principal component analysis with sub-linear sample complexity.
method Private minibatch gradient ascent with private mean estimation.
result Achieves optimal statistical error rates for sub-Gaussian data with n = i l d e O ( d ) n= ilde O(d) n = i l d e O ( d ) samples. New approach speeds up optimization with repeated gradient steps on same batch.
problem Performance bottlenecks in massive parallel pipelines with large batch sizes.
method Data echoing, taking repeated gradient steps on the same batch.
result Data echoing affords speedups on curvature-dominated part of convergence rate.
Memory-efficient optimizers fail to track a subspace, leading to unpredictable model performance.
problem Memory-efficient optimizers fail to track a subspace, leading to unpredictable model performance.
method Analyzing the behavior of memory-efficient optimizers like GaLore, which project gradients onto a rank-r subspace recomputed every T steps.
result Memory-efficient optimizers fail to track a subspace, leading to unpredictable model performance.
CD algorithm achieves near-optimal convergence rate for unnormalized models.
problem Training unnormalized models with high efficiency.
method Non-asymptotic analysis of contrastive divergence algorithm.
result CD can achieve O ( n − 1 / 2 ) O(n^{-1 / 2}) O ( n − 1/2 ) convergence rate under regularity assumptions. Study on gradient complexity of private optimization with private oracles.
problem Analyzing the efficiency of differentially private optimization algorithms.
method Lower bounds on the number of first-order oracle queries for private optimization.
result Lower bounds on the number of queries for private optimization algorithms, showing a dimension-dependent runtime penalty.
New convergence bounds for shuffling-based SGD methods in distributed learning.
problem Analyzing the performance of shuffling-based variants of SGD in distributed learning.
method Study of minibatch and local Random Reshuffling methods, proving convergence bounds and lower bounds.
result Shuffling-based variants converge faster than with-replacement sampling methods, and the bounds are tight.
The Lookahead optimizer improves SGD's performance and generalization without restrictive assumptions.
problem Improving the generalization of SGD with Lookahead.
method A rigorous stability and generalization analysis of the Lookahead optimizer with minibatch SGD, leveraging on-average model stability.
result Derives generalization bounds for convex and strongly convex problems without the restrictive Lipschitzness assumption, demonstrating a linear speedup with batch size.
EigenGame improves eigendecomposition by offering unbiased updates for larger datasets.
problem Minibatch bias in EigenGame limits convergence and parallelism.
method Proposed unbiased stochastic update for EigenGame.
result Asymptotic equivalence to EigenGame, greater parallelism, and improved performance.
Although stochastic gradient descent (SGD) is a driving force behind the recent success of deep learning, our understanding of its dynamics in a high-dimensional parameter space is limited. In recent years, some researchers have used the stochasticity of minibatch gradients, or the signal-to-noise ratio, to better char…
We analyze stochastic algorithms for optimizing nonconvex, nonsmooth finite-sum problems, where the nonconvex part is smooth and the nonsmooth part is convex. Surprisingly, unlike the smooth case, our knowledge of this fundamental problem is very limited. For example, it is not known whether the proximal stochastic gra…
We present an approach towards convex optimization that relies on a novel scheme which converts online adaptive algorithms into offline methods. In the offline optimization setting, our derived methods are shown to obtain favourable adaptive guarantees which depend on the harmonic sum of the queried gradients. We furth…
Adaptive batch sizes improve local gradient methods in distributed training.
problem Communication bottlenecks in distributed deep learning.
method Adaptive batch size strategies for local gradient methods.
result Adaptive batch sizes reduce minibatch gradient variance and improve training efficiency.
New DPP kernels improve minibatch efficiency for large datasets.
problem Efficiently generating minibatches from large datasets.
method Wavelet-based DPPs and novel discrete conversion methods.
result Improved minibatch efficiency with low-rank decompositions.
SGD efficiently learns the XOR function with near-optimal sample complexity.
problem Learning the XOR function with a 2-layer neural network.
method Minibatch SGD on a 2-layer neural network with ReLU activations, focusing on signal-finding and signal-heavy phases.
result Achieves population error o ( 1 ) o(1) o ( 1 ) with d e x t p o l y l o g ( d ) d \: ext{polylog}(d) d e x t p o l y l o g ( d ) samples. Study on the noise in SGD minibatches near local minima.
problem Understanding the noise in SGD minibatches near local minima.
method Detailed analysis of SGD noise in linear regression and derivation of a general formula for different types of minima.
result Provides insight into the stability of training neural networks and suggests large learning rates can help generalization.
New minibatching strategy reduces stochastic gradient bias in optimisation.
problem Reducing bias in stochastic gradient descent.
method Symmetric Minibatching Strategy combined with momentum.
result Reduced stochastic gradient bias from O ( h 2 ) \mathcal{O}(h^2) O ( h 2 ) to O ( h 4 ) \mathcal{O}(h^4) O ( h 4 ) . Minibatching is a very well studied and highly popular technique in supervised learning, used by practitioners due to its ability to accelerate training through better utilization of parallel processing power and reduction of stochastic variance. Another popular technique is importance sampling -- a strategy for prefer…
Optimal algorithm for LQR control with improved regret bound.
problem Nonstochastic control with quadratic losses (LQR control).
method Online algorithm with optimal dynamic regret of i l d e O ( e x t m a x { n 1 / 3 T V ( M 1 : n ) 2 / 3 , 1 } ) ilde{O}( ext{max}\{n^{1/3} \mathcal{TV}(M_{1:n})^{2/3}, 1\}) i l d e O ( e x t ma x { n 1/3 TV ( M 1 : n ) 2/3 , 1 }) . result Improves the best known rate of i l d e O ( n ( T V ( M 1 : n ) + 1 ) ) ilde{O}(\sqrt{n (\mathcal{TV}(M_{1:n})+1)} ) i l d e O ( n ( TV ( M 1 : n ) + 1 ) ) for general convex losses. New methods estimate transport-growth pairs in unbalanced optimal transport.
problem Statistical guarantees for Monge-type estimation in unbalanced optimal transport remain limited.
method Developed two estimators for transport-growth pairs under different setups.
result Achieved minimax optimal rate for estimation of transport-growth pairs.
Introduces statistical optimal transport for probabilistic lectures.
problem No specific problem stated; focuses on introduction.
method Lecture-based introduction to statistical optimal transport.
result Provides an introduction to statistical optimal transport.
Paper proves minibatch SGD for GP inference converges and improves generalization.
problem Theoretical understanding and practical use of SGD for correlated samples in Gaussian process inference.
method Proves minibatch SGD converges to a critical point with rate O(1/K) for K iterations, under certain kernel conditions.
result Minibatch SGD for GP inference improves generalization and reduces computational burden.