This paper analyzes minibatch optimal transport distances and their applications.
problem Optimal transport distances are complex and impractical for large datasets.
method Extended analysis of minibatch optimal transport distances, focusing on various kernels and debiased functions.
result Minibatch optimal transport distances are unbiased estimators and have statistical and optimisation properties.
New technique trains deep neural networks without normalization or minibatch statistics.
problem Training deep neural networks at high learning rates without normalization.
method Channel-wise zero-mean initialization and gradient modification to maintain common mode rejection.
result Achieves higher accuracy compared to batch normalization and shows minibatches are unnecessary.
Develops minibatch stochastic proximal gradient for large-scale learning models.
problem Finding optimal predictors with complex regularizers in large-scale learning models.
method Minibatch variants of stochastic proximal gradient algorithm for composite objective functions.
result Minibatch size N after O(Nε1) iterations achieves ε−suboptimality in expected quadratic distance. Researchers analyze SGD dynamics using von Mises-Fisher distributions.
problem Understanding the dynamics of stochastic gradient descent in high-dimensional spaces.
method Geometric analysis of minibatch gradient norms and directions through von Mises-Fisher distribution.
result Directional uniformity of minibatch gradients increases over SGD iterations.
This work improves SGD minibatch sampling using determinantal point processes based on orthogonal polynomials.
problem Improving variance reduction in stochastic gradient descent (SGD) for large datasets.
method Orthogonal polynomial-based determinantal point processes for sampling minibatches in SGD.
result DPP minibatches lead to a smaller mean square approximation error than uniform minibatches.
DP-PCA improves privacy in PCA computations with optimal statistical error.
problem Differentially private principal component analysis with sub-linear sample complexity.
method Private minibatch gradient ascent with private mean estimation.
result Achieves optimal statistical error rates for sub-Gaussian data with n=ildeO(d) samples. New algorithms accelerate model-based optimization for stochastic problems.
problem Optimizing model-based stochastic optimization problems efficiently.
method Proposed new model-based algorithms with acceleration and minibatch techniques.
result Non-asymptotic convergence guarantees with linear speedup in minibatch size.
Paper proves minibatch SGD for GP inference converges and improves generalization.
problem Theoretical understanding and practical use of SGD for correlated samples in Gaussian process inference.
method Proves minibatch SGD converges to a critical point with rate O(1/K) for K iterations, under certain kernel conditions.
result Minibatch SGD for GP inference improves generalization and reduces computational burden.
A new approach speeds up SGD training with large minibatches.
problem Slow convergence and poor generalization with large minibatches.
method Minibatch persistency, reusing the same minibatch for K consecutive iterations.
result Small persistency values (K=2 or 5) lead to faster convergence and comparable/generalization.
Local SGD outperforms minibatch SGD for quadratic objectives.
problem Theoretical foundations of local SGD are lacking.
method Proved local SGD strictly dominates minibatch SGD for quadratic objectives and accelerated local SGD is minimax optimal.
result Local SGD does not dominate minibatch SGD in general convex objectives.
Exact minibatch MH method improves scalability for large datasets.
problem Inexactness in minibatch MH methods causes inference errors.
method TunaMH proposes an exact minibatch MH method with a tunable batch size.
result TunaMH is asymptotically optimal in terms of batch size.
Minibatch SGD outperforms Local SGD in heterogeneous distributed learning.
problem Optimizing a combined convex objective with stochastic gradient estimates from different machines.
method Analysis of Minibatch SGD and Local SGD in a heterogeneous distributed setting.
result Minibatch SGD dominates Local SGD in the heterogeneous distributed setting.
New method speeds up Gibbs sampling for large graphs.
problem Efficiently sampling from large graphical models.
method Poisson-minibatching Gibbs sampling.
result Theoretical convergence rate guarantees for Poisson-minibatching Gibbs.
Paper proposes a method to estimate variance reduction in DNN training using importance sampling.
problem Challenges in assessing variance reduction during DNN training using importance sampling.
method Proposes a method for estimating variance reduction using minibatches sampled under importance sampling.
result Demonstrates consistent reduction in variance, improved training efficiency, and enhanced model accuracy.
Paper introduces ZOO-ADMM for online optimization with reduced gradient calculations.
problem Developing an efficient online optimization method for complex structured regularizers.
method Zeroth-order online alternating direction method of multipliers (ZOO-ADMM) with gradient-free operation and minibatch strategies.
result Improved convergence rate for ZOO-ADMM compared to first-order gradient-based methods.
New sampling technique improves KGC model performance.
problem Ignoring entity neighbors in minibatches affects KGC model training.
method Random-walk based minibatch sampling.
result Proposed method achieves state-of-the-art performance on DB100K.
This paper improves Minibatch SGD convergence through typicality sampling.
problem Slow convergence of Minibatch SGD due to large gradient noise.
method Typicality sampling for more efficient batch selection.
result Typical batch SGD outperforms conventional Minibatch SGD in convergence.
Differentiable learning via SGD and GD can simulate various learning problems, depending on precision and minibatch size.
problem Understanding the power of differentiable learning via SGD and GD compared to statistical query (SQ) learning.
method Comparing the learning power of SGD and GD on population and empirical losses with statistical query learning.
result The learning power of SGD and GD depends on the precision of gradient calculations relative to the minibatch size or sample size.
EIM improves prediction interval tightness without distributional assumptions.
problem Generating tight prediction intervals for asymmetric distributions.
method Expanded Interval Minimization (EIM) using minibatch statistics for coverage and width optimization.
result EIM produces on average 1.37x tighter prediction intervals than prior techniques.
New method converts online to offline optimization with adaptive minibatch sizes.
problem Optimizing online to offline conversions with adaptive minibatch sizes.
method A novel scheme that converts online adaptive algorithms into offline methods, with adaptive guarantees and implicit structure adaptation.
result Favourable adaptive guarantees and implicit structure adaptation in the offline optimization setting.
The paper improves Gibbs sampling for large graphs by minibatching.
problem High computational cost of single Gibbs sampling update step.
method Minibatching: subsampling factors to estimate their sum.
result Minibatched Gibbs can be made unbiased and converge faster.
Unbalanced minibatch Optimal Transport improves domain adaptation performance.
problem Optimal transport distances are computationally expensive for large datasets.
method Use unbalanced minibatch Optimal Transport to estimate distances over subsets of data.
result Unbalanced Optimal Transport leads to better domain adaptation results.
Anytime MiniBatch speeds up online distributed optimization by handling slow nodes.
problem Mitigating the impact of slow nodes (stragglers) in distributed optimization.
method Proposes an online distributed optimization method that averages minibatch gradients via consensus rounds.
result Prevents stragglers from slowing progress without wasting work.
New approach speeds up optimization with repeated gradient steps on same batch.
problem Performance bottlenecks in massive parallel pipelines with large batch sizes.
method Data echoing, taking repeated gradient steps on the same batch.
result Data echoing affords speedups on curvature-dominated part of convergence rate.
New DPP kernels improve minibatch efficiency for large datasets.
problem Efficiently generating minibatches from large datasets.
method Wavelet-based DPPs and novel discrete conversion methods.
result Improved minibatch efficiency with low-rank decompositions.
New optimization method speeds up learning from data.
problem Efficiently optimizing large datasets for machine learning.
method Minibatch stochastic variance reduced proximal iterations.
result Improved convergence speed for quadratic objectives.
Paper examines the structure of stochastic gradients in deep learning.
problem Exploring the structure and heavy tails of stochastic gradients in deep learning.
method Conducted formal statistical tests on stochastic gradients and gradient noise.
result Stochastic gradients and gradient noise do not exhibit power-law heavy tails, but their covariance spectra do.
New method shows stochastic momentum can converge quickly on optimization problems.
problem Improving convergence of stochastic optimization methods.
method Stochastic heavy ball momentum with minibatching.
result Stochastic heavy ball momentum retains fast linear rate on quadratic problems.
Paper analyzes minibatch Wasserstein for machine learning applications.
problem Optimal transport distances on large datasets.
method Analysis of minibatch optimal transport.
result Minibatch Wasserstein is equivalent to implicit regularization with desirable properties.
Stochastic NGD approximates Bayesian posterior samples near local minima.
problem Approximating Bayesian uncertainty in model parameters near local minima.
method Develops minibatch natural gradient descent (NGD) and introduces stochastic NGD to preserve Bayesian properties.
result Minibatch NGD's stationary distribution approaches a Bayesian posterior near local minima with small learning rates.
Paper explores how combining tail-averaging and minibatching improves SGD convergence.
problem Understanding and optimizing learning properties of SGD variants.
method Least squares learning in a nonparametric setting, focusing on multiple passes, mini-batching, and averaging.
result Tail averaging allows faster convergence rates than uniform averaging in nonparametric settings.
Study on the noise in SGD minibatches near local minima.
problem Understanding the noise in SGD minibatches near local minima.
method Detailed analysis of SGD noise in linear regression and derivation of a general formula for different types of minima.
result Provides insight into the stability of training neural networks and suggests large learning rates can help generalization.
New minibatching strategy reduces stochastic gradient bias in optimisation.
problem Reducing bias in stochastic gradient descent.
method Symmetric Minibatching Strategy combined with momentum.
result Reduced stochastic gradient bias from O(h2) to O(h4). Minibatching is a very well studied and highly popular technique in supervised learning, used by practitioners due to its ability to accelerate training through better utilization of parallel processing power and reduction of stochastic variance. Another popular technique is importance sampling -- a strategy for prefer…
Study shows DQN's performance degrades with temporal dependence in data.
problem Temporal dependence in replayed data affects DQN's performance.
method Modelled τ-mixing data, derived risk bounds, and empirical validation. result Temporal dependence leads to a degradation in DQN's performance rate.
Stochastic Gradient Descent (SGD) is a popular optimization method which has been applied to many important machine learning tasks such as Support Vector Machines and Deep Neural Networks. In order to parallelize SGD, minibatch training is often employed. The standard approach is to uniformly sample a minibatch at each…
New method reduces discrete flow transitions, improving perplexity estimation.
problem Stochasticity in discrete paths makes rectification strategies ineffective.
method Dynamic-optimal-transport-like minimization objective with minibatch strategies.
result 32 times reduction in transitions for same perplexity.
New algorithm samples Bayesian neural networks for improved calibration.
problem Improving calibration of Bayesian neural networks.
method Symmetric Minibatch Splitting-UBU (SMS-UBU) algorithm.
result SMS-UBU provides better calibration performance than standard methods.
The paper analyzes how larger minibatch sizes in SG-MCMC lead to faster convergence.
problem Theoretical analysis of impact of minibatch size on SG-MCMC convergence rate.
method Proposes a variance-reduction technique for SG-MCMC and proves its faster convergence rate.
result The proposed variance-reduction technique leads to a faster convergence rate than standard SG-MCMC.
A new Metropolis-Hastings method reduces the cost of testing for large datasets.
problem Reducing the cost of Metropolis-Hastings tests for large datasets.
method Uses small minibatches and a novel Barker acceptance test with additive correction.
result Achieves arbitrarily small batch sizes by adjusting proposal step size or temperature.
Solves GAN mode collapse by assigning minibatches to multiple discriminators.
problem Mode collapse in GANs where models generate similar samples.
method Multiple discriminators, microbatching, and changing tasks.
result Promotes sample diversity in generated sets.
Bayesian inference models power-law graphs with efficient algorithms.
problem Modeling networks with heavy-tailed degree distributions.
method Constructs graphs using BFRY random variables and applies variational Bayesian inference.
result Automatic selection of power law behavior from data.
Unified analysis of stochastic gradient methods for convex and smooth optimization.
problem Minimizing composite convex and smooth functions.
method Unified convergence analysis of various stochastic gradient methods.
result Unified convergence rates for a variety of methods including proximal SGD, variance reduced methods, quantization, and coordinate descent.
FedProx algorithm improved for non-smooth and heterogeneous data.
problem Theoretical understanding of FedProx for non-convex federated optimization.
method Local dissimilarity invariant convergence theory through algorithmic stability.
result Convergence guarantees for non-smooth FL problems and minibatch size.
Improved sampling for Bayesian neural networks reduces vanishing acceptance rates and increases predictive accuracy.
problem Sampling inefficiency in Bayesian neural networks, especially with deep architectures and large datasets.
method Approximate blocked Gibbs sampling to partition and sample subgroups of parameters.
result Increased predictive accuracy and quantification of predictive uncertainty in classification tasks.
New bounds for M-SGD show its error distribution is nearly Gaussian.
problem Understanding the error distribution of M-SGD.
method Proved non-asymptotic bounds for M-SGD in Wasserstein distance.
result Error distribution of M-SGD is approximately Gaussian.
New convergence bounds for shuffling-based SGD methods in distributed learning.
problem Analyzing the performance of shuffling-based variants of SGD in distributed learning.
method Study of minibatch and local Random Reshuffling methods, proving convergence bounds and lower bounds.
result Shuffling-based variants converge faster than with-replacement sampling methods, and the bounds are tight.
SGD converges to global minimum for structured non-convex functions.
problem Optimizing non-convex functions using SGD with slow convergence rates.
method Convergence theorems for SGD on structured non-convex functions, including Quasar and PL conditions.
result SGD converges to global minimum for specific non-convex functions under certain conditions.