A new algorithm is proposed which accelerates the mini-batch k-means algorithm of Sculley (2010) by using the distance bounding approach of Elkan (2003). We argue that, when incorporating distance bounds into a mini-batch algorithm, already used data should preferentially be reused. To this end we propose using nested …
Mini-batch EM algorithm speeds up convergence for large datasets.
problem Efficiently processing large datasets in latent variable models.
method Proposes mini-batch version of Stochastic Approximation EM algorithm for exponential models.
result Converges under classical conditions with mini-batch sampling.
Proposes m-POT to improve m-OT's misspecified mappings issue.
problem Misspecified mappings in mini-batch optimal transport.
method Partial optimal transport (POT) between mini-batch empirical measures.
result m-POT alleviates incorrect mappings compared to current methods.
DM-SGD uses DPP to diversify mini-batches for SGD, improving model performance.
problem Improving mini-batch diversity in SGD to reduce variance and improve model interpretability.
method DM-SGD uses Determinantal Point Processes (DPP) to select mini-batches with diverse data points.
result DM-SGD outperforms regular SGD and stratified sampling in various setups.
Paper proposes mini-batch ADMMs for solving nonconvex nonsmooth optimization problems.
problem Solving large-scale nonconvex nonsmooth optimization problems.
method Proposes mini-batch stochastic ADMMs for nonconvex nonsmooth optimization.
result Mini-batch stochastic ADMMs converge to a stationary point with rate O(1/T).
New method improves mini-batch efficiency for convex optimization problems.
problem Efficiently solving convex regularized empirical risk minimization with mini-batches.
method Doubly Accelerated Stochastic Variance Reduced Dual Averaging Method with double acceleration and variance reduction.
result Significantly reduces the number of mini-batches needed for optimal performance.
EE-Grad optimizes mini-batch sizes for SGD efficiency.
problem Optimizing cost-efficiency in mini-batch SGD.
method Sequential exploration and exploitation of mini-batch oracles.
result EE-Grad achieves optimal cost-efficiency with performance guarantees.
Small batch training improves deep neural network performance and stability.
problem Improving deep neural network performance and stability with limited computational resources.
method Experimental comparison of test performance for different mini-batch sizes, focusing on learning rate scaling and training duration.
result Best performance achieved for mini-batch sizes between 2 and 32, contrasting recent work advocating larger batch sizes.
New streaming methods improve convergence rates for optimization problems.
problem Optimizing large-scale, sequential data problems.
method Time-varying mini-batches and Polyak-Ruppert averaging for gradient-based algorithms.
result Time-varying mini-batches and averaging achieve optimal convergence and variance reduction.
A new method for mini-batch optimal transport improves scalability and accuracy.
problem Desired estimation and proper metric approximation in m-OT.
method BoMb-OT: Finds optimal coupling between mini-batches.
result BoMb-OT approximates a proper metric and improves m-OT's performance.
This work analyzes mini-batch inference limits and introduces Mini-AMP for clustering.
problem Analyzing information-theoretic limits of mini-batch inference in statistical learning.
method Detailed analysis of Mini-AMP for generalized linear models and low-rank matrix factorization.
result Characterization of optimal performance and phase transitions for mini-batch inference.
Federated multi-mini-batch improves efficiency in non-IID environments.
problem Performance and communication efficiency challenges in federated learning with non-IID data.
method Introduces federated multi-mini-batch approach to balance performance and communication.
result Federated multi-mini-batch outperforms federated averaging in non-IID settings.
Paper finds optimal mini-batch size for SGD to speed up learning.
problem Optimizing mini-batch size for faster SGD convergence.
method Empirical inverse law and theoretical bound on mini-batch SGD training.
result An accurate model for predicting training time and identifying implications for algorithm and hardware.
Study shows how mini-batch GD with random reshuffling affects least squares regression dynamics.
problem Analyzing the error dynamics of mini-batch GD with random reshuffling for least squares regression.
method Represented training and generalization errors through a sample cross-covariance matrix Z, compared with sample covariance matrix of original features X, and used linear scaling rule for analysis.
result Mini-batch GD with random reshuffling exhibits subtle step-size dependence not detectable by gradient flow analysis, converging to a limit dependent on the step size.
Faster convergence and handling larger mini-batches for deep neural networks.
problem Generalization gap in large-scale distributed training of deep neural networks.
method Second-order optimization using Kronecker-factored approximate curvature.
result Achieved 75% Top-1 validation accuracy with mini-batch size of 131,072 in 978 iterations.
Generalizes SGD convergence analysis and optimizes stepsize.
problem Analyzes convergence of stochastic gradient descent (SGD) under arbitrary sampling.
method Introduces expected smoothness and derives convergence rates for various mini-batching strategies.
result Derives optimal mini-batch size and stepsize-switching rules.
Improves SGD for convex functions with mini-batches, proving linear convergence.
problem Minimizing convex functions with constraints.
method Projected semi-stochastic gradient descent with mini-batches.
result Linear convergence under weak strong convexity assumption.
New method selects diverse mini-batches for active learning.
problem Reduce labeled data for deep learning models.
method Sequential selection of diverse mini-batches using K-means clustering.
result Achieves comparable or better performance than previous methods.
Improved SGD convergence by sampling data points less likely to be in the same mini-batch.
problem Improving the convergence speed of stochastic gradient descent (SGD).
method Active mini-batch sampling using repulsive point processes.
result Repulsive sampling schemes lower the variance of the gradient estimator.
Flipout decorrelates mini-batch weights for more variance reduction.
problem Limited variance reduction in mini-batches due to shared weight perturbations.
method Implicitly samples pseudo-independent weight perturbations for each example.
result Achieves ideal linear variance reduction for various network types.
SGD converges fast in over-parametrized models that interpolate data.
problem Understanding fast convergence of SGD in modern machine learning.
method Formal analysis of SGD convergence in over-parametrized models.
result SGD converges similarly to full gradient descent in certain regimes.
Improves deep learning training by matching mini-batch distributions.
problem Overfitting and noise in mini-batch training.
method ITDM, which matches the moments of mini-batch distributions to reduce overfitting.
result ITDM reduces overfitting and improves DNN training.
This study addresses the challenges of dynamic mini-batch sub-sampling in neural network training.
problem Challenges in training neural networks due to dynamic mini-batch sub-sampling.
method Distinguishes between static and dynamic sub-sampling, recasting optimization to find SNN-GPPs.
result SNN-GPPs are less susceptible to sub-sampling-induced discontinuities and better approximate true optima.
Improved robustness in optimization methods using second-order information.
problem Scalability and sensitivity to mini-batch size in optimization methods.
method Mini-Batch Stochastic Variance-Reduced Newton (extttMb−SVRN) algorithm incorporating partial second-order information. result Achieves a fast linear convergence rate independent of mini-batch size for large data sizes.
SP-NGD improves deep learning models' generalization with large mini-batch sizes.
problem Worse generalization performance with large mini-batch sizes in deep learning.
method SP-NGD, a natural gradient descent approach for large-scale deep learning.
result SP-NGD achieves similar generalization performance to first-order methods with accelerated convergence and negligible overhead.
In this work we develop a new algorithm for regularized empirical risk minimization. Our method extends recent techniques of Shalev-Shwartz [02/2015], which enable a dual-free analysis of SDCA, to arbitrary mini-batching schemes. Moreover, our method is able to better utilize the information in the data defining the ER…
New optimal step sizes and mini-batch sizes for SAGA.
problem Finding optimal step sizes and mini-batch sizes for SAGA.
method Provided closed-form expressions for expected smoothness constant and suggested new step sizes and mini-batch sizes.
result Total complexity of SAGA decreases linearly with mini-batch size up to an optimal value.
RMGD uses bandit theory to optimize mini-batch size for faster and better performance.
problem Determining the optimal mini-batch size for gradient descent is time-consuming.
method Resilient Mini-batch Gradient Descent (RMGD) using Multi-Armed Bandit.
result RMGD achieves better performance than grid search in less time.
New findings show mini-batch SGD operates in a 'Edge of Stochastic Stability' regime.
problem Understanding the stability and convergence of mini-batch SGD.
method Analyzing the mini-batch Hessian and its directional curvature.
result Mini-batch SGD operates in a different stability regime (Edge of Stochastic Stability) compared to full-batch GD.
CBN improves batch normalization for small mini-batch sizes.
problem Reduced effectiveness of Batch Normalization in small mini-batch sizes.
method CBN uses statistics from multiple recent iterations, compensating for network weight changes via Taylor polynomials.
result CBN outperforms original batch normalization and direct iteration statistics in object detection and image classification.
New statistical properties for mini-batch Cox-NN optimization.
problem Optimizing deep Cox neural networks using mini-batches.
method Developed mini-batch maximum partial-likelihood estimator (mb-MPLE) for Cox-NN.
result mb-MPLE is consistent and achieves optimal convergence rate.
Paper analyzes SARAH for nonconvex optimization with mini-batches.
problem Solving nonconvex optimization problems with mini-batches.
method Stochastic Recursive Gradient Algorithm (SARAH) for nonconvex losses.
result Sublinear and linear convergence rates for different types of nonconvex functions.
Stochastic dual coordinate ascent (SDCA) is an effective technique for solving regularized loss minimization problems in machine learning. This paper considers an extension of SDCA under the mini-batch setting that is often used in practice. Our main contribution is to introduce an accelerated mini-batch version of SDC…
Local SGD improves deep learning models' generalization without sacrificing efficiency.
problem Large mini-batches lead to poor model generalization.
method Proposes and evaluates post-local SGD.
result Post-local SGD significantly improves generalization compared to large-batch training.
We propose a mini-batching scheme for improving the theoretical complexity and practical performance of semi-stochastic gradient descent applied to the problem of minimizing a strongly convex composite function represented as the sum of an average of a large number of smooth convex functions, and simple nonsmooth conve…
Optimal rates found for learning with Nyström stochastic gradient methods.
problem Nonparametric regression learning with improved computational efficiency.
method Combination of stochastic gradient methods with Nyström subsampling, allowing multiple passes and mini-batches.
result Derivation of optimal learning rates considering various parameters.
We propose mS2GD: a method incorporating a mini-batching scheme for improving the theoretical complexity and practical performance of semi-stochastic gradient descent (S2GD). We consider the problem of minimizing a strongly convex function represented as the sum of an average of a large number of smooth convex function…
FRN layer eliminates batch dependence in deep learning, improving performance across various tasks.
problem Batch Normalization's dependency on mini-batch elements can degrade performance for small batches.
method Filter Response Normalization (FRN) operates independently on each activation channel of each batch element.
result FRN layer outperforms BN and other alternatives in various settings for all batch sizes.
New compression scheme improves mini-batch ML efficiency.
problem Improving efficiency of mini-batch stochastic gradient descent.
method Tuple-oriented compression tailored for mini-batch stochastic gradient descent.
result Substantial compression ratios and runtime reductions for mini-batch ML.
New method reduces variance in complex probabilistic model optimization.
problem High variance in stochastic optimisation of complex models.
method Use recognition network to approximate optimal control variate for each mini-batch.
result Sub-optimal variance reduction is improved with new approach.
The paper analyzes time-dependent streaming data with biased gradient estimates and proposes improved stochastic optimization methods.
problem Stochastic optimization in a streaming setting with time-dependent and biased gradient estimates.
method Analysis of several first-order methods including SGD, mini-batch SGD, and time-varying mini-batch SGD, along with their Polyak-Ruppert averages.
result Time-varying mini-batch SGD methods can break long- and short-range dependence structures, and biased SGD methods can achieve comparable performance to their unbiased counterparts.
HydaLearn dynamically adjusts task weights for better MTL performance.
problem Constant loss weights in MTL lead to poor results due to drifting relevance and varying mini-batch composition.
method HydaLearn uses mini-batch gradients to dynamically adjust task weights.
result HydaLearn improves performance on synthetic and real-world data.
NLCG optimizes DNN training, especially with large mini-batches.
problem Improving convergence speed in large-scale DNN training.
method Stochastic Preconditioned Nonlinear Conjugate Gradient (SP-NLCG) algorithm.
result NLCG improves DNN training accuracy by over 10 percentage points at large mini-batch sizes.
OT-GAN uses optimal transport to improve GANs stability and performance.
problem Improving stability and performance of GANs.
method Combines optimal transport and energy distance in adversarially learned feature space.
result OT-GAN achieves state-of-the-art results on image generation benchmarks.
SWAP uses large mini-batches to train DNNs faster with good generalization.
problem Training deep neural networks with small mini-batches is time-consuming.
method SWAP computes an approximate solution with large mini-batches and refines it by averaging weights of multiple parallel models.
result SWAP trains models as well as small-batch training but in significantly less time.
The paper debiases mini-batch approximations in deep learning for more accurate optimization and uncertainty quantification.
problem Bias in mini-batch approximations distorts the shape of quadratic approximations used in deep learning.
method Developed and evaluated debiasing strategies for mini-batch approximations.
result Debiasing strategies improve the accuracy of second-order optimization and uncertainty quantification in deep learning.
A new algorithm PBNN improves scalability of Bayesian Neural Networks.
problem Scalability issues in Neural Network posterior sampling with large datasets.
method Penalty Bayesian Neural Networks (PBNN) using subsampled batch data.
result PBNN achieves good predictive performance with small mini-batch sizes.
Study mini-batch SGD noise and its limits, proving complexity guarantees.
problem Analyzing the noise in mini-batch SGD and its impact on optimization.
method Examined the conditional covariance and diffusion limits of SGD under different sampling designs.
result Proved mean-square upper bounds and Fisher van Trees lower bounds for SGD, linking them to effective dimension and condition number.