This work analyzes mini-batch inference limits and introduces Mini-AMP for clustering.
problem Analyzing information-theoretic limits of mini-batch inference in statistical learning.
method Detailed analysis of Mini-AMP for generalized linear models and low-rank matrix factorization.
result Characterization of optimal performance and phase transitions for mini-batch inference.
Neural moving average model speeds up state space model inference for time series data.
problem Efficiently scaling approximate Bayesian inference for time series data.
method Proposes a novel generative model (neural moving average model) for latent temporal states in state space models.
result Achieves accurate parameter estimation in a short time for various models.
Adaptive Langevin dynamics reduces bias in Bayesian inference with mini-batching.
problem Bias in posterior sampling due to mini-batching in Bayesian inference.
method Adaptive Langevin dynamics with dynamical friction to correct noise.
result Quantified bias in posterior distribution due to mini-batching.
A new method for graph neural networks speeds up inference and training.
problem Challenges in constructing mini-batches for large graphs in graph neural networks.
method Theoretical model of batch construction via maximizing influence score of nodes on outputs.
result Accelerates inference by up to 130x compared to previous methods.
Adaptive scan Gibbs sampler improves large-scale inference performance.
problem Efficiently updating large-scale online inference problems.
method Derives an adaptive scan Gibbs sampler that optimizes mini-batch size selection.
result Demonstrates superior performance compared to collapsed Gibbs sampler.
New method reduces variance in complex probabilistic model optimization.
problem High variance in stochastic optimisation of complex models.
method Use recognition network to approximate optimal control variate for each mini-batch.
result Sub-optimal variance reduction is improved with new approach.
A new method for statistical inference using SGD under φ-mixing data.
problem Valid statistical inference for time series data with general correlation.
method Proposes a mini-batch SGD estimator and associated mini-batch bootstrap procedure for φ-mixing data. result The proposed method constructs valid confidence intervals for φ-mixing data. A new MH-MCMC method using mini-batches and stochastic gradient for scalable inference.
problem Computational inefficiency of traditional MCMC algorithms for large datasets.
method Mini-batch MH-MCMC with reversible stochastic gradient proposal.
result The method provides approximate tempered stationary distribution and reasonable acceptance probabilities.
New algorithm boosts deep learning training speed.
problem Efficiently train deep neural networks on large clusters.
method Synchronous distributed SGD with amortized inference model.
result Dynamic cutoff improves convergence and training time.
A new method for learning Bayesian neural networks using layerwise inference.
problem Learning Bayesian neural networks efficiently and accurately.
method Bayesian layerwise inference, treating neural networks as stacked Bayesian linear models, with pseudo-targets defined by backpropagated gradients.
result The method converges quickly and performs well on various benchmarks.
A new algorithm PBNN improves scalability of Bayesian Neural Networks.
problem Scalability issues in Neural Network posterior sampling with large datasets.
method Penalty Bayesian Neural Networks (PBNN) using subsampled batch data.
result PBNN achieves good predictive performance with small mini-batch sizes.
Sparse GPs improved with nearest neighbor inducing variables.
problem Sparse GPs struggle with large numbers of inducing variables.
method Introduced a hierarchical prior for inducing variables and used nearest neighbor information for sparsity.
result Significant computational gains compared to standard sparse GPs.
Paper proposes efficient inference for hidden Markov models with memory decay.
problem Challenges in scalability due to dependencies in hidden Markov model observation data.
method Utilizes memory decay to carry out forward and backward probabilities with subsequences, enabling efficient inference over long sequences.
result Developed an efficient algorithm to numerically estimate the gap of top Lyapunov exponents, which determines the length of subsequences.
A new algorithm is proposed which accelerates the mini-batch k-means algorithm of Sculley (2010) by using the distance bounding approach of Elkan (2003). We argue that, when incorporating distance bounds into a mini-batch algorithm, already used data should preferentially be reused. To this end we propose using nested …
Mini-batch EM algorithm speeds up convergence for large datasets.
problem Efficiently processing large datasets in latent variable models.
method Proposes mini-batch version of Stochastic Approximation EM algorithm for exponential models.
result Converges under classical conditions with mini-batch sampling.
New method speeds up Gaussian process training and inference for large datasets.
problem Training and inference in Gaussian processes are computationally expensive for large datasets.
method Iterative alternating projection method that accesses subblocks of the kernel matrix, reducing time and space complexity.
result Empirically, the method accelerates GP training and inference by up to 72x compared to conjugate gradients.
Proposes m-POT to improve m-OT's misspecified mappings issue.
problem Misspecified mappings in mini-batch optimal transport.
method Partial optimal transport (POT) between mini-batch empirical measures.
result m-POT alleviates incorrect mappings compared to current methods.
DM-SGD uses DPP to diversify mini-batches for SGD, improving model performance.
problem Improving mini-batch diversity in SGD to reduce variance and improve model interpretability.
method DM-SGD uses Determinantal Point Processes (DPP) to select mini-batches with diverse data points.
result DM-SGD outperforms regular SGD and stratified sampling in various setups.
Paper proposes mini-batch ADMMs for solving nonconvex nonsmooth optimization problems.
problem Solving large-scale nonconvex nonsmooth optimization problems.
method Proposes mini-batch stochastic ADMMs for nonconvex nonsmooth optimization.
result Mini-batch stochastic ADMMs converge to a stationary point with rate O(1/T).
New method improves mini-batch efficiency for convex optimization problems.
problem Efficiently solving convex regularized empirical risk minimization with mini-batches.
method Doubly Accelerated Stochastic Variance Reduced Dual Averaging Method with double acceleration and variance reduction.
result Significantly reduces the number of mini-batches needed for optimal performance.
EE-Grad optimizes mini-batch sizes for SGD efficiency.
problem Optimizing cost-efficiency in mini-batch SGD.
method Sequential exploration and exploitation of mini-batch oracles.
result EE-Grad achieves optimal cost-efficiency with performance guarantees.
Proposes a method to learn conditional VAEs from datasets with missing covariates.
problem Learning conditional VAEs from datasets with missing covariates.
method Augments conditional VAEs with a prior distribution for missing covariates and estimates their posterior using amortised variational inference.
result The proposed method outperforms previous methods in learning conditional VAEs from non-temporal, temporal, and longitudinal datasets.
Small batch training improves deep neural network performance and stability.
problem Improving deep neural network performance and stability with limited computational resources.
method Experimental comparison of test performance for different mini-batch sizes, focusing on learning rate scaling and training duration.
result Best performance achieved for mini-batch sizes between 2 and 32, contrasting recent work advocating larger batch sizes.
New streaming methods improve convergence rates for optimization problems.
problem Optimizing large-scale, sequential data problems.
method Time-varying mini-batches and Polyak-Ruppert averaging for gradient-based algorithms.
result Time-varying mini-batches and averaging achieve optimal convergence and variance reduction.
A new method for mini-batch optimal transport improves scalability and accuracy.
problem Desired estimation and proper metric approximation in m-OT.
method BoMb-OT: Finds optimal coupling between mini-batches.
result BoMb-OT approximates a proper metric and improves m-OT's performance.
Federated multi-mini-batch improves efficiency in non-IID environments.
problem Performance and communication efficiency challenges in federated learning with non-IID data.
method Introduces federated multi-mini-batch approach to balance performance and communication.
result Federated multi-mini-batch outperforms federated averaging in non-IID settings.
E2-Train reduces training energy by 80%+ for state-of-the-art CNNs.
problem Efficient training of energy-hungry CNNs on edge devices.
method Selective layer update, stochastic mini-batch dropping, and sign prediction for low-precision backpropagation.
result Achieves >90% energy savings for training ResNet-74 on CIFAR-10.
Paper finds optimal mini-batch size for SGD to speed up learning.
problem Optimizing mini-batch size for faster SGD convergence.
method Empirical inverse law and theoretical bound on mini-batch SGD training.
result An accurate model for predicting training time and identifying implications for algorithm and hardware.
Study shows how mini-batch GD with random reshuffling affects least squares regression dynamics.
problem Analyzing the error dynamics of mini-batch GD with random reshuffling for least squares regression.
method Represented training and generalization errors through a sample cross-covariance matrix Z, compared with sample covariance matrix of original features X, and used linear scaling rule for analysis.
result Mini-batch GD with random reshuffling exhibits subtle step-size dependence not detectable by gradient flow analysis, converging to a limit dependent on the step size.
Faster convergence and handling larger mini-batches for deep neural networks.
problem Generalization gap in large-scale distributed training of deep neural networks.
method Second-order optimization using Kronecker-factored approximate curvature.
result Achieved 75% Top-1 validation accuracy with mini-batch size of 131,072 in 978 iterations.
Generalizes SGD convergence analysis and optimizes stepsize.
problem Analyzes convergence of stochastic gradient descent (SGD) under arbitrary sampling.
method Introduces expected smoothness and derives convergence rates for various mini-batching strategies.
result Derives optimal mini-batch size and stepsize-switching rules.
Improves SGD for convex functions with mini-batches, proving linear convergence.
problem Minimizing convex functions with constraints.
method Projected semi-stochastic gradient descent with mini-batches.
result Linear convergence under weak strong convexity assumption.
New method selects diverse mini-batches for active learning.
problem Reduce labeled data for deep learning models.
method Sequential selection of diverse mini-batches using K-means clustering.
result Achieves comparable or better performance than previous methods.
Improved SGD convergence by sampling data points less likely to be in the same mini-batch.
problem Improving the convergence speed of stochastic gradient descent (SGD).
method Active mini-batch sampling using repulsive point processes.
result Repulsive sampling schemes lower the variance of the gradient estimator.
Flipout decorrelates mini-batch weights for more variance reduction.
problem Limited variance reduction in mini-batches due to shared weight perturbations.
method Implicitly samples pseudo-independent weight perturbations for each example.
result Achieves ideal linear variance reduction for various network types.
Batch normalisation doesn't affect variational inference but fails for larger batch sizes.
problem Failure of Monte Carlo Batch Normalisation (MCBN) for capturing epistemic uncertainty in larger batch sizes.
method Investigated MCBN as an approximate inference technique for Bayesian neural networks, showing its limitations and providing insights for improvement.
result For larger batch sizes, MCBN fails to capture epistemic uncertainty, requiring the batch size to be a variational parameter.
SGD converges fast in over-parametrized models that interpolate data.
problem Understanding fast convergence of SGD in modern machine learning.
method Formal analysis of SGD convergence in over-parametrized models.
result SGD converges similarly to full gradient descent in certain regimes.
Improves deep learning training by matching mini-batch distributions.
problem Overfitting and noise in mini-batch training.
method ITDM, which matches the moments of mini-batch distributions to reduce overfitting.
result ITDM reduces overfitting and improves DNN training.
This study addresses the challenges of dynamic mini-batch sub-sampling in neural network training.
problem Challenges in training neural networks due to dynamic mini-batch sub-sampling.
method Distinguishes between static and dynamic sub-sampling, recasting optimization to find SNN-GPPs.
result SNN-GPPs are less susceptible to sub-sampling-induced discontinuities and better approximate true optima.
Improved robustness in optimization methods using second-order information.
problem Scalability and sensitivity to mini-batch size in optimization methods.
method Mini-Batch Stochastic Variance-Reduced Newton (extttMb−SVRN) algorithm incorporating partial second-order information. result Achieves a fast linear convergence rate independent of mini-batch size for large data sizes.
SP-NGD improves deep learning models' generalization with large mini-batch sizes.
problem Worse generalization performance with large mini-batch sizes in deep learning.
method SP-NGD, a natural gradient descent approach for large-scale deep learning.
result SP-NGD achieves similar generalization performance to first-order methods with accelerated convergence and negligible overhead.
In this work we develop a new algorithm for regularized empirical risk minimization. Our method extends recent techniques of Shalev-Shwartz [02/2015], which enable a dual-free analysis of SDCA, to arbitrary mini-batching schemes. Moreover, our method is able to better utilize the information in the data defining the ER…
New optimal step sizes and mini-batch sizes for SAGA.
problem Finding optimal step sizes and mini-batch sizes for SAGA.
method Provided closed-form expressions for expected smoothness constant and suggested new step sizes and mini-batch sizes.
result Total complexity of SAGA decreases linearly with mini-batch size up to an optimal value.
RMGD uses bandit theory to optimize mini-batch size for faster and better performance.
problem Determining the optimal mini-batch size for gradient descent is time-consuming.
method Resilient Mini-batch Gradient Descent (RMGD) using Multi-Armed Bandit.
result RMGD achieves better performance than grid search in less time.
New findings show mini-batch SGD operates in a 'Edge of Stochastic Stability' regime.
problem Understanding the stability and convergence of mini-batch SGD.
method Analyzing the mini-batch Hessian and its directional curvature.
result Mini-batch SGD operates in a different stability regime (Edge of Stochastic Stability) compared to full-batch GD.
SGDM accelerates faster than SGD with large batch sizes and permits broader learning rates.
problem Understanding the role of momentum in SGDM and its convergence rates.
method Analysis of SGDM convergence rates under strongly convex settings, including finite-sample rates and asymptotic normality of the averaged estimator.
result SGDM converges faster than SGD with large batch sizes and permits broader learning rates.
CBN improves batch normalization for small mini-batch sizes.
problem Reduced effectiveness of Batch Normalization in small mini-batch sizes.
method CBN uses statistics from multiple recent iterations, compensating for network weight changes via Taylor polynomials.
result CBN outperforms original batch normalization and direct iteration statistics in object detection and image classification.
New statistical properties for mini-batch Cox-NN optimization.
problem Optimizing deep Cox neural networks using mini-batches.
method Developed mini-batch maximum partial-likelihood estimator (mb-MPLE) for Cox-NN.
result mb-MPLE is consistent and achieves optimal convergence rate.