Improved L-BFGS for large-scale ML with progressive batching.
problem Combining fast generalization with reliable line search in large-scale ML.
method Progressive batching, stochastic line search, stable quasi-Newton updating.
result Performs well on training logistic regression and deep neural networks.
Classical stochastic gradient methods for optimization rely on noisy gradient approximations that become progressively less accurate as iterates approach a solution. The large noise and small signal in the resulting gradients makes it difficult to use them for adaptive stepsize selection and automatic stopping. We prop…
Local SGD improves deep learning models' generalization without sacrificing efficiency.
problem Large mini-batches lead to poor model generalization.
method Proposes and evaluates post-local SGD.
result Post-local SGD significantly improves generalization compared to large-batch training.
Small batch training improves deep neural network performance and stability.
problem Improving deep neural network performance and stability with limited computational resources.
method Experimental comparison of test performance for different mini-batch sizes, focusing on learning rate scaling and training duration.
result Best performance achieved for mini-batch sizes between 2 and 32, contrasting recent work advocating larger batch sizes.
AdAdaGrad optimizes batch sizes for deep learning models, reducing the generalization gap.
problem The generalization gap between large-batch and small-batch training in deep learning.
method AdAdaGrad introduces adaptive batch size strategies derived from adaptive sampling methods.
result AdAdaGradNorm converges to a first-order stationary point with a rate of O(1/K) in K iterations.
Breaks the hardness conjecture for batch RL with a novel tournament-based approach.
problem Sample-efficient reinforcement learning from exploratory data.
method BVFT algorithm using pairwise comparison and state-action partition.
result Solves the learning problem in a setting previously thought impossible.
Optimal batch size minimizes training time for neural networks.
problem Minimizing training time for two-layer neural networks with SGD.
method Characterized optimal batch size as a function of target hardness (information exponents). Used Correlation loss SGD to overcome limitations.
result Optimal batch size minimizes training time without changing total sample complexity.
New method accelerates neural network training with large batch sizes using second-order information and adversarial training.
problem Training neural networks with large batch sizes leads to poorer generalization.
method Adaptive batch size training with second-order methods and adversarial training, using Ray for efficient autoscaling.
result Significantly improved training time and accuracy compared to existing solutions.
Accelerates BERT pretraining from 3 days to 54 minutes.
problem Long training time of BERT due to large mini-batch sizes.
method LANS method and learning rate scheduler for large mini-batch training.
result Achieved fastest BERT training time of 54 minutes.
Batch normalization improves deep networks by aligning their decision boundaries with data.
problem Improving the performance and generalization of deep networks.
method Theoretical analysis of batch normalization as a function approximation technique for continuous piecewise affine splines.
result Batch normalization adapts the geometry of a deep network's partition to match the data, improving learning and generalization.
Neumann Optimizer speeds up deep learning model training.
problem Slow training of large deep neural networks.
method A novel optimization algorithm that uses large batch sizes and implicitly computes the inverse Hessian.
result Successfully trained large ImageNet models with mini-batch sizes of up to 32000.
New study shows non-adaptive trials can be outperformed by adaptive designs in treatment selection.
problem Determining the best allocation of resources in clinical trials.
method Analysis of batched arm elimination designs and comparison with completely randomized trials.
result Simple adaptive designs universally and strictly dominate non-adaptive completely randomized trials for at least three treatment arms.
Despite the recent progress towards efficient multiple kernel learning (MKL), the structured output case remains an open research front. Current approaches involve repeatedly solving a batch learning problem, which makes them inadequate for large scale scenarios. We propose a new family of online proximal algorithms fo…
Accelerates ADMM for convex problems with linear convergence.
problem Improving convergence rate of ADMM for general convex problems.
method Introduces momentum acceleration trick into SVRG-ADMM for both strongly convex and general convex cases.
result ASVRG-ADMM achieves O(1/T^2) convergence rate for general convex problems.
Visualizes optimization landscapes to understand FCN performance.
problem Understanding why FCNs perform well empirically.
method Visualizing objective functions in 3D space, comparing networks, investigating skip-layer connections, and analyzing loss surfaces.
result Skip-layer connections in FCNs promote flat optimization landscapes, leading to better generalization.
A long-standing obstacle to progress in deep learning is the problem of vanishing and exploding gradients. Although, the problem has largely been overcome via carefully constructed initializations and batch normalization, architectures incorporating skip-connections such as highway and resnets perform much better than …
This paper analyzes how BN affects GD convergence and stability.
problem The effect of batch normalization on gradient descent convergence and stability.
method Quantitative analysis of gradient descent with and without batch normalization on ordinary least squares.
result Gradient descent with batch normalization converges for arbitrary learning rates and remains linear under mild conditions.
SCSG method optimizes stochastic gradient-based optimization for large-scale problems.
problem Lack of adaptability between theoretical optimality and practical applicability in stochastic gradient-based optimization.
method SCSG method with batch variance reduction and geometrization technique.
result SCSG achieves strictly better theoretical complexity and is adaptive to both strong convexity and target accuracy.
LTP learns per-layer thresholds for efficient pruning of deep networks.
problem Efficiently pruning deep neural networks to reduce computational cost and size.
method LTP learns thresholds via gradient descent, making pruning computationally efficient and scalable.
result LTP achieves competitive compression rates and maintains high accuracy on ImageNet networks.
Study uses Bayesian Optimization to analyze noise effects in materials research.
problem Optimizing materials with many variables and experimental noise.
method Batch Bayesian Optimization with synthetic data analysis.
result Noise sensitivity varies by problem landscape, impacting optimization outcomes.
Parallel Gaussian process surrogate for noisy likelihood evaluations in Bayesian inference.
problem Bayesian inference with limited noisy log-likelihood evaluations from complex models.
method Hierarchical Gaussian process surrogate model for log-likelihood, batch-sequential design strategies.
result Robust, highly parallelizable, and sample-efficient method.
Paper closes the gap in MP-MAB problems with novel adaptive communication and exploration.
problem Closing the gap between decentralized MP-MAB and natural centralized lower bound.
method BEACON: Batched Exploration with Adaptive COmmunicatioN, incorporating ADC and batched exploration.
result Proves logarithmic regret for a generalized MP-MAB problem.
Proposes STRON method for large-scale machine learning problems.
problem Large-scale machine learning problems.
method Stochastic Trust Region Inexact Newton (STRON) method using CG to solve trust region subproblem with progressive subsampling.
result Empirical results show efficacy of STRON method.
An emerging design principle in deep learning is that each layer of a deep artificial neural network should be able to easily express the identity transformation. This idea not only motivated various normalization techniques, such as \emph{batch normalization}, but was also key to the immense success of \emph{residual …
New algorithm learns from distributed, heterogeneous data without shuffling.
problem Efficiently learning from distributed, heterogeneous data in exascale simulations.
method Block-random gradient descent algorithm for distributed, heterogeneous data.
result Algorithm enables in situ learning without pre-shuffling data.
New benchmarks for offline RL from diverse datasets.
problem Measuring progress in offline RL due to lack of suitable benchmarks.
method Developed benchmarks tailored for offline RL, focusing on diverse dataset properties.
result Revealed deficiencies in existing offline RL algorithms.
BackPACK extends PyTorch to compute additional gradient info.
problem Lack of efficient tools for computing mini-batch variance and Hessian approximations.
method BackPACK builds on PyTorch to automatically compute additional derivatives.
result BackPACK enables efficient computation of various derivative quantities.
Study examines how training regime affects neural networks' forgetting.
problem Catastrophic forgetting in neural networks when learning multiple tasks sequentially.
method Analyzes the impact of different training regimes (learning rate, batch size, regularization) on forgetting.
result Training regimes that widen tasks' local minima help prevent catastrophic forgetting.
Integrates uncertainty of loss landscape into stochastic optimization.
problem Improving convergence and generalization in stochastic optimization.
method Incorporates variance of stochastic loss function into momentum updates.
result Improved convergence rates on MNIST and CIFAR-10 datasets.
RMGD uses bandit theory to optimize mini-batch size for faster and better performance.
problem Determining the optimal mini-batch size for gradient descent is time-consuming.
method Resilient Mini-batch Gradient Descent (RMGD) using Multi-Armed Bandit.
result RMGD achieves better performance than grid search in less time.
New research shows many batch selection methods for training work just as well as full batch training.
problem Finding optimal batch selection methods for training.
method Analysis of mini-batch Gradient Descent (GD) and Stochastic GD (SGD) with various batch selection rules.
result All mini-batch schedules, including deterministic ones, generalize optimally for smooth Lipschitz-convex/nonconvex/strongly-convex loss functions.
FRN layer eliminates batch dependence in deep learning, improving performance across various tasks.
problem Batch Normalization's dependency on mini-batch elements can degrade performance for small batches.
method Filter Response Normalization (FRN) operates independently on each activation channel of each batch element.
result FRN layer outperforms BN and other alternatives in various settings for all batch sizes.
We analyze how batch learning impacts bandit problems.
problem Impact of batch learning in stochastic bandits.
method Policy-agnostic regret analysis, upper and lower bounds demonstration.
result The impact of batch learning can be measured in terms of online behavior.
Batch augmentation improves deep learning training by reducing batch size requirements.
problem Training deep neural networks with large batches can lead to overfitting.
method Replicate samples within a batch with different data augmentations.
result Batch augmentation reduces the number of necessary SGD updates for achieving the same accuracy.
AdaBatch dynamically adjusts batch size during training for deep learning models.
problem Choosing optimal batch size for deep neural networks.
method Adaptive batch size adjustment during training.
result Adaptive batch sizes improve performance by up to 6.25x on 4 GPUs with minimal accuracy loss.
BatchGFN uses generative flow networks for efficient batch active learning.
problem Efficiently selecting informative batches for active learning.
method Generative flow networks to sample batches proportional to a batch reward.
result Constructs highly informative batches with a single forward pass per point.
Wider neural networks perform better with large batches.
problem Communication overheads in small-batch training.
method Theoretical analysis and experiments on neural networks.
result Wider networks are more suitable for large-batch training.
Adaptive batch sizes improve active learning efficiency and flexibility.
problem Fixed batch sizes in active learning are inefficient due to dynamic cost-speed trade-offs.
method Probabilistic Numerics framework that adaptively changes batch sizes based on integration error and precision objectives.
result Significant enhancement in learning efficiency and flexibility across various applications.
This paper tackles data-efficient CEE with scarce labelled data, proposing a method to progressively reduce generalization risk.
problem Data scarcity in CEE tasks, especially in high-stake domains like medical treatment effect prediction.
method Develops a principled label acquisition pipeline (MACAL) for CEE tasks, focusing on reducing generalization risk progressively.
result Proposes Model Agnostic Causal Active Learning (MACAL) algorithm for batch-wise label acquisition.
Opt-BBAI identifies the best arm with minimal batches and pulls, optimizing both sample and batch complexity.
problem Batched best arm identification (BBAI) problem, aiming to minimize policy switches and resource usage.
method Proposed Opt-BBAI algorithm, achieving near-optimal sample and batch complexity in non-asymptotic settings.
result First algorithm to achieve near-optimal sample and batch complexity in non-asymptotic settings.
CBN improves batch normalization for small mini-batch sizes.
problem Reduced effectiveness of Batch Normalization in small mini-batch sizes.
method CBN uses statistics from multiple recent iterations, compensating for network weight changes via Taylor polynomials.
result CBN outperforms original batch normalization and direct iteration statistics in object detection and image classification.
A new method designs batches for Bayesian optimization more efficiently.
problem Efficiently designing batches for Bayesian optimization to reduce total time.
method Minimal Terminal Variance (MTV) acquisition function, optimizing I-optimality criterion.
result MTV designs batches more efficiently than other methods, as shown by numerical experiments.
New framework trains large SciML models solving PDEs in reasonable time.
problem Training large SciML models solving PDEs is challenging and time-consuming.
method Data parallel distributed deep learning framework with optimized methods.
result Neural PDE solvers can be viably trained for practical applications.
BaSE policy optimizes multi-armed bandits with batched data.
problem Optimizing multi-armed bandits with batched data.
method BaSE (batched successive elimination) policy for batched multi-armed bandits.
result Achieves rate-optimal regrets with adaptive batch sizes.
Four improvements to Batch Normalization improve deep learning performance.
problem Improving Batch Normalization for better deep learning performance.
method Proposed improvements include reasoning about current examples, Ghost Batch Normalization, weight decay regularization, and a new normalization algorithm for small batch sizes.
result Performance gains across all batch sizes with no additional computation during training.
Riemannian stochastic gradient descent converges faster with increasing batch size.
problem Improving convergence rate of Riemannian stochastic gradient descent.
method Theoretical analysis and numerical investigation of increasing batch size effects.
result Riemannian stochastic gradient descent converges faster with increasing batch size.
Polynomial-time algorithm for list-decodable linear regression with batches.
problem Efficiently decoding linear regression with a fraction of adversarial data.
method Polynomial time algorithm using batches of i.i.d. samples.
result Returns a list of size O(1/α^2) with one item close to true parameter.
New findings show mini-batch SGD operates in a 'Edge of Stochastic Stability' regime.
problem Understanding the stability and convergence of mini-batch SGD.
method Analyzing the mini-batch Hessian and its directional curvature.
result Mini-batch SGD operates in a different stability regime (Edge of Stochastic Stability) compared to full-batch GD.