arXiv research

A new method designs batches for Bayesian optimization more efficiently.

TS-RSR improves batch Bayesian Optimization by minimizing redundancy and focusing on high uncertainty points.

E$^4$ algorithm optimizes batched linear bandits with minimal regret and batches.

SGD's performance improves with critical batch size, minimizing SFO complexity.

The paper analyzes risk bounds and Rademacher complexity in batch RL.

LSAM optimizes deep learning training with improved efficiency.

GN algorithm solves batched bandit for nondegenerate functions near-optimally.

Optimal batch size minimizes training time for neural networks.

Opt-BBAI identifies the best arm with minimal batches and pulls, optimizing both sample and batch complexity.

In this paper, we develop a new accelerated stochastic gradient method for efficiently solving the convex regularized empirical risk minimization problem in mini-batch settings. The use of mini-batches is becoming a golden standard in the machine learning community, because mini-batch settings stabilize the gradient es…

Use of an autoencoder (AE) as a normal model is a state-of-the-art technique for unsupervised-anomaly detection in sounds (ADS). The AE is trained to minimize the sample mean of the anomaly score of normal sounds in a mini-batch. One problem with this approach is that the anomaly score of rare-normal sounds becomes hig…

Paper tackles robust batched bandits for heavy-tailed rewards.

Batched Neural Bandits reduces policy updates in sequential decision-making.

The paper derives a new theorem for predicting batches of data.

Mini-batch sub-sampling in neural network training is unavoidable, due to growing data demands, memory-limited computational resources such as graphical processing units (GPUs), and the dynamics of on-line learning. In this study we specifically distinguish between static mini-batch sub-sampled loss functions, where mi…

Optimal algorithm for minimizing regret in heavy-tailed bandits.

Adaptive SGD learns optimal batch size for strong convex functions.

Mini-batch stochastic gradient descent and variants thereof have become standard for large-scale empirical risk minimization like the training of neural networks. These methods are usually used with a constant batch size chosen by simple empirical inspection. The batch size significantly influences the behavior of the …

Adaptive batching improves Gaussian process surrogates for noisy level set estimation.

New algorithm learns optimal policy with multi-step lookahead information.

We study the Stochastic Gradient Descent (SGD) method in nonconvex optimization problems from the point of view of approximating diffusion processes. We prove rigorously that the diffusion process can approximate the SGD algorithm weakly using the weak form of master equation for probability evolution. In the small ste…

A new batch ensemble method reduces regret in stochastic bandits.

This study explains how different training methods affect the minimizer of neural networks.

Batch normalization makes deep residual networks train faster.

SGD and weight decay encourage neural networks to learn low-rank weight matrices.

Stochastic dual coordinate ascent (SDCA) is an effective technique for solving regularized loss minimization problems in machine learning. This paper considers an extension of SDCA under the mini-batch setting that is often used in practice. Our main contribution is to introduce an accelerated mini-batch version of SDC…

Paper finds optimal mini-batch size for SGD to speed up learning.

We present Optimal Transport GAN (OT-GAN), a variant of generative adversarial nets minimizing a new metric measuring the distance between the generator distribution and the data distribution. This metric, which we call mini-batch energy distance, combines optimal transport in primal form with an energy distance define…

Gradient control plays an important role in feed-forward networks applied to various computer vision tasks. Previous work has shown that Recurrent Highway Networks minimize the problem of vanishing or exploding gradients. They achieve this by setting the eigenvalues of the temporal Jacobian to 1 across the time steps. …

New study reveals a polynomial penalty for adapting to unknown margin parameters in batched nonparametric bandits.

Study shows gradient variance increases during deep learning training, contrary to common belief.

Several deep models, esp. the generative, compare the samples from two distributions (e.g. WAE like AutoEncoder models, set-processing deep networks, etc) in their cost functions. Using all these methods one cannot train the model directly taking small size (in extreme -- one element) batches, due to the fact that samp…

In this work we develop a new algorithm for regularized empirical risk minimization. Our method extends recent techniques of Shalev-Shwartz [02/2015], which enable a dual-free analysis of SDCA, to arbitrary mini-batching schemes. Moreover, our method is able to better utilize the information in the data defining the ER…

mSAM improves generalization by making models flatter.

QC-ST and CoCo methods correct batch effects in metabolomics data.

Sources of variability in experimentally derived data include measurement error in addition to the physical phenomena of interest. This measurement error is a combination of systematic components, originating from the measuring instrument, and random measurement errors. Several novel biological technologies, such as ma…

Stochastic gradient descent (SGD) is almost ubiquitously used for training non-convex optimization tasks. Recently, a hypothesis proposed by Keskar et al. [2017] that large batch methods tend to converge to sharp minimizers has received increasing attention. We theoretically justify this hypothesis by providing new pro…

New method reduces training time for deep hedging networks.

GCBS improves contrastive learning performance efficiently.

This paper proposes a framework of L-BFGS based on the (approximate) second-order information with stochastic batches, as a novel approach to the finite-sum minimization problems. Different from the classical L-BFGS where stochastic batches lead to instability, we use a smooth estimate for the evaluations of the gradie…

SVRN accelerates Newton methods by reducing variance and improving performance.

We propose a projected semi-stochastic gradient descent method with mini-batch for improving both the theoretical complexity and practical performance of the general stochastic gradient descent method (SGD). We are able to prove linear convergence under weak strong convexity assumption. This requires no strong convexit…

New findings cast doubt on the role of $λ_{max}$ in generalizing neural networks.

Sparse feature selection improves batch RL efficiency.

Improves deep learning training by matching mini-batch distributions.

A new method reduces data movement in neural network training.

Most prior work on active learning of classifiers has focused on sequentially selecting one unlabeled example at a time to be labeled in order to reduce the overall labeling effort. In many scenarios, however, it is desirable to label an entire batch of examples at once, for example, when labels can be acquired in para…

We propose a new stochastic optimization framework for empirical risk minimization problems such as those that arise in machine learning. The traditional approaches, such as (mini-batch) stochastic gradient descent (SGD), utilize an unbiased gradient estimator of the empirical average loss. In contrast, we develop a co…