arXiv research

Distributed sensors compress and send features to a fusion center for linear regression.

GALA adapts learning rates online by aligning gradients, improving deep learning model performance.

New algorithm optimizes beam and rate allocation in mmWave systems for multiple users.

Master algorithm selects best contextual bandit from a collection.

Wireless systems perform rate adaptation to transmit at highest possible instantaneous rates. Rate adaptation has been increasingly granular over generations of wireless systems. The base-station uses SINR and packet decode feedback called acknowledgement/no acknowledgement (ACK/NACK) to perform rate adaptation. SINR i…

POLA adapts learning rates for online time series prediction.

Proposes a continuous flow model to understand and control instability in gradient descent for deep learning.

Our paper introduces an efficient combination of established techniques to improve classifier performance, in terms of accuracy and training time. We achieve two-fold to ten-fold speedup in nearing state of the art accuracy, over different model architectures, by dynamically tuning the learning rate. We find it especia…

We propose an algorithm for the adaptation of the learning rate for stochastic gradient descent (SGD) that avoids the need for validation set use. The idea for the adaptiveness comes from the technique of extrapolation: to get an estimate for the error against the gradient flow which underlies SGD, we compare the resul…

Study separates learning rate effects from adaptive gradient methods.

We introduce a general method for improving the convergence rate of gradient-based optimizers that is easy to implement and works well in practice. We demonstrate the effectiveness of the method in a range of optimization problems by applying it to stochastic gradient descent, stochastic gradient descent with Nesterov …

Develops a parameter-free SGD algorithm with optimal convergence rate.

Study minimax estimation of stratified structure from i.i.d. samples.

Bayesian mixture models are widely applied for unsupervised learning and exploratory data analysis. Markov chain Monte Carlo based on Gibbs sampling and split-merge moves are widely used for inference in these models. However, both methods are restricted to limited types of transitions and suffer from torpid mixing and…

Most optimizers including stochastic gradient descent (SGD) and its adaptive gradient derivatives face the same problem where an effective learning rate during the training is vastly different. A learning rate scheduling, mostly tuned by hand, is usually employed in practice. In this paper, we propose CProp, a gradient…

Pre-conditioning is a well-known concept that can significantly improve the convergence of optimization algorithms. For noise-free problems, where good pre-conditioners are not known a priori, iterative linear algebra methods offer one way to efficiently construct them. For the stochastic optimization problems that dom…

Bayesian PINNs learn elliptic PDEs with near-minimax posterior contraction rate.

The paper studies the benefits of curriculum learning in linear regression tasks.

Bayesian inference for Levy density with Gibbs posterior in discrete sampling.

Bayesian neural networks achieve optimal posterior contraction rates in Besov spaces with intrinsic dimensionality.

Prodigy estimates learning rate without tuning, improving convergence.

New method estimates hidden binary mixture model centers efficiently.

Characterizes corridors in loss surfaces for gradient-based optimization.

Layer normalization (LayerNorm) has been successfully applied to various deep neural networks to help stabilize training and boost model convergence because of its capability in handling re-centering and re-scaling of both inputs and weight matrix. However, the computational overhead introduced by LayerNorm makes these…

Bayesian deep learning with heavy-tailed weights achieves near-optimal performance.

New learned optimizers outperform baselines by incorporating known and novel mechanisms.

Continual learning aims to learn new tasks without forgetting previously learned ones. This is especially challenging when one cannot access data from previous tasks and when the model has a fixed capacity. Current regularization-based continual learning algorithms need an external representation and extra computation …

Functional BART adds shape priors to Bayesian tree regression for better curve fitting.

We introduce a new sparse estimator of the covariance matrix for high-dimensional models in which the variables have a known ordering. Our estimator, which is the solution to a convex optimization problem, is equivalently expressed as an estimator which tapers the sample covariance matrix by a Toeplitz, sparsely-banded…

Adaptive weights improve physics-informed neural networks and deep operator networks.

New algorithms optimize private convex optimization with faster rates for functions with κ-growth.

AdaScale SGD adapts learning rates for large-batch training efficiently.

Paper tackles adversarial attacks on nonparametric regression models.

Paper studies federated nonparametric testing with privacy constraints, achieving optimal rates and adaptive testing.

A new method for training GNNs without a teacher model.

Transfer learning for bandits with latent Lipschitz continuity.

Proposes flexible auto-encoders for varying data dimensions.

Learning rates in stochastic neural network training are currently determined a priori to training, using expensive manual or automated iterative tuning. This study proposes gradient-only line searches to resolve the learning rate for neural network training algorithms. Stochastic sub-sampling during training decreases…

Seesaw optimizes training by balancing learning rate and batch size, accelerating model pretraining.

Novel method estimates complex nonlinear systems with stochastic differential equations.

AdaPT-GMM improves multiple testing power with covariates.

Adaptive variational Bayes framework improves inference adaptively.

Self-attention optimizers converge to optimal weights, revealing bias patterns.

Recurrent neural networks are powerful models for processing sequential data, but they are generally plagued by vanishing and exploding gradient problems. Unitary recurrent neural networks (uRNNs), which use unitary recurrence matrices, have recently been proposed as a means to avoid these issues. However, in previous …

CAP algorithm controls FCR in online selective prediction.

Improved convergence rate for kNN graph Laplacians with adaptive bandwidth.