arXiv research

Extends hyperparameter transfer across model sizes and modules, improving training speed.

Derives a family of hyperparameter scaling strategies for neural networks.

This paper quantifies hyperparameter transfer and finds embedding layer learning rate is key.

This paper tackles hyperparameter tuning for large-scale kernel ridge regression.

Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.

FDS tackles long horizon hyperparameter optimization issues.

The paper simplifies quickshift hyperparameter tuning for larger images.

Modern learning models are characterized by large hyperparameter spaces and long training times. These properties, coupled with the rise of parallel computing and the growing demand to productionize machine learning workloads, motivate the need to develop mature hyperparameter optimization functionality in distributed …

Mango automates hyperparameter tuning for large-scale ML training.

New framework explains fast transfer of hyperparameters across model scales.

New algorithm reduces regret bounds for Bayesian optimization with unknown hyperparameters.

New method upscales models and transfers hyperparameters efficiently.

New algorithm improves Gaussian process hyperparameter tuning for large datasets.

Aims to describe neural network training dynamics using two-time-scale models.

Field theory explains optimal scaling in ResNets for signal propagation.

Bayesian optimization reduces hyperparameter tuning cost for stochastic models.

EvoGrad improves efficiency in meta-learning and hyperparameter optimization.

New study finds optimal hyperparameter tuning crucial for fair optimizer comparisons.

HET-XL improves heteroscedastic classifiers for large-scale image classification.

Hyperparameter optimization can be formulated as a bilevel optimization problem, where the optimal parameters on the training set depend on the hyperparameters. We aim to adapt regularization hyperparameters for neural networks by fitting compact approximations to the best-response function, which maps hyperparameters …

New method $μP^2$ improves neural network training by scaling perturbations layerwise.

Efficient auto-tuning for DR hyperparameters with BO.

Modern supervised machine learning algorithms involve hyperparameters that have to be set before running them. Options for setting hyperparameters are default values from the software package, manual configuration by the user or configuring them for optimal predictive performance by a tuning procedure. The goal of this…

Efficient implicit differentiation for Lasso hyperparameter optimization.

Differentially private hyperparameter tuning improves privacy in machine learning.

Develops hyperparameter transfer methods for Dense Associative Memories.

A new hyperparameter optimization method reduces overfitting.

New framework for understanding infinite-width neural networks.

New method selects best offline RL policies from logged data.

This research provides theoretical guarantees for hyperparameter estimation in complex network dynamical systems.

HyperSTAR optimizes deep network hyperparameters for specific tasks.

This study analyzes quantization in deep learning models using statistical physics methods.

Most Machine Learning (ML) methods, from clustering to classification, rely on a distance function to describe relationships between datapoints. For complex datasets it is hard to avoid making some arbitrary choices when defining a distance function. To compare images, one must choose a spatial scale, for signals, a te…

Knowledge graph embeddings rank among the most successful methods for link prediction in knowledge graphs, i.e., the task of completing an incomplete collection of relational facts. A downside of these models is their strong sensitivity to model hyperparameters, in particular regularizers, which have to be extensively …

Mixed precision training (MPT) is becoming a practical technique to improve the speed and energy efficiency of training deep neural networks by leveraging the fast hardware support for IEEE half-precision floating point that is available in existing GPUs. MPT is typically used in combination with a technique called los…

We study two procedures (reverse-mode and forward-mode) for computing the gradient of the validation error with respect to the hyperparameters of any iterative learning algorithm such as stochastic gradient descent. These procedures mirror two methods of computing gradients for recurrent neural networks and have differ…

Framework for uncertainty estimation in training parameters.

Study analyzes neural network models to understand generalization performance.

New method improves uncertainty quantification in latent variable models.

Prevalidated ridge regression simplifies logistic regression for high-dimensional data.

Develops methods to measure and set function-space learning rates in neural networks.

New tools reveal simple structure in complex hyperparameter loss surfaces near optima.

Bayesian optimization is popular for optimizing time-consuming black-box objectives. Nonetheless, for hyperparameter tuning in deep neural networks, the time required to evaluate the validation error for even a few hyperparameter settings remains a bottleneck. Multi-fidelity optimization promises relief using cheaper p…

Reshuffling splits improves hyperparameter optimization's generalization performance.

Study shows data attribution methods are sensitive to hyperparameters, making tuning costly.

Scalable tools for nested optimization in deep learning.

Recurrent neural networks (RNNs) have been successfully used on a wide range of sequential data problems. A well known difficulty in using RNNs is the \textit{vanishing or exploding gradient} problem. Recently, there have been several different RNN architectures that try to mitigate this issue by maintaining an orthogo…

Efficiently identifies promising hyperparameters for online learning models.