arXiv research

Extends hyperparameter transfer across model sizes and modules, improving training speed.

Develops hyperparameter transfer methods for Dense Associative Memories.

This paper quantifies hyperparameter transfer and finds embedding layer learning rate is key.

When hyperparameter optimization of a machine learning algorithm is repeated for multiple datasets it is possible to transfer knowledge to an optimization run on a new dataset. We develop a new hyperparameter-free ensemble model for Bayesian optimization that is a generalization of two existing transfer learning extens…

Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.

New framework explains fast transfer of hyperparameters across model scales.

Recently, several optimization methods have been successfully applied to the hyperparameter optimization of deep neural networks (DNNs). The methods work by modeling the joint distribution of hyperparameter values and corresponding error. Those methods become less practical when applied to modern DNNs whose training ma…

New method upscales models and transfers hyperparameters efficiently.

nGPT learns to transfer learning rates across model dimensions and token horizons.

A cost-efficient method for hyperparameter tuning using multi-fidelity Bayesian optimization.

Transfer learning improves prediction quality in high-dimensional sparse regression.

Transfer learning aims at transferring knowledge from a well-labeled domain to a similar but different domain with limited or no labels. Unfortunately, existing learning-based methods often involve intensive model selection and hyperparameter tuning to obtain good results. Moreover, cross-validation is not possible for…

Bayesian optimisation is a popular technique for hyperparameter learning but typically requires initial exploration even in cases where similar prior tasks have been solved. We propose to transfer information across tasks using learnt representations of training datasets used in those tasks. This results in a joint Gau…

Automated machine learning aims to automate the whole process of machine learning, including model configuration. In this paper, we focus on automated hyperparameter optimization (HPO) based on sequential model-based optimization (SMBO). Though conventional SMBO algorithms work well when abundant HPO trials are availab…

New method improves hyperparameter tuning efficiency across similar tasks.

The study analyzes deep linear networks from random initialization, capturing dynamics and hyperparameter effects.

New study reveals surprising adaptive rates in model selection for transfer learning.

Bayesian optimization (BO) is a popular methodology to tune the hyperparameters of expensive black-box functions. Traditionally, BO focuses on a single task at a time and is not designed to leverage information from related functions, such as tuning performance objectives of the same algorithm across multiple datasets.…

Trans-GCR uses GCR model for node classification, providing theoretical guarantees and superior performance.

Meta-models predict model hyperparameters for NDT experiments.

Develops methods to measure and set function-space learning rates in neural networks.

Bayesian Optimization (BO) is a common approach for hyperparameter optimization (HPO) in automated machine learning. Although it is well-accepted that HPO is crucial to obtain well-performing machine learning models, tuning BO's own hyperparameters is often neglected. In this paper, we empirically study the impact of o…

New method saves computational budget by ranking and transferring learning curves.

Simplifies transfer learning with deep neural networks using ridge regression.

Bayesian optimization (BO) is a successful methodology to optimize black-box functions that are expensive to evaluate. While traditional methods optimize each black-box function in isolation, there has been recent interest in speeding up BO by transferring knowledge across multiple related black-box functions. In this …

Muon optimizes training efficiency by improving data retention at large batch sizes.

Agents trained with deep reinforcement learning algorithms are capable of performing highly complex tasks including locomotion in continuous environments. We investigate transferring the learning acquired in one task to a set of previously unseen tasks. Generalization and overfitting in deep reinforcement learning are …

Fine-tuning from pre-trained ImageNet models has become the de-facto standard for various computer vision tasks. Current practices for fine-tuning typically involve selecting an ad-hoc choice of hyperparameters and keeping them fixed to values normally used for training from scratch. This paper re-examines several comm…

New method $μP^2$ improves neural network training by scaling perturbations layerwise.

A modular GP framework for efficient transfer learning.

Learning from small amounts of labeled data is a challenge in the area of deep learning. This is currently addressed by Transfer Learning where one learns the small data set as a transfer task from a larger source dataset. Transfer Learning can deliver higher accuracy if the hyperparameters and source dataset are chose…

A new method learns priors for Bayesian optimisation to improve performance.

PowerGossip compresses model differences for decentralized deep learning with low-rank linear compressors.

This work presents an approach to automatically induction for non-greedy decision trees constructed from neural network architecture. This construction can be used to transfer weights when growing or pruning a decision tree, allowing non-greedy decision tree algorithms to automatically learn and adapt to the ideal arch…

Automates RL with sample-efficient hyperparameter optimization.

Meta-learning improves neural networks by adapting learning algorithms.

We present an efficient block-diagonal ap- proximation to the Gauss-Newton matrix for feedforward neural networks. Our result- ing algorithm is competitive against state- of-the-art first order optimisation methods, with sometimes significant improvement in optimisation performance. Unlike first-order methods, for whic…

Optimal scaling found to depend on operator norm across large models and datasets.

The paper proposes a method to learn hyperparameters without validation sets, improving efficiency and accuracy.

Proposes learning regularization strength directly from data.

Randomized Geometric Algebra for Convex Neural Networks Optimizes Transfer Learning.

Learning-to-learn or meta-learning leverages data-driven inductive bias to increase the efficiency of learning on a novel task. This approach encounters difficulty when transfer is not advantageous, for instance, when tasks are considerably dissimilar or change over time. We use the connection between gradient-based me…

Unified framework for transfer learning regression without extra cost.

Two new techniques improve zero-shot HPO efficiency.

Transferring knowledge across tasks to improve data-efficiency is one of the open key challenges in the field of global black-box optimization. Readily available algorithms are typically designed to be universal optimizers and, therefore, often suboptimal for specific tasks. We propose a novel transfer learning method …

DeltaBO accelerates Bayesian optimization with theoretical guarantees.

Ensemble pre-trained models for low data transfer learning.

New study finds optimal hyperparameter tuning crucial for fair optimizer comparisons.