arXiv research

This paper quantifies hyperparameter transfer and finds embedding layer learning rate is key.

Extends hyperparameter transfer across model sizes and modules, improving training speed.

When hyperparameter optimization of a machine learning algorithm is repeated for multiple datasets it is possible to transfer knowledge to an optimization run on a new dataset. We develop a new hyperparameter-free ensemble model for Bayesian optimization that is a generalization of two existing transfer learning extens…

Recently, several optimization methods have been successfully applied to the hyperparameter optimization of deep neural networks (DNNs). The methods work by modeling the joint distribution of hyperparameter values and corresponding error. Those methods become less practical when applied to modern DNNs whose training ma…

nGPT learns to transfer learning rates across model dimensions and token horizons.

Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.

Transfer learning improves prediction quality in high-dimensional sparse regression.

Transfer learning aims at transferring knowledge from a well-labeled domain to a similar but different domain with limited or no labels. Unfortunately, existing learning-based methods often involve intensive model selection and hyperparameter tuning to obtain good results. Moreover, cross-validation is not possible for…

A cost-efficient method for hyperparameter tuning using multi-fidelity Bayesian optimization.

Develops hyperparameter transfer methods for Dense Associative Memories.

New framework explains fast transfer of hyperparameters across model scales.

A new method for hyperparameter tuning across multiple datasets and objectives.

Automated machine learning aims to automate the whole process of machine learning, including model configuration. In this paper, we focus on automated hyperparameter optimization (HPO) based on sequential model-based optimization (SMBO). Though conventional SMBO algorithms work well when abundant HPO trials are availab…

Bayesian optimisation is a popular technique for hyperparameter learning but typically requires initial exploration even in cases where similar prior tasks have been solved. We propose to transfer information across tasks using learnt representations of training datasets used in those tasks. This results in a joint Gau…

New method improves hyperparameter tuning efficiency across similar tasks.

New method upscales models and transfers hyperparameters efficiently.

Trans-GCR uses GCR model for node classification, providing theoretical guarantees and superior performance.

New study reveals surprising adaptive rates in model selection for transfer learning.

New method optimizes BO by learning search spaces from past evaluations.

The study analyzes deep linear networks from random initialization, capturing dynamics and hyperparameter effects.

Meta-models predict model hyperparameters for NDT experiments.

Develops methods to measure and set function-space learning rates in neural networks.

New method saves computational budget by ranking and transferring learning curves.

Bayesian Optimization (BO) is a common approach for hyperparameter optimization (HPO) in automated machine learning. Although it is well-accepted that HPO is crucial to obtain well-performing machine learning models, tuning BO's own hyperparameters is often neglected. In this paper, we empirically study the impact of o…

Simplifies transfer learning with deep neural networks using ridge regression.

Agents trained with deep reinforcement learning algorithms are capable of performing highly complex tasks including locomotion in continuous environments. We investigate transferring the learning acquired in one task to a set of previously unseen tasks. Generalization and overfitting in deep reinforcement learning are …

This paper re-evaluates hyperparameters for fine-tuning pre-trained models.

Learning from small amounts of labeled data is a challenge in the area of deep learning. This is currently addressed by Transfer Learning where one learns the small data set as a transfer task from a larger source dataset. Transfer Learning can deliver higher accuracy if the hyperparameters and source dataset are chose…

Meta-learning improves neural networks by adapting learning algorithms.

A new method learns priors for Bayesian optimisation to improve performance.

A modular GP framework for efficient transfer learning.

PowerGossip compresses model differences for decentralized deep learning with low-rank linear compressors.

Learning-to-learn or meta-learning leverages data-driven inductive bias to increase the efficiency of learning on a novel task. This approach encounters difficulty when transfer is not advantageous, for instance, when tasks are considerably dissimilar or change over time. We use the connection between gradient-based me…

New method $μP^2$ improves neural network training by scaling perturbations layerwise.

Automates RL with sample-efficient hyperparameter optimization.

Proposes learning regularization strength directly from data.

Randomized Geometric Algebra for Convex Neural Networks Optimizes Transfer Learning.

The paper proposes a method to learn hyperparameters without validation sets, improving efficiency and accuracy.

Unified framework for transfer learning regression without extra cost.

Optimal scaling found to depend on operator norm across large models and datasets.

Transferring knowledge across tasks to improve data-efficiency is one of the open key challenges in the field of global black-box optimization. Readily available algorithms are typically designed to be universal optimizers and, therefore, often suboptimal for specific tasks. We propose a novel transfer learning method …

This work presents an approach to automatically induction for non-greedy decision trees constructed from neural network architecture. This construction can be used to transfer weights when growing or pruning a decision tree, allowing non-greedy decision tree algorithms to automatically learn and adapt to the ideal arch…

A RL-enhanced quantum-inspired algorithm solves combinatorial optimization problems.

Two new techniques improve zero-shot HPO efficiency.

We present an efficient block-diagonal ap- proximation to the Gauss-Newton matrix for feedforward neural networks. Our result- ing algorithm is competitive against state- of-the-art first order optimisation methods, with sometimes significant improvement in optimisation performance. Unlike first-order methods, for whic…

Ensemble pre-trained models for low data transfer learning.

MetaPerturb learns to improve generalization across different tasks and architectures.

RaNNDy uses randomized neural networks to learn transfer operators efficiently.