CascadeML trains multi-label models without hyperparameter tuning.
problem Improving multi-label classification performance through hyperparameter optimization.
method CascadeML uses a two-phase incremental learning approach to train multi-label neural networks.
result CascadeML outperforms other multi-label classification algorithms without hyperparameter tuning.
Self-Tuning Networks optimize hyperparameters using bilevel optimization and gated best-response functions.
problem Optimizing hyperparameters for neural networks.
method Bilevel optimization with gated best-response functions to adapt regularization hyperparameters online.
result Self-Tuning Networks outperform fixed hyperparameter values on large-scale deep learning problems.
Proposes efficient multi-fidelity Bayesian optimization for deep neural network hyperparameter tuning.
problem Time-consuming validation error evaluation for hyperparameter tuning in deep neural networks.
method Introduces trace-aware knowledge-gradient acquisition function and a provably convergent optimization method.
result Outperforms state-of-the-art alternatives for hyperparameter tuning of deep neural networks.
New benchmark protocol evaluates neural network optimizers for efficiency and data shift sensitivity.
problem Benchmarking neural network optimizers with hyperparameter complexity and data shift sensitivity.
method Proposed a new evaluation protocol combining end-to-end and data-addition training efficiency, using bandit hyperparameter tuning and human study validation.
result No clear winner across all tasks, highlighting the complexity of optimizer performance.
HyperNOMAD automates deep neural network hyperparameter tuning.
problem Automating the calibration of hyperparameters in deep neural networks.
method Derivative-free optimization using MADS algorithm.
result Achieves comparable results to state-of-the-art methods.
Sherpa automates hyperparameter tuning for machine learning models.
problem Hyperparameter optimization for computationally expensive models.
method Powerful and interchangeable algorithms, interactive dashboard.
result Automates tedious aspects of model tuning.
Improved hypernetwork for efficient neural network hyperparameter tuning.
problem Efficiently optimizing hyperparameters in neural networks.
method Proposed Δ Δ Δ -STN architecture focusing on accurate best-response Jacobian approximation. result Significantly improved hyperparameter tuning accuracy and stability.
Bayesian optimization improves hyperparameter tuning for deep learning.
problem Efficient tuning of hyperparameters in deep learning is expensive and time-consuming.
method Bayesian optimization exploiting iterative learning progress.
result Our algorithm identifies optimal hyperparameters faster than existing methods.
This paper introduces an efficient method for optimizing deep learning hyperparameters.
problem The high dependency of deep learning algorithms on hyper-parameters.
method Orthogonal Array Tuning Method (OATM) for deep learning hyper-parameter tuning.
result The proposed OATM method significantly saves tuning time compared to state-of-the-art methods.
Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.
problem The challenge of hyperparameter tuning in deep learning, especially for large models.
method Combining μ μ μ P parameterization with residual networks having a residual branch scale of 1 / e x t d e p t h 1/\sqrt{ ext{depth}} 1/ e x t d e pt h . result Optimal hyperparameters transfer across width and depth in residual networks trained with this parameterization.
FIRE PBT improves neural network training by focusing on long-term performance.
problem Greedy decision mechanisms in PBT lead to poor long-term performance.
method FIRE PBT uses a fitness metric to encourage long-term performance over short-term improvements.
result FIRE PBT outperforms PBT on ImageNet and matches hand-tuned learning rates.
Bayesian optimization sped up with importance sampling.
problem Efficiently tuning hyperparameters for neural networks.
method Bayesian optimization with importance sampling.
result Significantly improved runtime and validation error.
This paper proposes reusing CNN layers to speed up hyperparameters tuning.
problem Time-consuming hyperparameters tuning in CNNs.
method Reuse trained convolutional layers among different trainings.
result Reduces training time and increases accuracy of neural networks.
Automatically tunes learning rate and momentum for SGD methods.
problem Manual hyperparameter tuning is costly and lacks theoretical justification.
method Uses statistics of gradient estimator to automatically adjust learning rate and momentum.
result Matches performance of best manual settings for CNN training.
AlgoPerf competition evaluates neural network training speed-ups.
problem Improving neural network training speed using better algorithms.
method Compared 18 diverse submissions from 10 teams on multiple workloads.
result Schedule Free AdamW algorithm achieved best results in self-tuning ruleset.
New method upscales models and transfers hyperparameters efficiently.
problem Efficiently scaling and tuning large neural networks.
method Introduces a general upscaling method and extends μ μ μ Transfer for hyperparameter tuning. result Effective hyperparameter transfer for upscaled models.
New method FedEx accelerates federated hyperparameter tuning.
problem Federated hyperparameter tuning challenges in distributed learning.
method FedEx method connecting to weight-sharing, adapted for federated optimization.
result FedEx outperforms natural baselines on various benchmarks.
Unified GP model optimizes hyperparameters with conditional dependence.
problem Efficient tuning of hyperparameters in neural networks.
method Unified Bayesian optimization framework based on a new Gaussian process (GP) model.
result Higher prediction accuracy and better optimization efficiency observed.
HyperLISTA simplifies LISTA training with adaptive hyperparameters.
problem Sparse recovery with LISTA networks.
method Adaptive hyperparameter tuning based on previous layers.
result HyperLISTA achieves similar performance on seen data and better on unseen data.
New method improves hyperparameter tuning efficiency across similar tasks.
problem Mismatch between evaluations in current and previous tasks.
method Nested drop-out and auto-relevance determination for learning basis functions of increasing complexity.
result Improves sample efficiency in hyperparameter tuning across different data regimes.
Proposes baselines for joint NAS and HPO optimization.
problem Joint optimization of neural architecture and hyperparameters for multiple objectives.
method Extends existing methods to jointly optimize with multiple objectives.
result Serves as simple baselines for future multi-objective joint NAS + HPO research.
HyperFast instantly classifies tabular data in one pass.
problem Computational burden and time-consuming hyperparameter tuning for deep learning models.
method Meta-trained hypernetwork for instant classification.
result Significantly faster than competing methods with competitive performance.
This work proposes a method to optimize hyperparameters without validation data.
problem Optimizing hyperparameters for neural networks with limited data.
method Neural network partitioning and subnetwork optimization.
result The method optimizes hyperparameters efficiently and without validation data.
Extends hyperparameter transfer across model sizes and modules, improving training speed.
problem Training stability and performance of large-scale models with optimal hyperparameters.
method Complete ( d ) ^{(d)} ( d ) Parameterisation, per-module hyperparameter optimisation and transfer. result Hyperparameter transfer holds even in the per-module hyperparameter regime, improving training speed.
Tuning hyperparameters of learning algorithms is hard because gradients are usually unavailable. We compute exact gradients of cross-validation performance with respect to all hyperparameters by chaining derivatives backwards through the entire training procedure. These gradients allow us to optimize thousands of hyper…
The paper examines how optimizer comparisons in deep learning are influenced by hyperparameter tuning.
problem The sensitivity of optimizer comparisons to hyperparameter tuning protocols.
method Empirical comparisons of optimizers with and without varying hyperparameter search spaces.
result Inclusion relationships between optimizers matter in practice and can contradict recent empirical comparisons.
LOBSTUR-GNN adapts bootstrapping for unsupervised GNNs, improving node representation learning.
problem Hyperparameter tuning and lack of established methodologies for unsupervised GNNs.
method Adapts bootstrapping techniques for local graph dependencies and uses CCA for embedding consistency.
result 65.9% improvement in classification accuracy compared to uninformed hyperparameter selection.
This paper optimizes neural network training by packing multiple models on a single GPU.
problem Efficiently sharing limited training resources among multiple neural network models.
method Proposes a primitive called 'pack' to jointly train multiple models on a single GPU.
result Significant performance improvements for hyperparameter tuning, up to 40% for two models.
GOLS-I automatically determines learning rates for various neural network training algorithms.
problem Adapting learning rates in stochastic training algorithms for neural networks.
method Gradient-Only Line Search (GOLS-I) for automatically setting learning rates.
result GOLS-I learning rate schedules are competitive with manually tuned rates across multiple algorithms, architectures, datasets, and loss functions.
AgEBO-Tabular combines NAS and hyperparameter tuning for fast, high-performing tabular models.
problem Developing high-performing predictive models for large tabular data sets is challenging.
method Combines aging evolution NAS and asynchronous Bayesian optimization for hyperparameter tuning in data-parallel training.
result Automatically discovered neural network models outperform state-of-the-art AutoML ensembles in inference speed by two orders of magnitude.
Develops hyperparameter transfer methods for Dense Associative Memories.
problem Challenges in transferring hyperparameters for DenseAMs due to unique architecture and activation functions.
method Derives explicit prescriptions for hyperparameter transfer from small to large models.
result Excellent agreement between theoretical and empirical results.
MR estimator simplifies causal inference by combining models without hyperparameter tuning.
problem Difficulty in choosing optimal hyperparameters for neural network models in causal inference.
method Multiply Robust (MR) estimator that combines multiple first-step models.
result MR estimator is n r n^r n r consistent and asymptotically normal under certain conditions. New hyperparameter ensembles boost neural network performance and uncertainty.
problem Improving neural network robustness and uncertainty quantification.
method Designing ensembles over both weights and hyperparameters, stratified across random initializations.
result Hyper-deep and hyper-batch ensembles outperform deep and batch ensembles on various architectures.
Conditional kernel mean embeddings are nonparametric models that encode conditional expectations in a reproducing kernel Hilbert space. While they provide a flexible and powerful framework for probabilistic inference, their performance is highly dependent on the choice of kernel and regularization hyperparameters. Neve…
Auptimizer simplifies hyperparameter tuning for machine learning models.
problem Difficulty and time-consuming hyperparameter tuning for machine learning models.
method General HPO framework that distributes computing resources and integrates various HPO techniques.
result Simplified model tuning and bookkeeping for data scientists.
New Muon and Momo variants improve neural network optimization robustness.
problem Improving neural network optimization methods.
method Systematic exploration of non-Euclidean gradient descent variants.
result Momo variants of Muon are more robust to hyperparameter tuning.
Paper presents adaptive Lipschitz bandit framework for efficient optimization.
problem Optimizing rewards and minimizing regret in stochastic Lipschitz bandit problems.
method Adaptive learning of partitions in context- and arm-space using hierarchical Bayesian models.
result Achieves state-of-the-art performance in real-world tasks like neural network hyperparameter tuning.
Systems based on artificial neural networks (ANNs) have achieved state-of-the-art results in many natural language processing tasks. Although ANNs do not require manually engineered features, ANNs have many hyperparameters to be optimized. The choice of hyperparameters significantly impacts models' performances. Howeve…
Unified framework for hyperparameter tuning in clustering problems.
problem Challenges in selecting hyperparameters for unsupervised learning, especially in clustering.
method A unified framework with provable guarantees for hyperparameter selection in various models.
result Framework outperforms other widely used tuning procedures in various settings.
New insights on stability in reservoir computing for better performance.
problem Stability in reservoir computing networks.
method Using the recurrent kernel limit for large reservoir sizes.
result Quantitative characterization of stability and chaos frontier.
Bayesian priors improve neural network performance on weak signals.
problem Challenges in encoding domain knowledge for weak signals in neural networks.
method Proposed a new joint prior over local scale parameters for feature sparsity and signal-to-noise ratio, optimized with Stein gradient.
result Improved prediction accuracy on various datasets, including genetics applications with weak and sparse signals.
A new method tunes LOF hyperparameters for better anomaly detection.
problem Tuning hyperparameters for LOF improves anomaly detection performance.
method Heuristic methodology to tune LOF hyperparameters.
result Tuned LOF model outperforms in simulations and real data sets.
Algorithm optimizes millions of hyperparameters efficiently.
problem Training modern network architectures with millions of hyperparameters.
method Combines implicit function theorem with efficient inverse Hessian approximations for gradient-based optimization.
result Jointly tuning weights and hyperparameters is only a few times more costly than standard training.
L2O-CFGD meta-learns hyperparameters for FGD, improving performance.
problem Challenges in convergence and hyperparameter selection for FGD.
method Learning to Optimize Caputo Fractional Gradient Descent (L2O-CFGD).
result Meta-learned schedule outperforms static hyperparameters and achieves comparable performance to black-box meta-learners.
This paper studies how optimizing Bayesian Optimization's own hyperparameters affects performance.
problem The impact of optimizing Bayesian Optimization's own hyperparameters.
method Empirical study using various benchmarks.
result Optimizing BO's own hyperparameters can improve any-time performance and generalize to similar problems.
AutoInit automatically finds good neural network initialization.
problem Finding optimal neural network initialization is crucial but time-consuming.
method Uses Jacobian tuning to automatically adjust network hyperparameters.
result The method finds good initialization for various network architectures.
PASHA optimizes model tuning for large datasets with limited resources.
problem Expensive HPO and NAS for large datasets.
method Dynamic resource allocation approach.
result Significantly reduces computational resources while maintaining performance.
KD improves DNN performance on unseen data without hyperparameter tuning.
problem Overfitting in DNNs on unseen data.
method Knowledge distillation for semi-supervised domain adaptation.
result KD achieves significantly higher WMH dice scores than baseline and ADA.