New optimizer improves privacy-protected hyperparameter tuning.
problem No practical methods for differentially private hyperparameter selection.
method Study honest hyperparameter selection under DP, show adaptive optimizers like DPAdam have an advantage.
result DPAdam optimizes hyperparameters more efficiently under DP constraints.
Self-Tuning Networks optimize hyperparameters using bilevel optimization and gated best-response functions.
problem Optimizing hyperparameters for neural networks.
method Bilevel optimization with gated best-response functions to adapt regularization hyperparameters online.
result Self-Tuning Networks outperform fixed hyperparameter values on large-scale deep learning problems.
New method optimizes unknown functions adaptively.
problem Tuning hyperparameters in machine learning.
method Gradually and adaptively optimizing an unknown function using estimated gradients.
result Advantages for tuning high dimensional hyperparameters in machine learning.
No-regret BO algorithm adapts hyperparameters to optimize unknown functions.
problem Misspecification of hyperparameters in BO leads to poor local optima.
method Adapts hyperparameters online to expand function class and converge to optimum.
result First provably no-regret BO algorithm with unknown hyperparameters.
Alpha MAML adapts MAML for fewer hyperparameter tuning.
problem Costly hyperparameter tuning in MAML.
method Online hyperparameter adaptation scheme.
result Substantial reduction in hyperparameter tuning needed.
POCA optimizes hyperparameters with adaptive allocation for faster convergence.
problem Optimizing hyperparameters for machine learning models.
method Adaptive allocation of computational budget using Bayesian sampling.
result POCA finds strong configurations faster than its competitors.
HASSO improves SO algorithms by dynamically tuning hyperparameters.
problem Inefficiency of hyperparameter tuning for SO algorithms.
method HASSO is a self-adjusting SO algorithm that dynamically tunes its own hyperparameters.
result HASSO enhances the performance of various SO algorithms across different test problems.
Adaptive online learning algorithms without manual tuning of a Lipschitz hyperparameter.
problem Designing adaptive online learning algorithms that require minimal user input and automatically adjust hyperparameters.
method Developing new versions of MetaGrad and Squint algorithms that dynamically update learning rates and adapt to the optimal Lipschitz hyperparameter.
result Automatic adaptation of the Lipschitz hyperparameter, improving performance and efficiency of online learning algorithms.
CrossTrainer adapts models across diverse datasets using loss reweighting.
problem Training models with data from varying sources or of varying quality.
method Loss reweighting with optimizations for hyperparameter tuning.
result High model accuracy across various datasets with improved training time.
The paper examines how optimizer comparisons in deep learning are influenced by hyperparameter tuning.
problem The sensitivity of optimizer comparisons to hyperparameter tuning protocols.
method Empirical comparisons of optimizers with and without varying hyperparameter search spaces.
result Inclusion relationships between optimizers matter in practice and can contradict recent empirical comparisons.
A meta-learning method learns adaptive robust loss functions for noisy labels.
problem Handling robust learning with noisy labels and optimizing hyperparameters.
method Adaptive learning of robust loss hyperparameters through mutual improvement with network parameters.
result Generalized and effective robust loss functions with good generalization capability.
Adaptive sampling method optimizes DNN compression for resource-constrained platforms.
problem Efficiently compressing DNNs for resource-constrained platforms with high accuracy.
method Adaptive sampling using genetic algorithm-inspired operations to optimize hyperparameters.
result Adaptive sampling outperforms rule-based and reinforcement learning methods in compression rate and accuracy.
Performance of machine learning algorithms depends critically on identifying a good set of hyperparameters. While recent approaches use Bayesian optimization to adaptively select configurations, we focus on speeding up random search through adaptive resource allocation and early-stopping. We formulate hyperparameter op…
HyperLISTA simplifies LISTA training with adaptive hyperparameters.
problem Sparse recovery with LISTA networks.
method Adaptive hyperparameter tuning based on previous layers.
result HyperLISTA achieves similar performance on seen data and better on unseen data.
AvaGrad optimizes vision tasks by decoupling learning rate and adaptability.
problem Improving optimization methods for vision tasks.
method Derives AvaGrad, a new optimizer that decouples learning rate and adaptability.
result AvaGrad outperforms SGD on vision tasks when adaptability is properly tuned.
Bayesian optimization reduces hyperparameters for mixed variable design problems.
problem Optimizing designs with a large number of mixed continuous, integer, and categorical variables.
method Adaptive dimension reduction using partial least squares for fewer hyperparameters.
result Significant improvement in performance compared to genetic algorithms.
aLTT selects hyperparameters efficiently with statistical guarantees.
problem Statistical validity and efficiency in hyperparameter selection.
method Sequential data-dependent multiple hypothesis testing with early termination.
result Reduces testing rounds while maintaining statistical validity.
Proposes GAGA algorithm for automatic hyperparameter learning in signal recovery.
problem Difficulty in selecting hyperparameters in traditional signal recovery methods.
method Global Adaptive Generative Adjustment (GAGA) algorithm for automatic hyperparameter learning and signal estimate.
result Consistency of model selection and signal estimate output.
Adapts multi-armed bandits to contextual bandits using logistic regression.
problem Online decision-making with contextual information and binary rewards.
method Adapts multi-armed bandits policies to contextual bandits using logistic regression and bootstrapping.
result Adaptive-Greedy algorithm shows better performance than upper confidence bound and Thompson sampling.
Adaptive speculative decoding framework for LLMs using bandit algorithms.
problem Adaptive speculative decoding for LLMs to balance speed and quality.
method Formulated as a Multi-Armed Bandit problem, proposed UCBSpec and EXP3Spec algorithms.
result UCBSpec algorithm achieves optimal regret performance up to universal constants.
New algorithm improves stability of optimization algorithms by adapting step-size.
problem Optimization algorithms' effectiveness is sensitive to step-size hyperparameters.
method Adapts NGN step-size method with momentum to enhance stability.
result Achieves convergence rate of O(1/√K) without restrictive assumptions.
Study separates learning rate effects from adaptive gradient methods.
problem Understanding the impact of learning rates on neural network training.
method Introduced a 'grafting' experiment to isolate learning rate effects.
result Many existing beliefs about adaptive gradient methods may be incorrect.
We propose an algorithm for a family of optimization problems where the objective can be decomposed as a sum of functions with monotonicity properties. The motivating problem is optimization of hyperparameters of machine learning algorithms, where we argue that the objective, validation error, can be decomposed as mono…
Bayesian optimization finds best hyperparameters for deep learning models.
problem Finding optimal hyperparameters in high-dimensional space.
method Bayesian optimization, Gaussian processes, and adaptive experimentation.
result Ax, BoTorch, and GPyTorch provide a powerful and simple framework for hyperparameter optimization.
Improved self-supervised denoising for Poisson-Gaussian noise.
problem Handling Poisson-Gaussian noise in self-supervised denoising.
method Extended blindspot model, improved training scheme without hyperparameters.
result Improved denoising performance on microscope image benchmarks.
Ortho-MADS optimizes SVM hyperparameters for better accuracy.
problem Optimizing hyperparameters for SVM with Gaussian kernel.
method Deterministic Mesh Adaptive Direct Search (MADS) with orthogonal directions (Ortho-MADS).
result Ortho-MADS consistently finds comparable or better solutions than other methods.
HOOF learns hyperparameters efficiently for policy gradients.
problem Efficient hyperparameter tuning for policy gradient methods.
method Gradient-free algorithm that optimizes hyperparameters directly through the gradient using existing trajectories.
result HOOF leads to faster learning and improved performance.
Self-Tuning Actor-Critic improves reinforcement learning performance.
problem Manual hyperparameter tuning is time-consuming and domain-specific.
method Uses metagradients for online hyperparameter adaptation.
result Improves performance across various domains and tasks.
FLAML automates model selection and hyperparameter tuning with low resource cost.
problem Automating model selection and hyperparameter tuning for ad-hoc datasets and metrics.
method Conducts trials of different configurations on training data, optimizing for low computational cost.
result Significantly outperforms top-ranked AutoML libraries under smaller budget constraints.
Extends hyperparameter transfer across model sizes and modules, improving training speed.
problem Training stability and performance of large-scale models with optimal hyperparameters.
method Complete(d) Parameterisation, per-module hyperparameter optimisation and transfer. result Hyperparameter transfer holds even in the per-module hyperparameter regime, improving training speed.
Efficiently tunes hyperparameters for online traffic time series prediction.
problem Online hyperparameter tuning for machine learning models in time series prediction.
method Online hyperparameter optimization algorithm for Kernel Ridge regression.
result Achieves better or similar prediction accuracy with significantly less computation time.
Two-Tailed Averaging improves generalization by optimizing the number of leading iterates to ignore.
problem Improving generalization in stochastic optimization with limited resources and hyperparameters.
method An anytime adaptive algorithm that balances the number of leading iterates to ignore for better generalization.
result Approximates the optimal tail at all optimization steps, improving generalization without hyperparameters.
KD improves DNN performance on unseen data without hyperparameter tuning.
problem Overfitting in DNNs on unseen data.
method Knowledge distillation for semi-supervised domain adaptation.
result KD achieves significantly higher WMH dice scores than baseline and ADA.
The paper improves Gaussian process regression by optimizing hyperparameters.
problem Hyperparameter tuning for Gaussian process regression models.
method Adaptive sparse variational approximations using variational Bayes.
result Minimax optimal rates of convergence for variational posterior.
Adaptive Bayesian model optimizes machine learning pipelines.
problem Automated model selection and hyperparameter tuning for datasets.
method Combines adaptive Bayesian regression and neural network basis function with acquisition function.
result Identifies high-performance pipelines efficiently and outperforms baseline methods.
Pipeline-aware hyperparameter tuning speeds up machine learning pipelines by reusing intermediate computations.
problem High computational burden in hyperparameter tuning of multi-stage pipelines.
method Proposes a hybrid hyperparameter tuning method and a caching problem formulated as an ILP to maximize reuse.
result Pipeline-aware approach offers over an order-of-magnitude speedup over independent evaluations.
This paper explores approximations for fully Bayesian Gaussian Process Regression.
problem Learning in Gaussian Process models through hyperparameter adaptation.
method Two approximation schemes: Hamiltonian Monte Carlo and Variational Inference.
result Predictive performance analysis on various benchmark datasets.
BLiE optimizes hyperparameters with theoretical guarantees and superior performance.
problem Hyperparameter optimization in machine learning.
method Lipschitz bandit approach exploiting Lipschitz continuity.
result BLiE finds ε-optimal hyperparameters with theoretical complexity.
HyperSTAR optimizes deep network hyperparameters for specific tasks.
problem Finding optimal hyperparameters for deep neural networks is computationally expensive and task-agnostic approaches are inefficient.
method HyperSTAR learns a dataset representation and a performance predictor from raw images, ranking and recommending hyperparameters conditioned on a joint dataset-hyperparameter space.
result HyperSTAR reduces HPO time by 50% on large-scale image classification datasets.
Proposes LAMA for principled SVDD hyperparameter estimation.
problem Estimating SVDD hyperparameters is difficult and heuristic methods are unreliable.
method LAMA (Local Active Min-Max Alignment) using active learning and kernel alignment.
result LAMA provides evidence-based estimates for SVDD hyperparameters with quality scores.
New method for adaptive estimation and inference in econometric models without knowing smoothness.
problem Adaptive estimation and inference in ill-posed linear inverse problems with unknown smoothness.
method Discrepancy principle-based framework for adaptive hyperparameter selection.
result Achieves optimal rates in weak and strong metrics for linear functionals.
Paper presents adaptive Lipschitz bandit framework for efficient optimization.
problem Optimizing rewards and minimizing regret in stochastic Lipschitz bandit problems.
method Adaptive learning of partitions in context- and arm-space using hierarchical Bayesian models.
result Achieves state-of-the-art performance in real-world tasks like neural network hyperparameter tuning.
New method FedEx accelerates federated hyperparameter tuning.
problem Federated hyperparameter tuning challenges in distributed learning.
method FedEx method connecting to weight-sharing, adapted for federated optimization.
result FedEx outperforms natural baselines on various benchmarks.
Novel hyperparameter optimization for target tasks under covariate shift.
problem Hyperparameter optimization under multi-source covariate shift.
method Construct variance reduced estimator to unbiasedly approximate target objective; propose no-regret hyperparameter optimization procedure.
result Proposed framework broadens applications of automated hyperparameter optimization.
Paper proposes a new method for efficient hyperparameter optimization.
problem Challenging task of optimizing hyperparameters in machine learning.
method Sequential Uniform Design (SeqUD) strategy for adaptive and efficient exploration of hyperparameter space.
result The proposed SeqUD strategy outperforms existing methods in hyperparameter optimization.
New algorithm adapts to optimize non-convex problems efficiently.
problem Inefficient hyperparameter tuning in existing optimization methods.
method Combining geometrization and SARAH algorithms.
result Achieves adaptivity to both accuracy and PL constant.
LOBSTUR-GNN adapts bootstrapping for unsupervised GNNs, improving node representation learning.
problem Hyperparameter tuning and lack of established methodologies for unsupervised GNNs.
method Adapts bootstrapping techniques for local graph dependencies and uses CCA for embedding consistency.
result 65.9% improvement in classification accuracy compared to uninformed hyperparameter selection.
This paper tackles hyperparameter tuning for large-scale kernel ridge regression.
problem Hyperparameter tuning is crucial but often left to users, hindering efficiency and usability.
method Proposes a complexity regularization criterion based on a data-dependent penalty for efficient optimization.
result Demonstrates the benefit of the proposed approach through extensive empirical evaluation.