Efficiently tunes hyperparameters with dynamic accuracy method.
problem Optimizing machine learning hyperparameters with inexact evaluations.
method Dynamic accuracy derivative-free optimization for hyperparameter tuning.
result Demonstrates robust and efficient hyperparameter tuning compared to fixed accuracy methods.
New algorithms optimize without tuning, matching tuned SGD performance.
problem Optimizing machine learning models without manual hyperparameter tuning.
method Formalizes tuning-free algorithms for matching SGD performance with loose hints.
result Tuning-free algorithms can match SGD performance, but not optimal convergence rates.
In this paper, we address the challenging problem of selecting tuning parameters for high-dimensional sparse regression. We propose a simple and computationally efficient method, called path thresholding (PaTh), that transforms any tuning parameter-dependent sparse regression algorithm into an asymptotically tuning-fre…
New algorithms eliminate stepsize tuning for bilevel optimization problems.
problem Bilevel optimization problems with unknown parameters and stepsizes.
method D-TFBO and S-TFBO algorithms with adaptive stepsizes.
result Achieve performance comparable to well-tuned approaches with theoretical guarantees.
New method improves hyperparameter tuning efficiency across similar tasks.
problem Mismatch between evaluations in current and previous tasks.
method Nested drop-out and auto-relevance determination for learning basis functions of increasing complexity.
result Improves sample efficiency in hyperparameter tuning across different data regimes.
Cyclical learning rates improve DRL performance without manual tuning.
problem Manual hyperparameter tuning in DRL is time-consuming and error-prone.
method Proposes cyclical learning rates for DRL problems.
result Cyclical learning achieves similar or better results than fixed learning rates.
Bayesian optimization tackles non-smooth tuning problems.
problem Optimizing black-box functions with non-smoothness and limited samples.
method Proposed a clustered Gaussian process (cGP) model for non-smooth optimization.
result Improvement of up to 90% in performance for repetitive experiments.
We study a budgeted hyper-parameter tuning problem, where we optimize the tuning result under a hard resource constraint. We propose to solve it as a sequential decision making problem, such that we can use the partial training progress of configurations to dynamically allocate the remaining budget. Our algorithm combi…
New method for tuning Graphical Lasso hyperparameters.
problem Tuning hyperparameters of Graphical Lasso.
method Bilevel optimization with first-order method.
result Derivation of Graphical Lasso Jacobian.
The paper introduces a Hessian-based method to improve generalization in fine-tuned deep neural networks.
problem Improving generalization in fine-tuned deep neural networks, especially in noisy conditions.
method PAC-Bayesian analysis to identify a Hessian-based distance measure, proving generalization bounds, and developing an algorithm with a generalization error guarantee.
result Hessian-based distance measure correlates well with observed generalization gaps and can match the scale of these gaps in practice.
Adjoint Matching improves flow and diffusion models with reward fine-tuning.
problem Improving generative models with reward fine-tuning.
method Casting reward fine-tuning as stochastic optimal control (SOC) and enforcing a specific noise schedule.
result Adjoint Matching outperforms existing SOC algorithms.
New approach uses PDE learning for faster RL fine-tuning.
problem Learning optimal control policy for diffusion process.
method Solves variational inequality based on HJB equations.
result Shows fine-tuning can be done via supervised regression.
We solve ElasticNet regularization tuning across multiple instances with provable guarantees.
problem Tuning ElasticNet regularization coefficients across multiple problem instances.
method Characterized ElasticNet loss as a piecewise-rational function, derived structural complexity bounds, and showed generalization and online learning guarantees.
result First general learning-theoretic guarantees for ElasticNet tuning without strong data distribution assumptions.
New method reduces fine-tuning cost for reused models.
problem Repeating fine-tuning costs with outdated foundation models.
method Portable Reward Tuning (PRT) trains a reward model to maximize the same loss function as fine-tuning.
result PRT achieves comparable accuracy to inference-time tuning with less inference cost.
In this short paper we investigate whether meta-learning techniques can be used to more effectively tune the hyperparameters of machine learning models using successive halving (SH). We propose a novel variant of the SH algorithm (MeSH), that uses meta-regressors to determine which candidate configurations should be el…
HASSO improves SO algorithms by dynamically tuning hyperparameters.
problem Inefficiency of hyperparameter tuning for SO algorithms.
method HASSO is a self-adjusting SO algorithm that dynamically tunes its own hyperparameters.
result HASSO enhances the performance of various SO algorithms across different test problems.
This paper proposes the first-ever algorithmic framework for tuning hyper-parameters of stochastic optimization algorithm based on reinforcement learning. Hyper-parameters impose significant influences on the performance of stochastic optimization algorithms, such as evolutionary algorithms (EAs) and meta-heuristics. Y…
Improved UCB method for stochastic bandits using distance tuning.
problem Improving performance in stochastic bandit problems.
method Tuning confidence bounds based on bandit distance.
result Empirically shows increased performance compared to existing methods.
Improved code translation by preserving structure with composed fine-tuning.
problem Improving code translation accuracy with unlabeled code outputs.
method Pre-trained denoiser to capture output structure, composed fine-tuning to fine-tune predictor.
result Composed fine-tuning significantly improves generalization over standard fine-tuning.
New algorithms for sampling and optimization without tuning.
problem Efficient sampling and optimization over probability measures.
method Optimization on the space of probability measures, using gradient flows.
result Strong theoretical guarantees and similar performance to optimally tuned algorithms.
A new method optimizes diffusion models for fine-tuning tasks efficiently.
problem Optimizing diffusion models for downstream tasks using nested bilevel structures.
method Formalizes the challenge as a generative bilevel optimization problem and introduces a first-order bilevel framework.
result Our method outperforms existing fine-tuning and hyperparameter search baselines.
New method for hyperparameter tuning in sparse matrix factorization.
problem Hyperparameter tuning in sparse matrix factorization.
method Numerical method based on evaluating the zero point of normalization factor in sparse matrix prior.
result Our method outperforms existing algorithms in ground-truth sparse matrix reconstruction.
Solution to sparse PCA tuning problem using Empirical Bayes.
problem Sparse PCA multiple tuning problem (MTP).
method Empirical Bayes covariance decomposition for penalized PCA.
result Empirical Bayes approach efficiently solves MTP in sparse PCA.
BBVI relies on adaptive stochastic optimization algorithms for posterior approximation, but these require extensive tuning.
problem BBVI posterior approximation
method Adaptive stochastic optimization algorithms
result No single method dominates, but a selection of 5 suffices.
Unified approach to linear regression using covariance fitting for optimal weights.
problem Finding optimal weights for linear regression models when weights are unknown.
method Covariance fitting SPICE-methodology to obtain data-adaptive weights.
result Tuned versions of known regularized estimators are unified under a common approach.
The paper proposes a new stochastic intervention control model conducted in various commodity and stock markets. The essence of the phenomenon of intervention is described in accordance with current economic theory. A review of papers on intervention research has been made. A general construction of the stochastic inte…
This paper tackles hyperparameter tuning for large-scale kernel ridge regression.
problem Hyperparameter tuning is crucial but often left to users, hindering efficiency and usability.
method Proposes a complexity regularization criterion based on a data-dependent penalty for efficient optimization.
result Demonstrates the benefit of the proposed approach through extensive empirical evaluation.
PID control architectures are widely used in industrial applications. Despite their low number of open parameters, tuning multiple, coupled PID controllers can become tedious in practice. In this paper, we extend PILCO, a model-based policy search framework, to automatically tune multivariate PID controllers purely bas…
PANDA improves linear discriminant analysis in high dimensions with minimal tuning.
problem Linear discriminant analysis in high-dimensional settings.
method PANDA: a tuning-insensitive method for linear discriminant analysis.
result PANDA achieves optimal convergence rates in estimation error and misclassification rate.
Motivated by the problem of tuning hyperparameters in machine learning, we present a new approach for gradually and adaptively optimizing an unknown function using estimated gradients. We validate the empirical performance of the proposed idea on both low and high dimensional problems. The experimental results demonstr…
A framework for auto-tuning hyper-parameters in contextual bandit algorithms.
problem Auto-tuning hyper-parameters in real-time for contextual bandit algorithms.
method Proposes a Syndicated Bandits framework to learn multiple hyper-parameters dynamically.
result Achieves optimal regret bounds under certain scenarios and handles multiple contextual bandit algorithms.
A common analytical problem in neuroscience is the interpretation of neural activity with respect to sensory input or behavioral output. This is typically achieved by regressing measured neural activity against known stimuli or behavioral variables to produce a "tuning function" for each neuron. Unfortunately, because …
New method FedEx accelerates federated hyperparameter tuning.
problem Federated hyperparameter tuning challenges in distributed learning.
method FedEx method connecting to weight-sharing, adapted for federated optimization.
result FedEx outperforms natural baselines on various benchmarks.
Hyperparameter tuning of multi-stage pipelines introduces a significant computational burden. Motivated by the observation that work can be reused across pipelines if the intermediate computations are the same, we propose a pipeline-aware approach to hyperparameter tuning. Our approach optimizes both the design and exe…
Bayesian optimisation tackles stochastic MPC hyper-parameter tuning.
problem Fine-tuning hyper-parameters in stochastic MPC models.
method Heteroscedastic Bayesian optimisation framework.
result Framework effectively tunes hyper-parameters in control problems.
MLtuner automatically tunes settings for training tunables (such as the learning rate, the momentum, the mini-batch size, and the data staleness bound) that have a significant impact on large-scale machine learning (ML) performance. Traditionally, these tunables are set manually, which is unsurprisingly error-prone and…
New algorithms learn latent variable models without tuning, outperforming existing methods.
problem Learning latent variable models without manual tuning.
method Two particle-based algorithms using free energy minimization and coin betting.
result Learning algorithms are entirely tuning-free and competitive with existing methods.
This paper shows RL with KL penalties is equivalent to Bayesian inference for fine-tuning LMs.
problem Fine-tuning large language models to avoid undesirable features.
method Analyzed KL-regularized RL and showed it's equivalent to variational inference.
result KL-regularized RL avoids distribution collapse and is more insightful as Bayesian inference.
Improved forecast accuracy for energy systems through decision-focused fine-tuning.
problem Challenges in integrating forecast values into time series models for diverse and specific instances.
method Decision-focused fine-tuning within time series foundation models for dispatchable feeder optimization.
result Improvement of 9.45% in average total daily costs.
This paper proposes automatic tuning of Bayesian Optimization's acquisition function.
problem Optimizing black-box functions with noisy, expensive evaluations and hyperparameter tuning.
method Exploring heuristics to automatically tune acquisition functions in Bayesian Optimization.
result Demonstrates effectiveness of heuristics in automatic Bayesian Optimization.
In high-dimensional and/or non-parametric regression problems, regularization (or penalization) is used to control model complexity and induce desired structure. Each penalty has a weight parameter that indicates how strongly the structure corresponding to that penalty should be enforced. Typically the parameters are c…
The paper analyzes the statistical cost of tuning kernel hyperparameters in robust regression.
problem Finding the best interpolant from a class of kernels with unknown hyperparameters under adversarial noise.
method Finite-sample guarantees, subsampling guarantee for linear regression, ε-net argument for discretizing kernel parameterizations.
result Hyperparameter optimization increases sample complexity by just a logarithmic factor, compared to known parameters.
Improved financial VA intent classification accuracy.
problem Determining user intents for unseen open intents.
method Supervised pre-training of intent representations using prefix-tuning and fine-tuning.
result 1.63% - 2.07% higher accuracy on banking77 benchmark.
When applying Machine Learning techniques to problems, one must select model parameters to ensure that the system converges but also does not become stuck at the objective function's local minimum. Tuning these parameters becomes a non-trivial task for large models and it is not always apparent if the user has found th…
Convex clustering solves a stable optimization problem for clustering.
problem Clustering with stable and scalable solutions.
method Solving a convex optimization problem with a single tuning parameter.
result The optimization problem has a unique global minimizer stable to inputs.
FisherSFT selects informative examples to fine-tune LLMs efficiently.
problem Adapting large language models to new domains efficiently.
method Selects examples maximizing information gain using Hessian of log-likelihood.
result Empirically demonstrates improved performance with reduced computational cost.
Machine learning applications often require hyperparameter tuning. The hyperparameters usually drive both the efficiency of the model training process and the resulting model quality. For hyperparameter tuning, machine learning algorithms are complex black-boxes. This creates a class of challenging optimization problem…
AdaSDBO solves decentralized bilevel optimization without problem parameters, achieving competitive performance.
problem Decentralized bilevel optimization problems without known parameters.
method AdaSDBO, a fully problem-parameter-free algorithm with adaptive stepsizes.
result AdaSDBO achieves a convergence rate of $\widetilde{\mathcal{O}}\left(\frac{1}{T}
ight)$, matching state-of-the-art methods up to polylogarithmic factors.