This paper explores hyperparameter optimization for machine learning models.
problem Finding the best hyper-parameters for machine learning models.
method Introduces state-of-the-art optimization techniques and discusses their application.
result Comparison of different optimization methods on benchmark datasets.
Hippo optimizes deep learning hyper-parameters by reducing redundant trials.
problem Redundant hyper-parameter trials in hyper-parameter optimization.
method Hippo breaks down hyper-parameter sequences into stages and executes them in a tree structure.
result Hippo reduces GPU-hours and training time significantly compared to existing methods.
Improved Bayesian optimization for conditional parameter spaces.
problem Efficient global optimization of expensive-to-evaluate functions in conditional parameter spaces.
method Additive tree-structured covariance function for conditional parameter optimization.
result Significantly improved sample-efficiency and wider applicability compared to existing methods.
Convex optimization models predict outputs from inputs via optimization problems.
problem Predicting outputs from inputs using convex optimization models.
method Proposed a heuristic for learning parameters of convex optimization models from datasets.
result Demonstrated the effectiveness of the proposed method on three model classes.
As deep learning techniques advance more than ever, hyper-parameter optimization is the new major workload in deep learning clusters. Although hyper-parameter optimization is crucial in training deep learning models for high model performance, effectively executing such a computation-heavy workload still remains a chal…
Unified GP model optimizes hyperparameters with conditional dependence.
problem Efficient tuning of hyperparameters in neural networks.
method Unified Bayesian optimization framework based on a new Gaussian process (GP) model.
result Higher prediction accuracy and better optimization efficiency observed.
Paper proposes a reinforcement learning framework for efficient hyper-parameter tuning of stochastic optimization algorithms.
problem Efficient tuning of hyper-parameters for stochastic optimization algorithms.
method Modeling hyper-parameter tuning as a Markov decision process and using policy gradient algorithms.
result The proposed framework significantly reduces the time required for hyper-parameter tuning compared to Bayesian optimization.
When applying Machine Learning techniques to problems, one must select model parameters to ensure that the system converges but also does not become stuck at the objective function's local minimum. Tuning these parameters becomes a non-trivial task for large models and it is not always apparent if the user has found th…
New Riemannian optimization improves variance estimation in mixed models.
problem Challenges in estimating variance parameters in linear mixed models due to constraints.
method Formulated as an optimization problem on a Riemannian manifold, using Riemannian gradient and Hessian.
result Yields higher quality variance parameter estimates compared to existing methods.
Bayesian model selection optimizes data augmentation for improved machine learning robustness.
problem Choosing optimal data augmentation parameters is challenging and often done through trial and error.
method Interprets augmentation parameters as model hyperparameters and uses Bayesian model selection to optimize them.
result Our approach improves calibration and robust performance on various tasks.
New neural network approach for optimizing latent variable models.
problem Stability issues in marginalizing Gaussian Bayesian networks.
method Developed a new graphical structure and a neural network algorithm.
result Established a duality between parameter optimization and neural network training.
This paper shows using sub-sample estimates can improve optimization results in large-scale problems.
problem Large-scale optimization problems with uncertain parameters often lead to suboptimal solutions due to mis-specifications or extreme sample characteristics.
method The paper introduces the use of sub-sample estimates to reduce errors in stochastic optimization models, providing theoretical analysis and numerical examples.
result Sub-sample optimization can achieve improved results over full-sample solution estimates in large-scale problems.
New phases identified in neural scaling laws with compute limits.
problem Understanding neural scaling laws under compute constraints.
method Solved neural scaling model with stochastic gradient descent, derived loss curves, analyzed model-parameter-count phases.
result Identified 4 phases (+3 subphases) in data-complexity/target-complexity phase-plane, derived exponents.
Integrates estimation and optimization for uncertain parameters.
problem Optimizing with uncertain parameters whose distributions can be estimated.
method Integrated Conditional Estimation-Optimization (ICEO) framework.
result Asymptotically consistent and provides finite performance guarantees.
Study values and optimizes forestry leases under risk and uncertainty.
problem Valuing and optimizing forestry leases in the presence of catastrophe risk and parameter uncertainty.
method Stochastic bio-economic models, Kalman filter, maximum likelihood estimation, RBSDEs, Monte Carlo simulations.
result Conservative strategy is recommended due to parameter uncertainty.
Optimal B-robust estimate is constructed for multidimensional parameter in drift coefficient of diffusion type process with small noise. Optimal mean-variance robust (optimal V -robust) trading strategy is find to hedge in mean-variance sense the contingent claim in incomplete financial market with arbitrary informatio…
Paper studies MCCR models with scale parameters tending to zero, revealing optimal learning rate and comparing robustness.
problem Analyzing MCCR models with scale parameters approaching zero.
method Investigates MCCR models with scale parameters tending to zero, revealing optimal learning rate and comparing robustness.
result Optimal learning rate of MCCR models is O(n−1) in the asymptotic sense. GoBOED optimizes experiments for specific decision-making objectives, improving downstream outcomes.
problem Reducing parameter uncertainty does not always improve decision-making in critical settings.
method Combines variational posterior surrogate and differentiable convex decision layer for gradient-based design optimization.
result GoBOED identifies designs that better align with specific decision objectives and reveals wider optimal design windows.
This work improves Bayesian Optimization for setting DNN hyper-parameters.
problem Manual setting of DNN hyper-parameters is error-prone and computationally expensive.
method Combines Bayesian Optimization with tuning rules to reduce search space and improve accuracy.
result Improves efficiency and accuracy of hyper-parameter tuning for deep neural networks.
Optimizing full likelihoods adapts loss scales and shapes for robust modeling.
problem Rigid loss functions limit model adaptability and robustness.
method Optimize full likelihoods with adjustable parameters.
result Adaptive tuning of loss scales and shapes improves model robustness.
Black box discrete optimization (BBDO) appears in wide range of engineering tasks. Evolutionary or other BBDO approaches have been applied, aiming at automating necessary tuning of system parameters, such as hyper parameter tuning of machine learning based systems when being installed for a specific task. However, auto…
The paper provides a method to find optimal machine learning model parameters with confidence.
problem Finding optimal machine learning model parameters that generalize well to the entire population.
method Constructs valid confidence sets for the optimal parameter using only training data.
result Valid confidence sets for optimal machine learning model parameters can be generated using bootstrapping techniques.
Researchers develop optimal methods to estimate rough volatility parameters.
problem Statistical inference for rough volatility models with fractional Brownian motion.
method Established minimax lower bounds and designed wavelet-based procedures.
result Optimal speed of convergence n−1/(4H+2) for estimating H. The paper provides a method to minimize regret in estimate-then-optimize decision-making.
problem Errors in estimation lead to sub-optimal decisions in data-driven decision-making.
method A novel bound on regret for smooth and unconstrained optimization problems, followed by experimental design to minimize this regret.
result A general procedure for experimental design to minimize regret resulting from estimate-then-optimize.
Tuning SVM and boosting models using optimization algorithms.
problem Tuning parameters for SVM and boosting models across various datasets.
method Used grid search to identify parameter ranges and optimization algorithms to select models.
result Optimization algorithms outperformed grid search in selecting well-performing models.
Bayesian method improves parameter reconstruction from many measurements.
problem Efficiently reconstructing parameters from many experimental measurements.
method Bayesian target-vector optimization considering all model outputs.
result Outperforms established optimization methods in accuracy and efficiency.
Novel method embeds generative model into Bayesian optimization for HD cardiac model parameter estimation.
problem High-dimensional optimization of patient-specific cardiac model parameters with limited data.
method Embeds a generative variational auto-encoder into the objective function of Bayesian optimization.
result Improves accuracy of parameter estimation with more than 10x gain in efficiency.
Optimize black-box simulators with local generative models.
problem Optimizing non-differentiable, stochastic simulators with intractable likelihoods.
method Differentiable local surrogate models based on deep generative models.
result Local surrogates enable gradient-based optimization, faster than baseline methods.
Study shows how numerical discretization affects reconstructions and parameter distributions in nano metrology.
problem Impact of numerical discretization on parameter reconstructions and model parameter distributions.
method Bayesian target vector optimization, finite element model, Gaussian process, stochastic machine learning surrogate models, Markov chain Monte Carlo sampler.
result Numerical discretization parameters impact the accuracy and distribution of reconstructed model parameters.
Unified Bayesian Optimization framework for model selection balancing effectiveness and training efficiency.
problem Balancing model effectiveness and training efficiency in machine learning model selection.
method Proposes a unified Bayesian Optimization framework to jointly optimize model effectiveness and training efficiency.
result Models selected using the proposed framework significantly improve training efficiency while maintaining strong effectiveness.
Machine learning with big data often involves large optimization models. For distributed optimization over a cluster of machines, frequent communication and synchronization of all model parameters (optimization variables) can be very costly. A promising solution is to use parameter servers to store different subsets of…
A novel machine learning optimization process coined Restrictive Federated Model Selection (RFMS) is proposed under the scenario, for example, when data from healthcare units can not leave the site it is situated on and it is forbidden to carry out training algorithms on remote data sites due to either technical or pri…
Generative approach speeds hyperparameter tuning for machine learning models.
problem Computational infeasibility of cross-validation and difficulty of fully Bayesian hyper-parameter learning.
method Combines optimization-based approximations and amortization techniques.
result Rapid evaluation of hyper-parameters over grids or ranges, supporting predictive tuning and uncertainty quantification.
This paper aims to explore models based on the extreme gradient boosting (XGBoost) approach for business risk classification. Feature selection (FS) algorithms and hyper-parameter optimizations are simultaneously considered during model training. The five most commonly used FS methods including weight by Gini, weight b…
In practical Bayesian optimization, we must often search over structures with differing numbers of parameters. For instance, we may wish to search over neural network architectures with an unknown number of layers. To relate performance data gathered for different architectures, we define a new kernel for conditional p…
Study optimizes sensor placement for accurate parameter estimation in complex systems.
problem Challenges in parameter estimation with limited or noisy data.
method Physics-Informed Neural Networks (PINNs) for optimal sensor placement and parameter estimation.
result PINNs-based framework achieves higher accuracy in parameter estimation compared to random sensor placements.
New study finds optimal hyperparameter tuning crucial for fair optimizer comparisons.
problem Inadequate hyperparameter tuning and misleading evaluation setups hinder fair comparisons of optimizers.
method Systematic study of ten optimizers across four model scales and data-to-model ratios.
result Optimal hyperparameters for one optimizer may be suboptimal for another, and many claimed speedups are lower than expected.
Paper characterizes optimal learning trajectories for high-dimensional nonlinear models.
problem Characterizing optimal learning trajectories in high-dimensional nonlinear models.
method Exploits maximum principle and dynamic programming for an optimal control problem of a gradient system.
result Constructs optimal learning trajectories leading to optimal model parameters.
The study calibrates neural networks' parameters through optimal contraction in prediction problems.
problem Ensuring the existence and uniqueness of optimal parameters in neural networks.
method Transforming RNNs into contractions and solving matrix equations involving Sylvester equations.
result Optimal parameters exist, are unique, and can be found through an algorithm with desired precision.
The quality of an induced model by a learning algorithm is dependent on the quality of the training data and the hyper-parameters supplied to the learning algorithm. Prior work has shown that improving the quality of the training data (i.e., by removing low quality instances) or tuning the learning algorithm hyper-para…
GeneCAI optimizes DNN compression hyper-parameters for mobile devices.
problem Efficient deployment of complex DNNs on resource-limited devices.
method GeneCAI uses genetic algorithm to learn optimal hyper-parameters.
result GeneCAI finds models with better accuracy-complexity trade-off.
Two simulation-based methods improve optimal sampling design in systems biology.
problem Optimal selection of sampling points for accurate parameter estimation in dynamical systems.
method E-optimal-ranking (EOR) and LSTM neural network-based methods.
result Simulation studies show the proposed methods outperform random selection and classical E-optimal design.
Novel covariance function improves Bayesian optimization efficiency.
problem Efficient global optimization of expensive black-box functions.
method Additive tree-structured covariance function and parallel optimization algorithm.
result Significantly outperforms state-of-the-art methods in conditional parameter optimization.
In engineering applications almost all processes are described with the help of models. Especially forming machines heavily rely on mathematical models for control and condition monitoring. Inaccuracies during the modeling, manufacturing and assembly of these machines induce model uncertainty which impairs the controll…
Memory-Augmented Meta-Optimization improves cold-start recommendation.
problem Cold-start problem in recommender systems for new users or items.
method Memory-Augmented Meta-Optimization approach with personalized and task-specific memories.
result Significant improvement in cold-start recommendation performance on multiple datasets.
New method improves anomaly detection in acoustic signals.
problem Poor anomaly detection performance in existing acoustic signal-based unsupervised methods.
method Deep autoencoding Gaussian mixture model with hyper-parameter optimization.
result Significantly improved anomaly detection performance compared to previous methods.
New method designs experiments robustly for nonlinear estimation, improving parameter knowledge.
problem Designing robust experiments for nonlinear estimation under parametric uncertainty.
method Multi-stage robust optimization framework for sequential experiments.
result Identifies experiments better conducted early for improved parameter knowledge.
BOFFIN TTS optimizes hyper-parameters for new speaker adaptation.
problem Fine-tuning a pre-trained TTS model for a new speaker with limited data.
method Bayesian optimization to efficiently find optimal hyper-parameters.
result Average 30% improvement in speaker similarity over standard techniques.