Stage-based hyper-parameter optimization reduces GPU-hours and training time.
problem Efficiently executing hyper-parameter optimization for deep learning models.
method Stage-based execution strategy to remove redundant computations.
result Stage-based execution outperforms trial-based method by up to 6.60 times in GPU-hours and 4.13 times in training time.
Paper proposes a reinforcement learning framework for efficient hyper-parameter tuning of stochastic optimization algorithms.
problem Efficient tuning of hyper-parameters for stochastic optimization algorithms.
method Modeling hyper-parameter tuning as a Markov decision process and using policy gradient algorithms.
result The proposed framework significantly reduces the time required for hyper-parameter tuning compared to Bayesian optimization.
This paper proposes an efficient autoHPO method based on data-to-hyper-parameter mapping.
problem Manual hyper-parameter tuning is costly and dependent.
method The approach is based on mapping from data to hyper-parameters using a sophisticated network structure and effective construction algorithms.
result The proposed approach significantly outperforms state-of-the-art methods.
This work improves Bayesian Optimization for setting DNN hyper-parameters.
problem Manual setting of DNN hyper-parameters is error-prone and computationally expensive.
method Combines Bayesian Optimization with tuning rules to reduce search space and improve accuracy.
result Improves efficiency and accuracy of hyper-parameter tuning for deep neural networks.
Hippo optimizes deep learning hyper-parameters by reducing redundant trials.
problem Redundant hyper-parameter trials in hyper-parameter optimization.
method Hippo breaks down hyper-parameter sequences into stages and executes them in a tree structure.
result Hippo reduces GPU-hours and training time significantly compared to existing methods.
Evolutionary Strategies optimize hyper-parameters for off-policy learning.
problem Hyper-parameter sensitivity in off-policy learning.
method Application of Evolutionary Strategies for online hyper-parameter tuning.
result Our method outperforms state-of-the-art baselines.
In the regression setting, given a set of hyper-parameters, a model-estimation procedure constructs a model from training data. The optimal hyper-parameters that minimize generalization error of the model are usually unknown. In practice they are often estimated using split-sample validation. Up to now, there is an ope…
Two-tier approach optimizes RL hyper-parameters for better agent learning.
problem Optimizing hyper-parameters in reinforcement learning to improve agent performance.
method Two-step optimization: first categorical hyper-parameters, then solution-level hyper-parameters.
result Promising results in simulated control tasks, suggesting user-independent reinforcement learning applications.
Federated learning improves with adaptive hyper-parameters and representation matching.
problem Heterogeneous client data leads to divergent local models in federated learning.
method Representation matching and adaptive hyper-parameters.
result Significant performance and robustness improvements in federated learning.
This paper explores hyperparameter optimization for machine learning models.
problem Finding the best hyper-parameters for machine learning models.
method Introduces state-of-the-art optimization techniques and discusses their application.
result Comparison of different optimization methods on benchmark datasets.
A new method selects algorithms and optimizes their hyper-parameters efficiently.
problem Redundant hyper-parameter search space in AutoML.
method Cascaded algorithm selection and hyper-parameter optimization with ER-UCB bandit.
result ER-UCB strategy achieves optimal regret bound for algorithm selection.
KD technique improves QDNN performance with reduced hyper-parameters.
problem Restoring performance loss in QDNNs due to quantization.
method Applied KD with reduced hyper-parameters, including a new coefficient reduction technique.
result Achieved 92.7% test accuracy on CIFAR-10 and 67.0% on CIFAR-100 with 2-bit weights.
A framework for auto-tuning hyper-parameters in contextual bandit algorithms.
problem Auto-tuning hyper-parameters in real-time for contextual bandit algorithms.
method Proposes a Syndicated Bandits framework to learn multiple hyper-parameters dynamically.
result Achieves optimal regret bounds under certain scenarios and handles multiple contextual bandit algorithms.
Generative approach speeds hyperparameter tuning for machine learning models.
problem Computational infeasibility of cross-validation and difficulty of fully Bayesian hyper-parameter learning.
method Combines optimization-based approximations and amortization techniques.
result Rapid evaluation of hyper-parameters over grids or ranges, supporting predictive tuning and uncertainty quantification.
The quality of an induced model by a learning algorithm is dependent on the quality of the training data and the hyper-parameters supplied to the learning algorithm. Prior work has shown that improving the quality of the training data (i.e., by removing low quality instances) or tuning the learning algorithm hyper-para…
BCV method helps estimate clusters and hyper-parameters in large data sets.
problem Determining the number of clusters in large-scale data.
method Bi-cross validation (BCV) for spectral clustering.
result BCV directly applies to spectral clustering for estimating clusters and hyper-parameters.
ABPS improves RL training efficiency by sharing policies and evolving hyper-params.
problem Data inefficiency in training deep RL models for real-world applications.
method ABPS: adaptive behavior policy sharing; ABPS-PBT: hybridizing ABPS with PBT for evolving hyper-params.
result ABPS achieves superior performance and reduced variance compared to conventional hyper-parameter tuning.
New estimators outperform maximum likelihood without hyper-parameter estimation.
problem Improving system identification performance without hyper-parameter estimation.
method Developed generalized Bayes and closed-form biased estimators using excess MSE.
result New estimators have comparable performance to empirical-Bayes-based regularized estimator.
Paper proposes efficient GP hyper-parameter optimization methods.
problem Efficient hyper-parameter optimization for Gaussian process regression.
method Cross-validation and ADMM for O(n2) complexity. result Proposed methods outperform traditional ML-based routines.
Bayesian optimisation has gained great popularity as a tool for optimising the parameters of machine learning algorithms and models. Somewhat ironically, setting up the hyper-parameters of Bayesian optimisation methods is notoriously hard. While reasonable practical solutions have been advanced, they can often fail to …
Paper proposes CNN-LSTM model for multi-speaker speech separation.
problem Multi-speaker source separation using deep learning.
method Parallel CNN-LSTM architecture with Bayesian hyperparameter optimization.
result Parallel CNN-LSTM model outperforms LSTM-only and CNN-only models.
Bayesian optimisation tackles stochastic MPC hyper-parameter tuning.
problem Fine-tuning hyper-parameters in stochastic MPC models.
method Heteroscedastic Bayesian optimisation framework.
result Framework effectively tunes hyper-parameters in control problems.
This paper studies the effect of various hyper-parameters and their selection for the best performance of the deep learning model proposed in [1] for distributed attack detection in the Internet of Things (IoT). The findings show that there are three hyper-parameters that have more influence on the best performance ach…
A new Randomized-Hyperopt method improves XGBoost hyperparameter tuning.
problem Improving the performance of XGBoost through hyperparameter optimization.
method Proposes Randomized-Hyperopt for XGBoost hyperparameter tuning.
result Randomized-Hyperopt outperforms other methods in terms of accuracy and execution time.
New method improves anomaly detection in acoustic signals.
problem Poor anomaly detection performance in existing acoustic signal-based unsupervised methods.
method Deep autoencoding Gaussian mixture model with hyper-parameter optimization.
result Significantly improved anomaly detection performance compared to previous methods.
In many signal processing problems, it may be fruitful to represent the signal under study in a frame. If a probabilistic approach is adopted, it becomes then necessary to estimate the hyper-parameters characterizing the probability distribution of the frame coefficients. This problem is difficult since in general the …
Optimal Word2Vec hyper-parameters improve NLP tasks.
problem Finding the best Word2Vec hyper-parameters for NLP tasks.
method Empirical evaluation of various hyper-parameter combinations on NLP tasks.
result The best hyper-parameters vary by task, and high analogy scores don't always correlate with performance.
Meta-Semi learns to optimize SSL with minimal hyper-parameter tuning.
problem Limited labeled data in SSL makes it impractical to tune many hyper-parameters.
method Meta-learning approach that dynamically reweights unlabeled data loss.
result Meta-Semi achieves competitive performance on various SSL tasks.
This paper reviews hyper-parameter optimization methods for deep learning.
problem Designing and training deep neural networks is challenging and unpredictable.
method Reviews major optimization algorithms and services for hyper-parameter tuning.
result Comprehensive comparison of optimization algorithms and services.
GeneCAI optimizes DNN compression hyper-parameters for mobile devices.
problem Efficient deployment of complex DNNs on resource-limited devices.
method GeneCAI uses genetic algorithm to learn optimal hyper-parameters.
result GeneCAI finds models with better accuracy-complexity trade-off.
BOFFIN TTS optimizes hyper-parameters for new speaker adaptation.
problem Fine-tuning a pre-trained TTS model for a new speaker with limited data.
method Bayesian optimization to efficiently find optimal hyper-parameters.
result Average 30% improvement in speaker similarity over standard techniques.
Paper tackles hyper-gradient estimation in decentralized FL over time-varying networks.
problem Excessive communication costs and inability to use robust networks.
method Introduces an optimality condition and uses Push-Sum for averaging model parameters and gradients over time-varying directed networks.
result Derives a hyper-gradient estimator that operates over time-varying directed networks and converges to the true hyper-gradient.
Machine learning pipeline potentially consists of several stages of operations like data preprocessing, feature engineering and machine learning model training. Each operation has a set of hyper-parameters, which can become irrelevant for the pipeline when the operation is not selected. This gives rise to a hierarchica…
Although deep learning has produced dazzling successes for applications of image, speech, and video processing in the past few years, most trainings are with suboptimal hyper-parameters, requiring unnecessarily long training times. Setting the hyper-parameters remains a black art that requires years of experience to ac…
Predicts neural network accuracy with minimal training, speeding up hyper-parameter tuning.
problem Expensive full training of neural networks during hyper-parameter tuning.
method Train network multiple times, store initial and final accuracies, use curve fitting and SVM for prediction.
result Predicted accuracies match known optimal settings with low computational cost.
Non-compact G_2 holonomy metrics that arise from a T^2 bundle over a hyper-Kahler space are discussed. These are one parameter deformations of the metrics studied by Gibbons, Lu, Pope and Stelle in hep-th/0108191. Seven-dimensional spaces with G_2 holonomy fibered over the Taub-Nut and the Eguchi-Hanson gravitational i…
Study shows how network width affects SGD hyper-parameters and generalization.
problem Understanding how network width impacts SGD hyper-parameters and generalization.
method Generated model families by increasing network width, performed hyper-parameter search.
result Wider networks achieve higher test accuracy and optimal normalized noise scale.
Purpose: Machine learning is broadly used for clinical data analysis. Before training a model, a machine learning algorithm must be selected. Also, the values of one or more model parameters termed hyper-parameters must be set. Selecting algorithms and hyper-parameter values requires advanced machine learning knowledge…
In the Minkowski space-time, a world hyper-sheet is a timelike hypersurface consisting of a one-parameter family of spacelike submanifolds. Recently, Bousso and Randall introduced the notion of caustics of world hyper-sheets in order to define the notion of holographic domains in space-time. Here, we give a mathematica…
HyperVAE encodes distributions of distributions using variational inference.
problem Modeling distributions of distributions efficiently and preserving information.
method Variational inference with Gaussian mixture models and matrix-network decoders.
result HyperVAE encodes parameters of a VAE in a low-dimensional Gaussian distribution, preserving more information.
Improves risk control in predictions using semi-supervised calibration.
problem Noisy hyper-parameter tuning from limited labeled data.
method Semi-supervised calibration using unlabeled data to tune hyper-parameters rigorously.
result Improves prediction accuracy without sacrificing statistical validity.
This paper introduces an efficient method for optimizing deep learning hyperparameters.
problem The high dependency of deep learning algorithms on hyper-parameters.
method Orthogonal Array Tuning Method (OATM) for deep learning hyper-parameter tuning.
result The proposed OATM method significantly saves tuning time compared to state-of-the-art methods.
Gradient boosting decision trees (GBDTs) have seen widespread adoption in academia, industry and competitive data science due to their state-of-the-art performance in many machine learning tasks. One relative downside to these models is the large number of hyper-parameters that they expose to the end-user. To maximize …
This paper aims to explore models based on the extreme gradient boosting (XGBoost) approach for business risk classification. Feature selection (FS) algorithms and hyper-parameter optimizations are simultaneously considered during model training. The five most commonly used FS methods including weight by Gini, weight b…
Black box discrete optimization (BBDO) appears in wide range of engineering tasks. Evolutionary or other BBDO approaches have been applied, aiming at automating necessary tuning of system parameters, such as hyper parameter tuning of machine learning based systems when being installed for a specific task. However, auto…
Efficiently models learning curves using Gaussian processes with latent Kronecker structure.
problem Joint modeling of machine learning model performance across hyper-parameters and training progress.
method Imposes latent Kronecker structure to leverage efficient product kernels and handle missing values.
result Matches the performance of a Transformer on a learning curve prediction task.
This paper proposes an automatic neural network compression method.
problem Reducing resource requirements for deep neural networks on resource-constrained devices.
method Jointly prunes and quantizes neural networks without manual hyper-parameter tuning.
result Significant reduction in model size with minimal accuracy loss.
Hyper-parameters play a major role in the learning and inference process of latent Dirichlet allocation (LDA). In order to begin the LDA latent variables learning process, these hyper-parameters values need to be pre-determined. We propose an extension for LDA that we call 'Latent Dirichlet allocation Gibbs Newton' (LD…