This paper compares Grid Search, Random Search, and Genetic Algorithm for NAS.
problem Hyperparameter optimization for neural architecture search.
method Comparison of Grid Search, Random Search, and Genetic Algorithm.
result Genetic Algorithm outperforms Grid Search and Random Search in terms of accuracy and execution time.
Proposes learning regularization strength directly from data.
problem Computational expense and data reduction in grid search for deep learning hyperparameters.
method Modified Evidence Lower Bound (ELBo) objective for model selection on full training set.
result Comparable heldout accuracy to grid search with less compute time.
Context: One of the black arts of data mining is learning the magic parameters which control the learners. In software analytics, at least for defect prediction, several methods, like grid search and differential evolution (DE), have been proposed to learn these parameters, which has been proved to be able to improve t…
The paper optimizes grid search for machine learning parameters with improved complexity.
problem Challenging tuning of machine learning regularization parameters.
method Unified framework for approximating regularization paths with refined complexity.
result Optimized grid search with improved complexity for various loss functions.
Determining the appropriate batch size for mini-batch gradient descent is always time consuming as it often relies on grid search. This paper considers a resizable mini-batch gradient descent (RMGD) algorithm based on a multi-armed bandit for achieving best performance in grid search by selecting an appropriate batch s…
Paper reviews and compares methods for handling imbalanced data.
problem Handling imbalanced data sets in financial industry.
method Reviewed and compared under-sampling/over-sampling methodologies.
result Performance analysis of class-imbalance methods, modeling algorithms, and grid search criteria.
Tuning SVM and boosting models using optimization algorithms.
problem Tuning parameters for SVM and boosting models across various datasets.
method Used grid search to identify parameter ranges and optimization algorithms to select models.
result Optimization algorithms outperformed grid search in selecting well-performing models.
A new Randomized-Hyperopt method improves XGBoost hyperparameter tuning.
problem Improving the performance of XGBoost through hyperparameter optimization.
method Proposes Randomized-Hyperopt for XGBoost hyperparameter tuning.
result Randomized-Hyperopt outperforms other methods in terms of accuracy and execution time.
Inexact acquisition solutions in BO lead to sublinear cumulative regret.
problem Inexact maximization of acquisition functions in Bayesian optimization.
method Define inaccuracy measure, establish cumulative regret bounds for GP-UCB and GP-TS.
result Inexact BO algorithms can achieve sublinear cumulative regret under appropriate inaccuracy conditions.
Optimizes SGD for text classification using hyper-parameter tuning.
problem Improving performance of SGD in text classification.
method Fine-tuning hyper-parameters using Grid-Search approach.
result Optimized SGD classification leads to better accuracy and execution time.
SMAC method optimizes tree-boosting hyperparameters best.
problem Optimizing hyperparameters for tree-boosting to improve model accuracy.
method Compared and evaluated various hyperparameter optimization methods.
result SMAC method outperforms other methods for hyperparameter tuning.
Learning rate annealing improves robustness in stochastic optimization.
problem Tuning learning rates in large-scale models is costly and prone to errors.
method We analyze and demonstrate the benefits of learning rate annealing schemes.
result Stochastic gradient descent with annealed schedules converges more robustly to the optimal solution.
Empirical comparison of 18 hyperparameter tuning algorithms for SVM.
problem Tuning hyperparameters C and γ for SVM with RBF kernel. method Compared 18 search algorithms on 115 real-life data sets.
result Trees of Parzen estimators and particle swarm optimization perform similarly to grid search.
Paper reproduces and enhances a method for cross-lingual word embeddings.
problem Creating robust cross-lingual mappings of word embeddings without supervision.
method Reproduces and enhances a self-learning method with grid search for hyperparameters.
result Model's robustness is demonstrated across four new languages.
Paper optimizes trading profits by predicting price direction using ensemble models.
problem Predicting short-term price movements in financial markets.
method Developed a two-layer ensemble architecture optimized through grid search.
result Improvement of 20% over benchmark strategies.
A hybrid ML model detects fraudulent transactions with high accuracy.
problem Detecting and preventing fraudulent credit card transactions.
method Intelligent combination of multiple algorithms with Grid search and IHT-LR.
result Achieves impressive accuracy rates of 99.66% for ENS model.
In order to find hyperparameters for a machine learning model, algorithms such as grid search or random search are used over the space of possible values of the models hyperparameters. These search algorithms opt the solution that minimizes a specific cost function. In language models, perplexity is one of the most pop…
This work simplifies SVM parameter selection using S&S ratio.
problem SVM parameter tuning for optimal performance.
method S&S ratio to model SVM performance; automatic RP, kernel, and parameter selection.
result Optimized SVM parameters with reduced computational complexity.
This paper introduces a fast, general method for dictionary-free parameter estimation in quantitative magnetic resonance imaging (QMRI) via regression with kernels (PERK). PERK first uses prior distributions and the nonlinear MR signal model to simulate many parameter-measurement pairs. Inspired by machine learning, PE…
Bayesian optimization improves classifier selection for acute infection and mortality.
problem Improving accuracy of acute infection and mortality prediction.
method Comparison of hyperparameter optimization methods (grid search, random sampling, Bayesian optimization).
result Bayesian optimization outperforms grid search or random sampling for in-hospital mortality classifiers.
A new method selects optimal temperature for Bayesian Deep Learning.
problem Finding the optimal temperature for improving predictive performance in Bayesian Deep Learning.
method Data-driven approach to estimate temperature as a model parameter.
result Our method performs comparably to grid search but at a fraction of the cost.
Unified framework for understanding TVO and improving model learning.
problem Improving the tightness and efficiency of variational inference bounds.
method Exponential family interpretation and equal spacing in moment parameters.
result Unified framework and improved gradient estimator for TVO.
The paper proposes a method to learn hyperparameters without validation sets, improving efficiency and accuracy.
problem Training large models on limited data to avoid overfitting and reduce validation set usage.
method Gradient-based learning of hyperparameters via a data-emphasized evidence lower bound (ELBO) objective.
result The data-emphasized ELBO reduces hyperparameter search time from 88+ hours to under 3 hours while maintaining comparable accuracy.
Efficient implicit differentiation for Lasso hyperparameter optimization.
problem Difficult hyperparameter optimization for Lasso-type models.
method Implicit differentiation algorithm tailored for Lasso-type problems, avoiding matrix inversion and solving linear systems.
result Outperforms standard methods in optimizing error on held-out data or Stein Unbiased Risk Estimator (SURE).
In this article we select the unknown dimension of the feature by re- versible jump MCMC inside a simulated annealing in bayesian set up of collaborative filter. We implement the same in MovieLens small dataset. We also tune the hyper parameter by using a modified empirical bayes. It can also be used to guess an initia…
In this article we provide a formulation of empirical bayes described by Atchade (2011) to tune the hyperparameters of priors used in bayesian set up of collaborative filter. We implement the same in MovieLens small dataset. We see that it can be used to get a good initial choice for the parameters. It can also be used…
A new method uses SVM classification to efficiently compute confidence sets.
problem Computing confidence sets for moment inequalities is computationally intensive.
method Converts confidence set construction into a classification problem using SVM.
result Asymptotically reproduces the test in the confidence set using SVM classification.
A new method uses recurrent nets to efficiently estimate SEIR model parameters.
problem Estimating SEIR model parameters is slow and inaccurate with grid search.
method Transform non-differentiable problem to differentiable one using recurrent nets.
result Significantly better parameter estimations with fewer simulations.
Kernelized Support Vector Machines (SVMs) are among the best performing supervised learning methods. But for optimal predictive performance, time-consuming parameter tuning is crucial, which impedes application. To tackle this problem, the classic model selection procedure based on grid-search and cross-validation was …
New framework explains fast transfer of hyperparameters across model scales.
problem Understanding and optimizing hyperparameters for large-scale models.
method Developed a conceptual framework for HP transfer across scale, showing fast transfer is equivalent to useful transfer for compute-optimal grid search.
result Fast transfer of hyperparameters is equivalent to useful transfer for compute-optimal grid search, offering asymptotic computational advantage.
A new model framework called Realized Conditional Autoregressive Expectile (Realized-CARE) is proposed, through incorporating a measurement equation into the conventional CARE model, in a manner analogous to the Realized-GARCH model. Competing realized measures (e.g. Realized Variance and Realized Range) are employed a…
Hyperparameter optimization aims to find the optimal hyperparameter configuration of a machine learning model, which provides the best performance on a validation dataset. Manual search usually leads to get stuck in a local hyperparameter configuration, and heavily depends on human intuition and experience. A simple al…
Model classifies hand gestures with low EMG samples augmented with noise.
problem Classify hand gestures with limited EMG data.
method Random variance Gaussian noise augmentation, Random Forest, XGBoost.
result High accuracy model for six hand gestures.
Paper introduces COBRA variations for multivariate time series forecasting.
problem Multivariate time series forecasting challenges.
method Innovative COBRA variations, data preprocessing, Bayesian optimisation vs. grid search.
result Proposed methodologies outperform state-of-the-art models.
New method finds graphene nanocrystals with reduced DFT calculations.
problem Efficiently discovering materials with desired properties in high-dimensional chemical space.
method Bayesian optimization with neural network kernel to minimize DFT calculations.
result Reduced computational cost by 20% for discovering materials with target properties.
Our team won the second prize of the Safe Aging with SPHERE Challenge organized by SPHERE, in conjunction with ECML-PKDD and Driven Data. The goal of the competition was to recognize activities performed by humans, using sensor data. This paper presents our solution. It is based on a rich pre-processing and state of th…
Auto-Surprise automates recommender system selection and optimization.
problem Finding the best algorithm and hyperparameters for recommender systems.
method Extends Surprise library with TPE optimization for algorithm selection and hyperparameter tuning.
result Significantly faster in finding optimal hyperparameters compared to grid search.
ALMAB-DC optimizes expensive black-box experiments using active learning and distributed computing.
problem Efficiently optimizing expensive, gradient-free objectives in computational statistics and machine learning.
method Combines active learning, multi-armed bandits, and distributed asynchronous computing.
result Achieves lower simple regret and superior performance in various tasks compared to non-ALMAB baselines.
Proposes a gradient-based bilevel optimization method for efficient hyperparameter tuning.
problem Efficiently tuning hyperparameters in machine learning models.
method Gradient-based bilevel optimization approach.
result The proposed method is multiple times faster than existing techniques.
New algorithm speeds up path computation for optimal models.
problem Finding the exact path of optimal models from a finite set.
method Dynamic programming approach for linear time computation.
result Dynamic programming achieves linear time for breakpoints computation.
Fourier-transform infra-red (FTIR) spectra of samples from 7 plant species were used to explore the influence of preprocessing and feature extraction on efficiency of machine learning algorithms. Wavelet Tensor Train (WTT) and Discrete Wavelet Transforms (DWT) were compared as feature extraction techniques for FTIR dat…
This paper studies sparse density estimation via ℓ1 penalization (SPADES). We focus on estimation in high-dimensional mixture models and nonparametric adaptive density estimation. We show, respectively, that SPADES can recover, with high probability, the unknown components of a mixture of probability densities an…
Study optimizes CANN for actuarial tasks using RSM.
problem Optimizing hyperparameters for neural networks in actuarial science.
method Factorial design and response surface methodology (RSM).
result Reduced hyperparameter optimization from 288 to 188, achieving near-optimal performance.
Dropout is used as a practical tool to obtain uncertainty estimates in large vision models and reinforcement learning (RL) tasks. But to obtain well-calibrated uncertainty estimates, a grid-search over the dropout probabilities is necessary - a prohibitive operation with large models, and an impossible one with RL. We …
Skip-gram with negative sampling, a popular variant of Word2vec originally designed and tuned to create word embeddings for Natural Language Processing, has been used to create item embeddings with successful applications in recommendation. While these fields do not share the same type of data, neither evaluate on the …
Driven by the need for parallelizable hyperparameter optimization methods, this paper studies \emph{open loop} search methods: sequences that are predetermined and can be generated before a single configuration is evaluated. Examples include grid search, uniform random search, low discrepancy sequences, and other sampl…
New method optimizes hyperparameters for non-smooth problems efficiently.
problem Efficiently tuning hyperparameters for non-smooth cost functions.
method Combines hyperparameter search with proximal gradient updates.
result Method converges to local optimum of LOO validation error.
In high-dimensional and/or non-parametric regression problems, regularization (or penalization) is used to control model complexity and induce desired structure. Each penalty has a weight parameter that indicates how strongly the structure corresponding to that penalty should be enforced. Typically the parameters are c…