arXiv research

New stochastic gradient descent with random search directions improves efficiency and convergence.

Paper develops online statistical inference methods for stochastic optimization using Kiefer-Wolfowitz algorithms.

Neural architecture search (NAS) is a promising research direction that has the potential to replace expert-designed networks with learned, task-specific architectures. In this work, in order to help ground the empirical results in this field, we propose new NAS baselines that build off the following observations: (i) …

A new method optimizes neural sequence models for better task performance.

Many applications in machine learning require optimizing a function whose true gradient is unknown, but where surrogate gradient information (directions that may be correlated with, but not necessarily identical to, the true gradient) is available instead. This arises when an approximate gradient is easier to compute t…

Scientific discovery is limited by hypothesis redundancy, and hybrid methods can exploit non-local exploration.

Survey of techniques for automating hyperparameter optimization in machine learning.

ConMeZO speeds up zeroth-order optimization for large language models.

Derivative-free optimization has become an important technique used in machine learning for optimizing black-box models. To conduct updates without explicitly computing gradient, most current approaches iteratively sample a random search direction from Gaussian distribution and compute the estimated gradient along that…

Efficient search methods can outperform random search on challenging tasks.

This paper compares Grid Search, Random Search, and Genetic Algorithm for NAS.

New method solves constrained optimization problems efficiently.

Direct optimization is an appealing framework that replaces integration with optimization of a random objective for approximating gradients in models with discrete random variables. A$^\star$ sampling is a framework for optimizing such random objectives over large spaces. We show how to combine these techniques to yiel…

ImmuNeCS uses AI immune system to build neural committees for efficient deep learning model creation.

Context: One of the black arts of data mining is learning the magic parameters which control the learners. In software analytics, at least for defect prediction, several methods, like grid search and differential evolution (DE), have been proposed to learn these parameters, which has been proved to be able to improve t…

A new cost-frugal HPO method controls training cost during optimization.

A new Randomized-Hyperopt method improves XGBoost hyperparameter tuning.

We introduce a method that uses the Cauchy-Crofton formula and a new curvature formula from integral geometry to reweight the sampling probabilities of Metropolis-within-Gibbs algorithms in order to increase their convergence speed. We consider algorithms that sample from a probability density conditioned on a manifold…

A new imitation learning method uses random search for simple policies, outperforming complex models.

Proposes a link between randomness and compression in deep learning.

The paper analyzes Random Search and introduces BLiN-MOS for bandit learning.

RandomNet uses random search to design neural architectures without much human intervention.

New reshaping method improves hyperparameter search over random search.

This paper surveys various methods for dimensionality reduction and nearest neighbor search.

Early stopping method saves up to 75% computation time in policy search tasks.

Neural Architecture Search (NAS) aims to facilitate the design of deep networks for new tasks. Existing techniques rely on two stages: searching over the architecture space and validating the best architecture. NAS algorithms are currently compared solely based on their results on the downstream task. While intuitive, …

Driven by the need for parallelizable hyperparameter optimization methods, this paper studies \emph{open loop} search methods: sequences that are predetermined and can be generated before a single configuration is evaluated. Examples include grid search, uniform random search, low discrepancy sequences, and other sampl…

In this paper, we propose the use of a black-box optimization method called deterministic Mesh Adaptive Direct Search (MADS) algorithm with orthogonal directions (Ortho-MADS) for the selection of hyperparameters of Support Vector Machines with a Gaussian kernel. Different from most of the methods in the literature that…

Weight-sharing is rarely helpful in NAS, especially on small datasets.

Improved GSPGS estimators reduce bias in noisy function measurements.

Analyzes Lévy flights on manifolds for finding small targets.

Yau's Affine Normal Descent optimizes smooth unconstrained problems with geometrically adapted directions.

Bonsai-Net efficiently discovers state-of-the-art models with fewer parameters.

This letter presents a new spectral-clustering-based approach to the subspace clustering problem. Underpinning the proposed method is a convex program for optimal direction search, which for each data point d finds an optimal direction in the span of the data that has minimum projection on the other data points and non…

Simpler neural architecture search method using random architectures and regression.

Previously, the exploding gradient problem has been explained to be central in deep learning and model-based reinforcement learning, because it causes numerical issues and instability in optimization. Our experiments in model-based reinforcement learning imply that the problem is not just a numerical issue, but it may …

Choosing appropriate step sizes is critical for reducing the computational cost of training large-scale neural network models. Mini-batch sub-sampling (MBSS) is often employed for computational tractability. However, MBSS introduces a sampling error, that can manifest as a bias or variance in a line search. This is bec…

This paper provides a block coordinate descent algorithm to solve unconstrained optimization problems. In our algorithm, computation of function values or gradients is not required. Instead, pairwise comparison of function values is used. Our algorithm consists of two steps; one is the direction estimate step and the o…

A common belief in model-free reinforcement learning is that methods based on random search in the parameter space of policies exhibit significantly worse sample complexity than those that explore the space of actions. We dispel such beliefs by introducing a random search method for training static, linear policies for…

New local-search methods close the gap in sparse tensor PCA.

Random layer-wise pruning profiles are as effective as metric-based ones for various datasets.

Improved Random Search for hyperparameter optimization.

We introduce an approximate search algorithm for fast maximum a posteriori probability estimation in probabilistic programs, which we call Bayesian ascent Monte Carlo (BaMC). Probabilistic programs represent probabilistic models with varying number of mutually dependent finite, countable, and continuous random variable…

New proof shows how to identify DAGs with weakly increasing errors.

WRS improves CNN hyperparameter optimization.

In this paper, we will provide an introduction to the derivative-free optimization algorithms which can be potentially applied to train deep learning models. Existing deep learning model training is mostly based on the back propagation algorithm, which updates the model variables layers by layers with the gradient desc…

NeuralArTS categorizes neural ops in a type system for NAS.

Improved SSD for faster and more accurate goodness-of-fit tests and model learning.