Random search outperforms leading NAS methods in neural architecture search.
problem Improving the reproducibility and efficiency of neural architecture search.
method Proposed random search with early-stopping and weight-sharing as NAS baselines.
result Random search with weight-sharing achieves state-of-the-art results on PTB and highly competitive results on CIFAR-10.
EA-LSTM improves LSTM for time series prediction by evolving attention.
problem LSTMs struggle with assigning varying attention to sub-windows in time series data.
method Evolutionary attention-based LSTM with competitive random search.
result EA-LSTM achieves competitive performance in multivariate time series prediction.
We present a broad agenda for meaningful banking regulation reform aiming the creation of evolutive competitive environment to maximize the effectiveness of international financial system through the introduction of fair competition process among the banks in free market capitalism. We assume that the international fin…
Automatically tunes hyperparameters for faster approximate nearest neighbor search.
problem Tuning hyperparameters for efficient approximate nearest neighbor search is slow and impractical.
method Proposes an algorithm using randomized space-partitioning trees to automatically tune hyperparameters.
result Significantly faster than existing approaches and competitive in query time.
A common belief in model-free reinforcement learning is that methods based on random search in the parameter space of policies exhibit significantly worse sample complexity than those that explore the space of actions. We dispel such beliefs by introducing a random search method for training static, linear policies for…
A new imitation learning method uses random search for simple policies, outperforming complex models.
problem Complexity and reward dependency issues in imitation learning.
method Derivative-free optimization with simple linear policies and random search.
result The proposed method achieves competitive performance on MuJoCo locomotion tasks without a direct reward signal.
We explore efficient neural architecture search methods and show that a simple yet powerful evolutionary algorithm can discover new architectures with excellent performance. Our approach combines a novel hierarchical genetic representation scheme that imitates the modularized design pattern commonly adopted by human ex…
In deep learning, performance is strongly affected by the choice of architecture and hyperparameters. While there has been extensive work on automatic hyperparameter optimization for simple spaces, complex spaces such as the space of deep architectures remain largely unexplored. As a result, the choice of architecture …
HMCNAS generates competitive neural architectures without human-defined parameters.
problem Lack of human-defined parameters in Neural Architecture Search.
method Combines Hidden Markov Chains and Bayesian Optimization for autonomous search space generation and competitive model generation.
result HMCNAS generates competitive models in a short time without human-defined parameters.
Bayesian optimization outperformed random search in machine learning hyperparameter tuning challenge.
problem Optimizing hyperparameters of machine learning models using derivative-free methods.
method Bayesian optimization vs. random search on real datasets.
result Bayesian optimization significantly outperformed random search in held-out objective functions.
A novel unsupervised outlier detection method using Randomized PCA Forest.
problem Unsupervised outlier detection in datasets.
method Randomized Principal Component Analysis (RPCA) Forest for deriving an outlier score.
result Superior performance compared to classical and state-of-the-art methods.
Paper tackles unfair advantages in DARTS, presenting Fair DARTS to improve neural architecture search.
problem Performance collapse in DARTS due to unfair advantages in skip connections.
method Relax exclusive competition to collaborative, let architectural weights be independent, and use zero-one loss for discretization.
result New state-of-the-art results on CIFAR-10 and ImageNet, demonstrating the effectiveness of Fair DARTS.
FiDi-RL combines deep RL with FiDi policy search for efficient continuous control.
problem Challenges in continuous control tasks using reinforcement learning.
method Combines Deep Deterministic Policy Gradients (DDPG) with Augment Random Search (ARS) using FiDi policy search.
result Improves data efficiency and stability of ARS.
Efficient search methods can outperform random search on challenging tasks.
problem Comparing the performance of efficient and random search methods in neural architecture search.
method Comparison of weight sharing and random search methods on progressively larger search spaces for image classification and detection.
result Efficient search methods can provide substantial gains over random search on large, realistic tasks.
This paper compares Grid Search, Random Search, and Genetic Algorithm for NAS.
problem Hyperparameter optimization for neural architecture search.
method Comparison of Grid Search, Random Search, and Genetic Algorithm.
result Genetic Algorithm outperforms Grid Search and Random Search in terms of accuracy and execution time.
RGBM improves GBM efficiency with randomization.
problem Limited theoretical understanding of Gradient Boosting Machine.
method Randomization scheme to reduce search in weak-learners space.
result RGBM leads to substantial computational gains compared to GBM.
AMEIR automates recommender system design using NAS.
problem Manual feature and architecture engineering in recommender systems.
method Three-stage search space and tailored three-step pipeline.
result AMEIR outperforms manual and complex NAS methods.
einspace expands NAS search space to include diverse neural architectures.
problem NAS results are often limited to existing structures; new designs are rare.
method einspace uses a probabilistic context-free grammar to create a versatile search space.
result einspace discovers novel and improved architectures, including convolutions and attention.
A new Randomized-Hyperopt method improves XGBoost hyperparameter tuning.
problem Improving the performance of XGBoost through hyperparameter optimization.
method Proposes Randomized-Hyperopt for XGBoost hyperparameter tuning.
result Randomized-Hyperopt outperforms other methods in terms of accuracy and execution time.
The paper analyzes Random Search and introduces BLiN-MOS for bandit learning.
problem Understanding and optimizing hyperparameter tuning in metric measure spaces.
method Introducing scattering dimension to quantify performance, and developing BLiN-MOS for bandit learning.
result Random Search converges to optimal values with specific rates in noise-free and noisy environments.
This paper improves keyword recommendation for sponsored search using deep reinforcement learning.
problem Selecting keywords from given candidates considering internal and external competitions.
method Solves the combinatorial optimization problem of keyword recommendations with a modified pointer network structure trained in a deep reinforcement learning framework.
result Remarkable improvements in performance observed both offline and online.
Paper proposes a new evaluation method for NAS search phase.
problem NAS search phase effectiveness not well evaluated.
method Compare NAS solutions with random selection; evaluate weight sharing strategy.
result State-of-the-art NAS algorithms perform similarly to random selection.
Simple adaptive attacks can jailbreak state-of-the-art safety-aligned language models.
problem Vulnerability of safety-aligned language models to simple adaptive attacks.
method Designing adversarial prompt templates, random search on suffixes, and transfer/prefilling attacks.
result 100% attack success rate on various models including GPT-4, Vicuna, Mistral, and Claude.
MICO uses mutual information co-training to improve selective search efficiency.
problem Efficiently search and route unseen queries in large-scale search systems.
method Mutual Information Co-training framework for selective search with minimal supervision.
result Significantly improves performance on multiple metrics of selective search.
This work proposes an efficient autoregressive model for text generation.
problem The challenge of generating high-quality text with autoregressive models.
method Introduces a cascaded decoding approach using Markov transformers to achieve sub-linear parallel time generation.
result Shows competitive accuracy/speed tradeoff compared to existing methods on five machine translation datasets.
RandomNet uses random search to design neural architectures without much human intervention.
problem Designing neural architectures without excessive human intervention.
method Random search strategy for multimodal neural architecture design.
result RandomNet performs close to state-of-the-art on AV-MNIST with minimal human supervision.
MemNet optimizes neural architectures for memory efficiency.
problem Memory constraints in mobile devices limit the use of large neural networks.
method Augment-trim learning with memory consumption ranking score.
result MemNet finds architectures with 24.17% less memory usage compared to state-of-the-art methods.
BUSTLE synthesizes programs by learning from intermediate values.
problem Challenges in synthesizing complex programs due to large search space.
method Bottom-up search guided by a neural network trained on input-output examples.
result Bottom-up search with execution of intermediate programs provides valuable semantic information.
New stochastic gradient descent with random search directions improves efficiency and convergence.
problem Efficiency and convergence of stochastic gradient descent methods.
method Developed a new class of stochastic gradient descent algorithms with random search directions.
result Established almost sure convergence and provided Lp rates of convergence. New reshaping method improves hyperparameter search over random search.
problem Improving hyperparameter search efficiency.
method Introducing reshaping techniques to optimize search distributions.
result Substantial gains over random search in various experiments.
This paper surveys various methods for dimensionality reduction and nearest neighbor search.
problem Efficiently reducing high-dimensional data to lower dimensions while preserving essential information.
method Linear and nonlinear random projections, including sparse random projections, random Fourier Features, and Random Kitchen Sinks.
result Various methods for dimensionality reduction and nearest neighbor search are explained and compared.
The study develops a supervised and unsupervised WTA model for sparse binary projections.
problem Sparse binary projections in high-dimensional spaces.
method Supervised and unsupervised WTA models with efficient algorithms.
result Significantly improved results in similarity search tasks.
rpForests uses random projections to speed up kNN search.
problem Efficient kNN search in high-dimensional data.
method Random projection forests aggregating results from multiple random projection trees.
result rpForests achieves fast decay in missing rate and kNN distance discrepancy.
Efficient neural architecture search by sampling structure and operations.
problem Efficiently searching for optimal neural architectures.
method Decouples structure and operation search, using reinforcement learning with policy vectors.
result Significantly improved efficiency compared to traditional methods.
One-shot neural architecture search limits depth search space and prunes networks for better performance and uncertainty.
problem Finding optimal depth in residual networks for efficient training and inference.
method Formulated a variational objective to approximate the depth distribution and pruned networks based on this distribution.
result Pruned networks achieve competitive accuracy with unpruned networks and better uncertainty calibration.
The design of neural network architectures for a new data set is a laborious task which requires human deep learning expertise. In order to make deep learning available for a broader audience, automated methods for finding a neural network architecture are vital. Recently proposed methods can already achieve human expe…
Driven by the need for parallelizable hyperparameter optimization methods, this paper studies \emph{open loop} search methods: sequences that are predetermined and can be generated before a single configuration is evaluated. Examples include grid search, uniform random search, low discrepancy sequences, and other sampl…
Theory explains why DARTS favors deep architectures over shallow ones.
problem DARTS selects architectures with dominated skip connections, leading to performance degradation.
method Theoretical analysis of operations' effects on network optimization; introduces sparse binary gates and path-depth-wise regularization.
result Theoretical proof that architectures with more skip connections converge faster.
Weight-sharing is rarely helpful in NAS, especially on small datasets.
problem The efficiency of weight-sharing in Neural Architecture Search (NAS) is questionable.
method Comparison of a state-of-the-art weight-sharing approach to random search on the nasbench dataset.
result Weight-sharing is only rarely significantly helpful in NAS, highlighting the importance of the search space.
We consider machine learning in a comparison-based setting where we are given a set of points in a metric space, but we have no access to the actual distances between the points. Instead, we can only ask an oracle whether the distance between two points i and j is smaller than the distance between the points i an…
Analyzes Lévy flights on manifolds for finding small targets.
problem Finding small targets using Lévy flights on various manifolds.
method Analytic description of Lévy flights on closed Riemannian manifolds, including asymptotics of expected stopping time.
result Computes the expected time for finding a small target by Lévy flight on surfaces.
Bonsai-Net efficiently discovers state-of-the-art models with fewer parameters.
problem Efficiently discovering state-of-the-art neural architectures with minimal computational expense.
method Bonsai-Net uses a modified differential pruner to explore a relaxed search space.
result Bonsai-Net consistently discovers better architectures than random search with fewer parameters.
Simpler neural architecture search method using random architectures and regression.
problem Complex algorithms in neural architecture search.
method Train N random architectures, use them to train a regression model, predict validation accuracies, and select top-K architectures.
result More sample efficient and competitive with complex approaches.
Petridish efficiently searches neural architectures by iteratively adding shortcut connections.
problem Finding efficient neural architectures for various tasks.
method Iteratively adds shortcut connections to existing network layers, motivated by feature selection.
result Petridish efficiently finds competitive models with few GPU days.
RPSF improves shapelet-based random forest accuracy and speed.
problem Limitations of random shapelet forest, including high training cost and limited information per node.
method Combines pairs of shapelets, omits threshold searching, and uses DMDI for interpretability.
result Improves accuracy and training speed of shapelet-based random forest.
New local-search methods close the gap in sparse tensor PCA.
problem Sparse tensor PCA underperforms compared to other methods.
method Proposes new local-search methods including greedy and random-threshold variants.
result Proves local-search methods close the gap to best known polynomial-time procedures.
Paper develops online statistical inference methods for stochastic optimization using Kiefer-Wolfowitz algorithms.
problem Online statistical inference of model parameters in stochastic optimization problems.
method Kiefer-Wolfowitz algorithm with random search directions, asymptotic distribution analysis.
result Developed valid confidence intervals for online statistical inference.
Improved Random Search for hyperparameter optimization.
problem Optimizing machine learning hyperparameters efficiently.
method Weighted Random Search with probabilistic hyperparameter updates.
result Our method outperforms standard Random Search within the same budget.