This paper proposes automatic tuning of Bayesian Optimization's acquisition function.
problem Optimizing black-box functions with noisy, expensive evaluations and hyperparameter tuning.
method Exploring heuristics to automatically tune acquisition functions in Bayesian Optimization.
result Demonstrates effectiveness of heuristics in automatic Bayesian Optimization.
D-Adaptation automatically sets optimal learning rates without manual tuning.
problem Optimizing learning rates for efficient convergence in machine learning.
method D-Adaptation, which asymptotically achieves optimal learning rates without back-tracking or additional evaluations.
result D-Adaptation automatically matches hand-tuned learning rates across diverse problems.
Develops algorithm to differentiate Metropolis-Hastings for optimization.
problem Optimizing intractable densities with discrete components.
method Fuses stochastic automatic differentiation with Markov chain coupling schemes.
result Unbiased and low-variance gradient estimator for intractable densities.
Novel approach trains ASR models with less supervision using bilevel optimization.
problem Training acoustic models for ASR with minimal supervision.
method Bilevel optimization with unsupervised and supervised losses.
result Achieves superior performance compared to existing methods.
AutoGD automatically adjusts learning rates for gradient descent.
problem Optimizing learning rates for gradient descent methods.
method AutoGD automatically adjusts learning rates based on iteration.
result AutoGD can recover the optimal rate of GD for a broad class of functions.
A new method for automatically aligning and clustering time series data.
problem Challenges in aligning and clustering time series data, especially without a template signal.
method TROUT (Temporal Registration using Optimal Unitary Transformations) method based on a novel dissimilarity measure.
result TROUT outperforms competitors in clustering time series data.
A new method automatically and dynamically sets learning rates in deep learning.
problem Determining the appropriate learning rate in deep learning tasks is challenging and often subjective.
method Local Quadratic Approximation (LQA) to automatically and dynamically set learning rates.
result The proposed method leads to nearly optimal learning rates in a computationally efficient way.
We present a system for the automatic differentiation of a higher-order functional array-processing language. The core functional language underlying this system simultaneously supports both source-to-source automatic differentiation and global optimizations such as loop transformations. Thanks to this feature, we demo…
Machine learning algorithms have made remarkable achievements in the field of artificial intelligence. However, most machine learning algorithms are sensitive to the hyper-parameters. Manually optimizing the hyper-parameters is a common method of hyper-parameter tuning. However, it is costly and empirically dependent. …
Proposes efficient, modular method for implicit differentiation.
problem Implicit differentiation of optimization problems.
method Automatic implicit differentiation using autodiff and implicit function theorem.
result Automatic differentiation of optimization problems is made easier and more modular.
New method speeds up Bayesian optimization in high dimensions.
problem High-dimensional expensive function optimization struggles.
method Structured automatic differentiation for kernel matrices.
result First-order Bayesian optimization scalable to high dimensions.
In min-min optimization or max-min optimization, one has to compute the gradient of a function defined as a minimum. In most cases, the minimum has no closed-form, and an approximation is obtained via an iterative algorithm. There are two usual ways of estimating the gradient of the function: using either an analytic f…
Purpose: Machine learning is broadly used for clinical data analysis. Before training a model, a machine learning algorithm must be selected. Also, the values of one or more model parameters termed hyper-parameters must be set. Selecting algorithms and hyper-parameter values requires advanced machine learning knowledge…
Random Forests automatically prune a latent 'true' tree, explaining their overfitting without tuning.
problem Difficulty in building bad Random Forests and overfitting without apparent consequences.
method Bootstrap aggregation and model perturbation in Random Forests.
result Randomized ensembles implicitly perform optimal early stopping out-of-sample, explaining overfitting.
Amazon SageMaker AMT automates machine learning model tuning.
problem Challenging hyperparameter tuning for complex machine learning systems.
method Gradient-free optimization using random search or Bayesian optimization.
result AMT finds the best hyperparameter configurations for models.
This paper works out fair values of stock loan model with automatic termination clause, cap and margin. This stock loan is treated as a generalized perpetual American option with possibly negative interest rate and some constraints. Since it helps a bank to control the risk, the banks charge less service fees compared …
AutoSGD automatically adjusts learning rates for SGD.
problem Inefficient manual learning rate selection for SGD.
method AutoSGD automatically adjusts learning rates based on iteration.
result Empirical results show strong performance across various optimization tasks.
Paper learns dictionaries for sparse signal recovery using automatic differentiation.
problem Learning dictionaries for sparse signal recovery from noisy data.
method Approximates reconstructions using FB algorithm and learns dictionaries with projected gradient descent.
result Successfully learns 1D TV dictionary from piecewise constant signals.
Automatic Chemical Design is a framework for generating novel molecules with optimized properties. The original scheme, featuring Bayesian optimization over the latent space of a variational autoencoder, suffers from the pathology that it tends to produce invalid molecular structures. First, we demonstrate empirically …
A JAX toolbox solves optimal transport problems for point clouds and histograms.
problem Optimal transport problems between point clouds and histograms.
method Automatic and custom reverse mode differentiation, vectorization, just-in-time compilation, and accelerators support.
result Solves a wide range of optimal transport problems including regularized OT, barycenters, Gromov-Wasserstein, and low-rank solvers.
Variational inference is a scalable technique for approximate Bayesian inference. Deriving variational inference algorithms requires tedious model-specific calculations; this makes it difficult to automate. We propose an automatic variational inference algorithm, automatic differentiation variational inference (ADVI). …
VeLO learns versatile optimizers from deep learning tasks.
problem Training deep learning models with hand-designed optimizers.
method Meta-training a neural network optimizer on a wide variety of optimization tasks.
result The learned optimizer automatically adapts to different optimization tasks without hyperparameter tuning.
A new method for automatic gradient tree boosting using information theory.
problem Automatic selection of tree complexity and number in gradient boosting.
method Optimism of greedy leaf splitting procedure modeled as a Cox-Ingersoll-Ross process, leading to an information criterion for model selection.
result The method achieves significant speedups (10-1400) compared to xgboost without sacrificing predictive power.
MARS automatically selects tensor decomposition ranks, improving performance in neural network tasks.
problem Determining optimal decomposition ranks in tensor decompositions.
method MARS uses binary masks to learn optimal tensor structure during training via relaxed MAP estimation.
result MARS achieves better results than previous methods in various tasks.
In this paper, a neural architecture search (NAS) framework is proposed for 3D medical image segmentation, to automatically optimize a neural architecture from a large design space. Our NAS framework searches the structure of each layer including neural connectivities and operation types in both of the encoder and deco…
Machine learning algorithms frequently require careful tuning of model hyperparameters, regularization terms, and optimization parameters. Unfortunately, this tuning is often a "black art" that requires expert experience, unwritten rules of thumb, or sometimes brute-force search. Much more appealing is the idea of deve…
Meta-SAC automatically tunes SAC's entropy temperature for better exploration.
problem Exploration-exploitation dilemma in reinforcement learning.
method Meta-SAC uses metagradient and a novel meta objective to automatically adjust SAC's entropy temperature.
result Meta-SAC outperforms SAC-v2 by 10% on the humanoid-v2 task.
Automatically learns optimal data augmentation for image classification.
problem Finding optimal data augmentation hyperparameters is computationally demanding and requires domain knowledge.
method Proposes an online bilevel optimization framework to learn data augmentation parameters directly.
result Jointly trained method achieves comparable or better classification accuracy than hand-crafted data augmentation without an external validation loop.
DoWG optimizer automatically adapts to convex and nonsmooth problems without tuning.
problem Optimizing machine learning models efficiently and adaptively.
method DoWG uses a distance-based weighted version of gradient averaging for optimization.
result DoWG achieves convergence rates similar to optimally tuned gradient descent.
In the classic sparsity-driven problems, the fundamental L-1 penalty method has been shown to have good performance in reconstructing signals for a wide range of problems. However this performance relies on a good choice of penalty weight which is often found from empirical experiments. We propose an algorithm called t…
This study optimizes crypto-market trading conditions without assuming convexity.
problem Optimizing crypto-market trading conditions without convexity.
method Rigorous mathematical analysis of constant function market makers under quasilinear trade functions.
result Quasilinear trade functions can replicate convex functions' robustness against arbitrage.
Automatic machine learning performs predictive modeling with high performing machine learning tools without human interference. This is achieved by making machine learning applications parameter-free, i.e. only a dataset is provided while the complete model selection and model building process is handled internally thr…
Teaches matrix calculus for machine learning and optimization.
problem Computing derivatives of functions involving matrices.
method Extends differential calculus to vector spaces, focusing on practical applications in machine learning.
result Introduction of adjoint and reverse-mode differentiation for efficient computation.
Bayesian optimization has been proposed as a practical and efficient tool through which to tune parameters in many difficult settings. Recently, such techniques have been combined with real-time fMRI to propose a novel framework which turns on its head the conventional functional neuroimaging approach. This closed-loop…
GP model for time series forecasting with priors.
problem Automatic selection of optimal kernels and reliable estimation of hyperparameters.
method Fixed composition of kernels, automatic relevance determination (ARD), empirical Bayes priors.
result GP model is more accurate than state-of-the-art models.
Optimizes contrastive learning with individualized temperatures for better performance on imbalanced datasets.
problem The common practice of using a global temperature parameter ignores the varying semantic similarity across different anchor data.
method Proposes a new robust contrastive loss inspired by distributionally robust optimization (DRO) and an efficient stochastic algorithm for automatic temperature individualization.
result Our method automatically learns a suitable temperature for each sample, improving performance on imbalanced datasets.
AutoInit automatically finds good neural network initialization.
problem Finding optimal neural network initialization is crucial but time-consuming.
method Uses Jacobian tuning to automatically adjust network hyperparameters.
result The method finds good initialization for various network architectures.
In this paper, we propose a simple, fast and easy to implement algorithm LOSSGRAD (locally optimal step-size in gradient descent), which automatically modifies the step-size in gradient descent during neural networks training. Given a function f, a point x, and the gradient ∇xf of f, we aim to find the s…
HyperImpute improves iterative imputation by automatically selecting models and hyperparameters.
problem Imputing missing values in datasets with variable model specifications.
method Generalized iterative imputation framework that adapts and configures models and hyperparameters automatically.
result Demonstrates superior imputation accuracy compared to benchmarks.
Proposes baselines for joint NAS and HPO optimization.
problem Joint optimization of neural architecture and hyperparameters for multiple objectives.
method Extends existing methods to jointly optimize with multiple objectives.
result Serves as simple baselines for future multi-objective joint NAS + HPO research.
RAD estimates gradients with less memory, faster than small batch sizes.
problem Training deep models with stochastic gradient descent requires exact gradients, but they are not needed.
method Developed a framework for randomized automatic differentiation (RAD) to compute unbiased gradient estimates with reduced memory.
result RAD converges in fewer iterations than using a small batch size for feedforward networks and similar number for recurrent networks.
Develops a new flexible grid trading model using ANN and SSO.
problem Improving automated trading strategies in financial markets.
method Combines SSO algorithm with ANN for optimizing trading parameters.
result Provides a robust and efficient trading model with better returns.
A new method for optimizing language-based agentic systems using semantic backpropagation.
problem Lack of proper feedback assignment in optimizing agentic systems.
method Formalization of semantic backpropagation with semantic gradients and semantic gradient descent.
result Our method outperforms existing state-of-the-art methods for solving GASO problems.
In deep learning, performance is strongly affected by the choice of architecture and hyperparameters. While there has been extensive work on automatic hyperparameter optimization for simple spaces, complex spaces such as the space of deep architectures remain largely unexplored. As a result, the choice of architecture …
New BO method efficiently optimizes high-dimensional functions by automatically selecting variables.
problem Efficiently optimizing functions with high-dimensional domains.
method Exploits variable selection to automatically learn sub-spaces without pre-specified dimensions.
result Empirically validated on synthetic and real problems, demonstrating efficiency.
The paper generalizes optimization algorithms using category theory.
problem Optimizing functions in a category-theoretic setting.
method Using the Cartesian reverse derivative to generalize gradient descent and Newton's method.
result Properties of optimization algorithms are preserved in the generalized setting, including invariances and convergence.
AutoBayes simplifies variational inference by composing models and optimizing them.
problem Generalized variational inference complexities and optimization challenges.
method Compositional framework exploiting chain rules for automatic differentiation.
result Optimized models and parameterized statistical games can be locally optimized.
Optimized CNNs for AMC on edge devices reduce complexity without sacrificing accuracy.
problem Developing efficient DL models for AMC on resource-constrained edge devices.
method Pruning, quantization, and knowledge distillation techniques applied to CNNs.
result Optimized models maintain or improve AMC accuracy with reduced complexity.