Efficiently tunes hyperparameters with dynamic accuracy method.
problem Optimizing machine learning hyperparameters with inexact evaluations.
method Dynamic accuracy derivative-free optimization for hyperparameter tuning.
result Demonstrates robust and efficient hyperparameter tuning compared to fixed accuracy methods.
Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.
problem The challenge of hyperparameter tuning in deep learning, especially for large models.
method Combining μ μ μ P parameterization with residual networks having a residual branch scale of 1 / e x t d e p t h 1/\sqrt{ ext{depth}} 1/ e x t d e pt h . result Optimal hyperparameters transfer across width and depth in residual networks trained with this parameterization.
HASSO improves SO algorithms by dynamically tuning hyperparameters.
problem Inefficiency of hyperparameter tuning for SO algorithms.
method HASSO is a self-adjusting SO algorithm that dynamically tunes its own hyperparameters.
result HASSO enhances the performance of various SO algorithms across different test problems.
SPICE estimates sparse linear dynamic networks without hyperparameters.
problem Estimating topology and dynamics of sparse linear dynamic networks.
method SPICE (Sparse Iterative Covariance Estimation) method in an iterative framework.
result Directly reveals the underlying topology of the network.
Optimizes deep learning models for ocean dynamics using Fourier neural operators.
problem Efficiently training deep learning models for ocean dynamics with optimal hyperparameters.
method Multiobjective hyperparameter optimization with DeepHyper for Fourier neural operators.
result Optimal hyperparameters significantly improved model performance in ocean dynamics forecasting.
Transforms game optimization dynamics into frequency domain for precise hyperparameter analysis.
problem Analyzing convergence of hyperparameters in game optimization.
method Frequency-domain framework using High-Resolution Differential Equations (HRDEs) and Laplace transforms.
result Derives precise convergence criteria for the Lookahead algorithm.
Aims to describe neural network training dynamics using two-time-scale models.
problem Lack of a general mathematical description of neural network training.
method Introduces a theoretical framework based on two-time-scale population dynamics.
result Derives selection-mutation equations and effective fitness for hyperparameters.
Develops hyperparameter transfer methods for Dense Associative Memories.
problem Challenges in transferring hyperparameters for DenseAMs due to unique architecture and activation functions.
method Derives explicit prescriptions for hyperparameter transfer from small to large models.
result Excellent agreement between theoretical and empirical results.
This research provides theoretical guarantees for hyperparameter estimation in complex network dynamical systems.
problem Theoretical guarantees for hyperparameter estimation in large, inhomogeneous complex network dynamical systems.
method Formulating the system's evolution in a measure transport perspective, proposing a theoretical framework for estimating hyperparameters with mean-type observations.
result A nonasymptotic bound for the deviation of hyperparameter estimates in inhomogeneous complex network dynamical systems with respect to network population size.
The study analyzes deep linear networks from random initialization, capturing dynamics and hyperparameter effects.
problem Understanding training dynamics in deep linear networks from random initialization.
method Theoretical analysis of gradient descent dynamics in deep linear networks with random initialization and large data.
result Captures the 'wider is better' effect and hyperparameter transfer effects, contrasting with neural-tangent parameterization.
Tuning hyperparameters of learning algorithms is hard because gradients are usually unavailable. We compute exact gradients of cross-validation performance with respect to all hyperparameters by chaining derivatives backwards through the entire training procedure. These gradients allow us to optimize thousands of hyper…
New method optimizes hyperparameters for randomized algorithms like random feature regression.
problem Optimizing hyperparameters in randomized algorithms is challenging due to their stochastic nature.
method Introduced a random objective function and used ensemble Kalman inversion (EKI) for gradient-free optimization.
result Demonstrated successful optimization of hyperparameters in various randomized algorithms.
Derives a family of hyperparameter scaling strategies for neural networks.
problem Optimizing hyperparameters for wide and deep neural networks.
method Introduces a one-parameter family of hyperparameter scaling strategies.
result Reveals proper scaling of depth with width for large-scale models.
Proposes glocal hypergradient estimation for hyperparameter optimization.
problem Combining reliability and efficiency in hyperparameter optimization.
method Uses Koopman operator theory to approximate global hypergradients from local ones.
result Achieves both reliability and efficiency in hyperparameter optimization.
Hamiltonian RNN controls hidden states gradient for long-term dependencies.
problem Challenges in learning long-term dependencies in RNNs.
method Symplectic discretization of Hamiltonian system to control gradient.
result Hamiltonian RNN outperforms other RNNs without hyperparameter optimization.
Improved sampling in generative models using CLDs with a hyperparameter.
problem Improving sampling performance in generative models.
method Extending Critically-damped Langevin Diffusions with a hyperparameter to control noise.
result Derivation of a novel upper bound on Wasserstein sampling error.
Automatically searching for optimal hyperparameter configurations is of crucial importance for applying deep learning algorithms in practice. Recently, Bayesian optimization has been proposed for optimizing hyperparameters of various machine learning algorithms. Those methods adopt probabilistic surrogate models like G…
L2O-CFGD meta-learns hyperparameters for FGD, improving performance.
problem Challenges in convergence and hyperparameter selection for FGD.
method Learning to Optimize Caputo Fractional Gradient Descent (L2O-CFGD).
result Meta-learned schedule outperforms static hyperparameters and achieves comparable performance to black-box meta-learners.
Unified framework for hyperparameter optimization and meta-learning using bilevel programming.
problem Hyperparameter optimization and meta-learning in deep learning.
method Bilevel programming approach to unify gradient-based optimization and meta-learning.
result Approximate bilevel problem solutions converge to exact problem solutions under certain conditions.
PSAEM combines EM and particle methods for efficient dynamical system learning.
problem Learning dynamical systems with stochastic approximation and particle methods.
method Particle stochastic approximation EM (PSAEM) algorithm combining stochastic approximation EM and particle Gibbs with ancestor sampling (PGAS).
result PSAEM achieves superior computational performance and convergence compared to existing methods.
We study two procedures (reverse-mode and forward-mode) for computing the gradient of the validation error with respect to the hyperparameters of any iterative learning algorithm such as stochastic gradient descent. These procedures mirror two methods of computing gradients for recurrent neural networks and have differ…
New method dynamically adjusts UTD ratio to balance under- and overfitting in RL.
problem Balancing under- and overfitting in world model learning for RL.
method Dynamic adjustment of UTD ratio based on validation performance on a small subset of experience data.
result Our method improves balance between under- and overfitting compared to default settings and competitive with extensive hyperparameter search.
A new stochastic method tackles bi-level optimization problems in deep learning.
problem Bi-level optimization problems in deep learning, including hyperparameter optimization and meta learning.
method Turning a BLO problem into a stochastic optimization, using SGLD MCMC and a recurrent algorithm to compute MC-estimated hypergradient.
result Our method is more robust to suboptimal inner optimization and non-unique inner minima, leading to more reliable solutions.
A new framework tunes hyperparameters in real-time for contextual bandits.
problem Optimizing hyperparameters for contextual bandits in real-time.
method CDT (Continuous Dynamic Tuning) framework using Zooming TS algorithm.
result Achieves sublinear regret and performs better than existing methods.
Random feature maps improve forecasting of chaotic dynamical systems.
problem Forecasting chaotic dynamical systems with high accuracy.
method Data-driven random feature maps with tanh activation, skip connections, and localization.
result Effective forecasting skill for dynamical systems with dimensions up to 512.
Improved Anderson acceleration speeds up nonlinear optimization.
problem Optimizing nonlinear functions efficiently.
method Combining Anderson acceleration with Chebyshev polynomials.
result Achieves optimal convergence rate for nonlinear problems.
AutoML-GPT uses GPT to automate AI model training.
problem Manual model selection and tuning requires significant human effort.
method Develops task-oriented prompts and utilizes LLMs for automated training.
result Achieves remarkable results in various AI tasks.
Study of two-layer ReLU neural network phase diagram at infinite-width limit.
problem Characterize the dynamical regimes of two-layer ReLU neural networks.
method Combining experimental and theoretical approaches, including phase diagram analogy.
result Identification of three regimes: linear, critical, and condensed.
New algorithms for model selection in off-policy evaluation of reinforcement learning.
problem Hyperparameter tuning for off-policy evaluation methods in reinforcement learning.
method Developed new model-free and model-based selectors with theoretical guarantees and a new experimental protocol.
result New model-free selector, LSTD-Tournament, demonstrates promising empirical performance.
Ortho-MADS optimizes SVM hyperparameters for better accuracy.
problem Optimizing hyperparameters for SVM with Gaussian kernel.
method Deterministic Mesh Adaptive Direct Search (MADS) with orthogonal directions (Ortho-MADS).
result Ortho-MADS consistently finds comparable or better solutions than other methods.
LC-SAC tackles non-stationary dynamics in reinforcement learning.
problem Degradation of deep RL methods in non-stationary environments.
method LC-SAC uses latent context encoders and contrastive loss for dynamic information capture.
result LC-SAC outperforms SAC on environments with drastic dynamics changes.
We consider a class of a nested optimization problems involving inner and outer objectives. We observe that by taking into explicit account the optimization dynamics for the inner objective it is possible to derive a general framework that unifies gradient-based hyperparameter optimization and meta-learning (or learnin…
Automates RL with sample-efficient hyperparameter optimization.
problem Challenges in applying deep RL due to hyperparameter sensitivity and inefficiency.
method Population-based AutoRL framework for meta-optimizing RL algorithms and architectures.
result Reduces the number of environment interactions needed for meta-optimization by up to an order of magnitude.
GP-TS optimizes TLM pre-training hyperparameters efficiently.
problem Resource inefficiency in TLM pre-training.
method Bayesian optimization with Thompson sampling and Gaussian process.
result GP-TS achieves lower MLM loss in fewer epochs.
A new principle for optimizer selection improves training speed and performance.
problem Finding the best optimizer hyperparameters for faster training.
method Formulate optimizer selection as maximizing the expected drop rate in loss, treating gradients and updates as signals and an optimizer as a causal filter.
result Greedy optimizer selection yields stable and effective momentum rules.
New insights on stability in reservoir computing for better performance.
problem Stability in reservoir computing networks.
method Using the recurrent kernel limit for large reservoir sizes.
result Quantitative characterization of stability and chaos frontier.
Adaptive online learning algorithms without manual tuning of a Lipschitz hyperparameter.
problem Designing adaptive online learning algorithms that require minimal user input and automatically adjust hyperparameters.
method Developing new versions of MetaGrad and Squint algorithms that dynamically update learning rates and adapt to the optimal Lipschitz hyperparameter.
result Automatic adaptation of the Lipschitz hyperparameter, improving performance and efficiency of online learning algorithms.
Annealed Langevin dynamics improves sampling from composite scores in SBI.
problem Irreducible bias in sampling from composite scores of SBI methods.
method Derive Wasserstein bounds and decision rules for hyperparameters.
result Explicit decision rules for hyperparameters guarantee prescribed sampling accuracy.
SGBD algorithm improves robustness in Bayesian sampling.
problem Inefficiency of existing MCMC algorithms in large datasets.
method Extends Barker MCMC to stochastic gradient framework, introducing bias-corrected version.
result SGBD is more robust to hyperparameter tuning and gradient noise.
PASHA optimizes model tuning for large datasets with limited resources.
problem Expensive HPO and NAS for large datasets.
method Dynamic resource allocation approach.
result Significantly reduces computational resources while maintaining performance.
Lyapunov exponents help understand RNN stability.
problem Optimizing RNNs is sensitive to various parameters.
method Use Lyapunov exponents as dynamical system tools.
result Lyapunov spectrum measures training stability.
Bayesian sparse learning method improves deep neural network efficiency.
problem Sparse learning in deep neural networks with complex geometry.
method Preconditioned stochastic gradient Langevin Dynamics (PSGLD) for sampling and adaptive optimization of hyperparameters.
result The proposed algorithm achieves asymptotic convergence with controlled bias.
This paper presents a semi-parametric algorithm for online learning of a robot inverse dynamics model. It combines the strength of the parametric and non-parametric modeling. The former exploits the rigid body dynamics equa- tion, while the latter exploits a suitable kernel function. We provide an extensive comparison …
Optuna introduces a new hyperparameter optimization framework.
problem Efficient hyperparameter optimization for machine learning models.
method Define-by-run API, efficient implementation, versatile architecture.
result Optuna is the first software meeting next-generation hyperparameter optimization criteria.
Fourier Neural Operators accurately predict dynamics of high-dimensional ionic models.
problem Approximating stiff, multiscale ionic models using neural networks.
method Fourier Neural Operators for learning dynamics of high-dimensional ionic models.
result Fourier Neural Operators can accurately predict dynamics of high-dimensional ionic models.
New framework for understanding infinite-width neural networks.
problem Understanding the infinite-width limit behavior of neural networks.
method General framework to study limit behavior of neural models based on hyperparameter scaling.
result Derives scaling for existing mean-field and neural tangent kernel limits and introduces new dynamically stable limits.
Anomalous diffusion in SGD reveals interactions between hyperparameters and Hessian.
problem Understanding the limiting dynamics of SGD in deep neural networks.
method Continuous-time model of SGD as an underdamped Langevin equation, derived for linear regression.
result Anomalous diffusion is explained by modified loss and probability currents in phase space.
A framework benchmarks and dissects one-shot neural architecture search methods.
problem Understanding the dynamics and components of one-shot NAS methods.
method General framework for one-shot NAS, benchmarking with NAS-Bench-101.
result Comparison of one-shot NAS methods and their sensitivity to hyperparameters.