New learned optimizers outperform baselines by incorporating known and novel mechanisms.
problem Understanding how learned optimizers outperform traditional ones.
method Careful analysis and visualization of learned optimizers trained on various tasks.
result Learned optimizers incorporate known techniques like momentum and gradient clipping, as well as new forms of learning rate adaptation.
VeLO learns versatile optimizers from deep learning tasks.
problem Training deep learning models with hand-designed optimizers.
method Meta-training a neural network optimizer on a wide variety of optimization tasks.
result The learned optimizer automatically adapts to different optimization tasks without hyperparameter tuning.
Study on computable online learning with new conditions and complexities.
problem Characterizing optimal online learning under varying optimality requirements.
method Introduced anytime optimal (a-optimal) online learning and explored computational separations.
result Found a computational separation between a-optimal and optimal online learning.
This work analyzes and optimizes memory and compute costs of learned optimizers.
problem High memory and compute costs of learned optimizers.
method Identified and quantified design features of learned and hand-designed optimizers, constructed a more efficient learned optimizer.
result A learned optimizer that is faster and more memory efficient than previous work.
L2O uses ML to optimize traditional optimization techniques.
problem Real-world optimization problems with shared structures.
method Exploiting shared structures to enhance optimization techniques.
result Better or faster solutions through machine learning integration.
Machine learning develops rapidly, which has made many theoretical breakthroughs and is widely applied in various fields. Optimization, as an important part of machine learning, has attracted much attention of researchers. With the exponential growth of data amount and the increase of model complexity, optimization met…
Deep RL learns optimal trading strategies.
problem Optimizing trading strategies using deep reinforcement learning.
method Deep deterministic policy gradient algorithm applied to simple trading environments.
result Deep RL can recover optimal trading strategies and achieve close-to-optimal rewards.
Learning to Optimize is a recently proposed framework for learning optimization algorithms using reinforcement learning. In this paper, we explore learning an optimization algorithm for training shallow neural nets. Such high-dimensional stochastic optimization problems present interesting challenges for existing reinf…
Bayesian optimization outperformed random search in machine learning hyperparameter tuning challenge.
problem Optimizing hyperparameters of machine learning models using derivative-free methods.
method Bayesian optimization vs. random search on real datasets.
result Bayesian optimization significantly outperformed random search in held-out objective functions.
Improved stability and generalization for blackbox learned optimizers.
problem Stability and generalization issues in blackbox learned optimizers.
method Investigation using dynamical systems, modifications to optimizer architecture and meta-training procedure.
result Improved stability and generalization of learned optimizers.
Meta-learning balances task-specific modeling and optimization complexity.
problem Balancing accurate task-specific modeling with ease of optimization in meta-learning.
method Theoretical and empirical analysis of trade-off between modeling and optimization in meta-learning.
result Explicit bounds on modeling and optimization errors for non-convex and linear regression problems.
Optimal Transport enhances machine learning with new methods.
problem Comparing and manipulating probability distributions in machine learning.
method Probabilistic framework rooted in rich history and theory.
result New solutions in generative modeling and transfer learning.
Examines learned optimizers in deep learning, revealing their dynamics and potential improvements.
problem Stability and generalization issues with learned optimizers.
method Analyzes optimization trajectories using network architecture symmetries and parameter update distributions, contrasts with manual designs.
result Identifies key insights for improving learned optimizers by leveraging strengths from both learned and manual designs.
Learning optimal feedback control laws capable of executing optimal trajectories is essential for many robotic applications. Such policies can be learned using reinforcement learning or planned using optimal control. While reinforcement learning is sample inefficient, optimal control only plans an optimal trajectory fr…
Learning to optimize has emerged as a powerful framework for various optimization and machine learning tasks. Current such "meta-optimizers" often learn in the space of continuous optimization algorithms that are point-based and uncertainty-unaware. To overcome the limitations, we propose a meta-optimizer that learns i…
This paper explores hyperparameter optimization for machine learning models.
problem Finding the best hyper-parameters for machine learning models.
method Introduces state-of-the-art optimization techniques and discusses their application.
result Comparison of different optimization methods on benchmark datasets.
We learn recurrent neural network optimizers trained on simple synthetic functions by gradient descent. We show that these learned optimizers exhibit a remarkable degree of transfer in that they can be used to efficiently optimize a broad range of derivative-free black-box functions, including Gaussian process bandits,…
Real-world applications often combine learning and optimization problems on graphs. For instance, our objective may be to cluster the graph in order to detect meaningful communities (or solve other common graph optimization problems such as facility location, maxcut, and so on). However, graphs or related attributes ar…
Proof of convergence for multi-objective optimization using inverse reinforcement learning.
problem Proving convergence in multi-objective optimization problems.
method Wasserstein inverse reinforcement learning with projective subgradient method and gradient descent.
result Convergence of inverse reinforcement learning for multi-objective optimization.
BOML unifies meta-learning methods into a common bilevel optimization framework.
problem Meta-learning methods with diverse modeling aspects.
method Modularized bilevel optimization library in Python.
result Unified solution for various meta-learning formulations.
A novel Federated Learning scheme using Optimal Transport for personalized model training.
problem Training models with data from clients having non-identically distributed data.
method Personalized Federated Learning scheme based on Optimal Transport (FedOT).
result FedOT scheme effectively transfers data from multiple distributions to a common domain and optimizes the prediction model.
Scalable tools for nested optimization in deep learning.
problem Solving nested optimization problems on a large scale in deep learning.
method Building scalable tools for bilevel optimization.
result Tools for nested optimization scale to deep learning setups.
A new method removes policy optimization in adversarial imitation learning.
problem Adversarial imitation learning's delicate alternated optimization.
method Explicitly condition discriminator on two policies, solving generator's optimization problem directly.
result Simpler approach competitive to prevalent methods.
Framework learns linear programs from optimal decisions.
problem Learning linear programs from optimal decisions is challenging.
method Gradient-based framework for learning linear programs from optimal decisions.
result Successfully learns linear programs and multi-commodity flow instances.
Paper improves adversarial training using a learned optimizer.
problem Improving robustness of deep learning models against adversarial attacks.
method Empirically identified PGD attack's limitations and used a learning-to-learn framework to train an adaptive inner optimizer.
result The proposed framework consistently improves model robustness over traditional adversarial training methods.
Proof shows imitation of expert's reward and solutions in multi-objective optimization.
problem Multi-objective optimization with reward and solution imitation.
method Wasserstein inverse reinforcement learning.
result Wasserstein inverse reinforcement learning enables imitation of expert's reward and solutions in multi-objective optimization.
Learning-rate schedules for large models match optimization theory closely, leading to better training.
problem Improving training of large models with optimal learning rates.
method Used a bound from non-smooth convex optimization theory to match learning-rate schedules with practical benefits.
result Extending the learning-rate schedule with optimal learning-rate and transferring it across schedules improves model training.
FanG-HPO optimizes machine learning models for fairness and low energy consumption.
problem Bias in machine learning models and high energy consumption in hyperparameter optimization.
method Combines multi-objective and multiple information source Bayesian optimization.
result FanG-HPO identifies fair and energy-efficient machine learning models.
Learned optimizers trained on many tasks improve model training efficiency and generalization.
problem Training models efficiently and effectively with minimal user intervention.
method Developed a neural network-based hierarchical optimizer trained on thousands of tasks.
result The learned optimizers generalize better to unseen tasks and exhibit unique behaviors.
A new method learns DAGs from data using permutation optimization.
problem Discovering latent DAGs from observational data.
method Optimizes over the Permutahedron to learn topological orderings and edges.
result Our method optimizes exact DAGs, is modular, and performs well on real-world data.
Deep learning has shown that learned functions can dramatically outperform hand-designed functions on perceptual tasks. Analogously, this suggests that learned optimizers may similarly outperform current hand-designed optimizers, especially for specific problems. However, learned optimizers are notoriously difficult to…
Paper characterizes optimal learning trajectories for high-dimensional nonlinear models.
problem Characterizing optimal learning trajectories in high-dimensional nonlinear models.
method Exploits maximum principle and dynamic programming for an optimal control problem of a gradient system.
result Constructs optimal learning trajectories leading to optimal model parameters.
This paper reviews recent advances in the field of optimization under uncertainty via a modern data lens, highlights key research challenges and promise of data-driven optimization that organically integrates machine learning and mathematical programming for decision-making under uncertainty, and identifies potential r…
BL learns interpretable optimization structures from data.
problem Learning interpretable optimization structures from data.
method BL parameterizes a compositional utility function from intrinsically interpretable modular blocks.
result BL supports architectures from single to hierarchical compositions, modeling hierarchical optimization structures.
Convex optimization models predict outputs from inputs via optimization problems.
problem Predicting outputs from inputs using convex optimization models.
method Proposed a heuristic for learning parameters of convex optimization models from datasets.
result Demonstrated the effectiveness of the proposed method on three model classes.
This paper reviews zeroth-order optimization in signal processing and machine learning.
problem Optimization problems without gradient information.
method Iterative steps: gradient estimation, descent direction computation, solution update.
result Demonstrates applications in robustness evaluation and black-box model explanations.
Deep learning improves optimal trading strategies in complex markets.
problem Finding optimal trading strategies in markets with multiple time scales.
method Deep differentiable reinforcement learning applied to optimal trading.
result Deep learning leads to more accurate and stable optimal trading strategies.
This paper proposes a new global optimization algorithm using deep learning.
problem Developing efficient algorithms for global optimization of non-convex functions.
method Two-phase approach: minimization phase with model-driven deep learning, escaping phase with reinforcement learning.
result The proposed algorithm significantly outperforms classical optimization methods and handles ill-posed functions.
SCoreBO improves Bayesian optimization by learning hyperparameters and self-correcting.
problem Efficient hyperparameter tuning for Gaussian process models in Bayesian optimization.
method Introduces SAL and SCoreBO, which prioritize hyperparameter learning and perform simultaneous optimization and learning.
result SCoreBO outperforms state-of-the-art methods on traditional benchmarks and atypical tasks.
This paper tackles optimizee generalization in L2O, improving generalization on various models.
problem L2O methods often struggle with optimizee generalization, leading to poor performance on unseen data.
method The paper introduces flatness-aware regularizers based on local entropy and Hessian to improve optimizee generalization.
result The proposed method significantly improves generalization performance on multiple sophisticated L2O models and diverse optimizees.
A new stochastic method tackles bi-level optimization problems in deep learning.
problem Bi-level optimization problems in deep learning, including hyperparameter optimization and meta learning.
method Turning a BLO problem into a stochastic optimization, using SGLD MCMC and a recurrent algorithm to compute MC-estimated hypergradient.
result Our method is more robust to suboptimal inner optimization and non-unique inner minima, leading to more reliable solutions.
When hyperparameter optimization of a machine learning algorithm is repeated for multiple datasets it is possible to transfer knowledge to an optimization run on a new dataset. We develop a new hyperparameter-free ensemble model for Bayesian optimization that is a generalization of two existing transfer learning extens…
This paper proposes a gossip-based algorithm for distributed bilevel optimization over networks.
problem Distributed bilevel optimization over networks with limited communication.
method Gossip-based distributed bilevel learning algorithm.
result Achieves optimal sample complexities for general and strongly convex objectives.
Derives a method to optimize high-dimensional functions on low-dimensional manifolds.
problem High-dimensional derivative-free optimization with high sample complexity.
method Online learning approach that learns the manifold while optimizing the function.
result Significantly reduces sample complexity compared to existing methods.
This paper introduces multi-objective hyperparameter optimization in machine learning.
problem Optimizing machine learning pipelines for multiple objectives, not just accuracy.
method Survey of optimization strategies and applications in multi-objective hyperparameter optimization.
result The importance and utility of multi-objective hyperparameter optimization in applied machine learning.
The topic of learning to solve optimization problems has received interest from both the operations research and machine learning communities. In this work, we combine techniques from both fields to address the problem of learning to generate decisions to instances of continuous optimization problems where the feasible…
Optimization geometry affects deep learning performance.
problem The impact of optimization geometry on deep learning performance.
method Analysis of pseudogradient methods for learning generalized linear models.
result Non-asymptotic bounds on generalization error characterize model performance.
Paper shows how gradient concentration helps in learning from inexact data.
problem Learning from inexact and stochastic training data.
method Combines probabilistic gradient concentration with inexact optimization techniques.
result Derives sharp test error guarantees for learning.