Optimal scaling found to depend on operator norm across large models and datasets.
problem Lack of unifying principle for optimal hyperparameter scaling across models and datasets.
method Discovered that optimal scaling is conditioned on the operator norm of the output layer.
result The optimal learning rate/batch size pair (η∗,B∗) consistently has the same operator norm value. EF21-Muon optimizes deep learning with error feedback, improving efficiency and accuracy.
problem Lack of principled distributed frameworks for non-Euclidean LMO-based optimizers.
method Introduces EF21-Muon, a communication-efficient, non-Euclidean LMO-based optimizer with convergence guarantees.
result First efficient distributed implementation of non-Euclidean LMO-based optimizers, achieving up to 7x communication savings.
Gluon optimizes LMO-based methods for large-scale tasks, improving performance and theory-practice gap.
problem LMO-based methods lack theoretical support for practical implementation and smoothness assumptions.
method Introduces Gluon, a new LMO-based method with refined smoothness model.
result Gluon's theoretical stepsizes match fine-tuned values, closing the theory-practice gap.
New optimization method combines gradient clipping and non-Euclidean smoothness.
problem Improving optimization in non-Euclidean spaces for machine learning.
method Hybrid of steepest descent and conditional gradient, incorporating weight decay.
result Achieves optimal convergence rate and demonstrates effectiveness in deep learning.
Muon optimizes Transformer training with heavy-tailed data, achieving optimal sample complexity.
problem Theoretical understanding of non-Euclidean optimisation methods for heavy-tailed data in training Transformers.
method Addressing the gap in theoretical understanding, we show Muon achieves optimal sample complexity under heavy-tailed noise.
result Muon finds an ε-stationary point in nuclear norm with optimal sample complexity, absorbing heavy-tailed noise without dimension dependence.
New analysis reveals batch size effects on stochastic conditional gradient methods.
problem Understanding the role of batch size in stochastic conditional gradient methods.
method Deriving a new analysis focusing on momentum-based stochastic conditional gradient algorithms (e.g., Scion).
result Increasing batch size initially improves optimization accuracy but can degrade performance beyond a critical threshold.
Non-Euclidean BPM extends optimization theory to non-Euclidean norms.
problem Extending BPM's convergence theory to non-Euclidean norms.
method Iteratively minimizing over norm balls in non-Euclidean geometry.
result Most BPM guarantees carry over to non-Euclidean norms.
New optimizer designs respect symmetry, improving deep learning models.
problem Optimizers lack respect for symmetry in neural networks.
method Introduce symmetry-compatible principle for optimizer design.
result Symmetry-compatible optimizers improve model performance.
Bayesian optimization reduces computational effort in aircraft design optimization.
problem High computational cost in industrial aircraft design optimization.
method Constrained Bayesian optimization (Super Efficient Global Optimization with Mixture of Experts)
result Significant computational efficiency improvements over existing Isight optimizers.
Bayesian optimization method tackles combinatorial spaces, scalable for large data.
problem Optimization over combinatorial categorical spaces in natural sciences.
method Combines variational optimization and continuous relaxations for gradient-based optimization.
result Method performs comparably to state-of-the-art methods while scaling well.
New algorithm solves complex stopping problems with robust optimization.
problem Solving complex stochastic optimal stopping problems.
method Simulation-based robust optimization with exact reformulation as a zero-one bilinear program.
result Developed polynomial-time heuristics and algorithms for practical solution.
L2O uses ML to optimize traditional optimization techniques.
problem Real-world optimization problems with shared structures.
method Exploiting shared structures to enhance optimization techniques.
result Better or faster solutions through machine learning integration.
When hyperparameter optimization of a machine learning algorithm is repeated for multiple datasets it is possible to transfer knowledge to an optimization run on a new dataset. We develop a new hyperparameter-free ensemble model for Bayesian optimization that is a generalization of two existing transfer learning extens…
Meta algorithm solves multivariate optimization using univariate optimizers.
problem Multivariate global optimization problems.
method Meta algorithm combining univariate global optimizers.
result Meta algorithm provides robust regret guarantees.
A novel neural network approach for optimization problems.
problem Constrained optimization problems.
method Neural Optimization Machine (NOM) using a specially designed NN architecture and training procedure.
result Solves optimization problems efficiently, especially in high-dimensional spaces.
New algorithms ensure reproducibility and optimal convergence in convex optimization.
problem Trade-off between reproducibility and convergence rate in convex optimization.
method Regularization-based algorithms for smooth convex minimization and minimax optimization.
result Achieves optimal reproducibility and near-optimal gradient complexity for various oracle settings.
New algorithm selects robust martingale for optimal stopping problems.
problem Optimal stopping problems in stochastic processes.
method Randomized dual martingale minimization algorithm.
result Efficiently selects Doob martingale as close as possible.
Numerical optimization is an important tool in the field of computational physics in general and in nano-optics in specific. It has attracted attention with the increase in complexity of structures that can be realized with nowadays nano-fabrication technologies for which a rational design is no longer feasible. Also, …
This paper shows how to combine optimal tests into log-optimal processes.
problem How to combine optimal sequential tests into log-optimal processes.
method Using a new class of WAIT e-processes, the paper aggregates asymptotically optimal sequential tests into asymptotically log-optimal processes.
result It is possible to aggregate asymptotically optimal sequential tests into asymptotically log-optimal e-processes.
Learning optimal feedback control laws capable of executing optimal trajectories is essential for many robotic applications. Such policies can be learned using reinforcement learning or planned using optimal control. While reinforcement learning is sample inefficient, optimal control only plans an optimal trajectory fr…
Paper studies optimal control for a specific geometric problem.
problem Optimal control problem associated with the Paneitz obstacle problem.
method Existence and regularity results for optimal controls.
result Existence of optimal controls and their properties.
New algorithm AG-OG optimizes separable convex-concave problems efficiently.
problem Efficiently solving separable convex-concave minimax optimization problems.
method Leverages Nesterov acceleration and optimistic gradient on component and coupling parts of the problem.
result Achieves optimal convergence rate for various settings including bilinearly coupled problems.
Adapts Bayesian optimization for mixed constraints in aircraft design.
problem Optimizing expensive black box functions with mixed constraints.
method Super efficient global optimization with upper trust bound for constraints, Gaussian process uncertainty, refinement procedure.
result Superior performance on aircraft design problem compared to state-of-the-art solvers.
Adam optimizer converges to zeros of a new vector field, not just gradient zeros.
problem Prove convergence rates for Adam optimizer in simple quadratic optimization problems.
method Introduced Adam vector field to analyze Adam optimizer's convergence.
result Established optimal convergence rates for Adam optimizer.
Optimal crypto asset routing with CFMMs, including fixed costs.
problem Optimizing order execution on a network of CFMMs with fixed costs.
method Convex optimization for no fixed costs, mixed-integer convex for fixed costs, heuristics for approximate solutions.
result Approximate solutions to optimal routing and arbitrage certification problems.
BOSH optimizes functions with stochastic evaluations more efficiently and precisely.
problem Optimizing functions with noisy evaluations can lead to suboptimal solutions.
method BOSH uses a hierarchical Gaussian process to generate a growing pool of realizations.
result BOSH provides more efficient and higher-precision optimization than standard BO.
VeLO learns versatile optimizers from deep learning tasks.
problem Training deep learning models with hand-designed optimizers.
method Meta-training a neural network optimizer on a wide variety of optimization tasks.
result The learned optimizer automatically adapts to different optimization tasks without hyperparameter tuning.
New learned optimizers outperform baselines by incorporating known and novel mechanisms.
problem Understanding how learned optimizers outperform traditional ones.
method Careful analysis and visualization of learned optimizers trained on various tasks.
result Learned optimizers incorporate known techniques like momentum and gradient clipping, as well as new forms of learning rate adaptation.
PAGE optimizes nonconvex problems with optimal convergence rates.
problem Nonconvex optimization problems.
method PAGE algorithm for achieving optimal convergence rates.
result PAGE achieves optimal convergence rates for nonconvex optimization.
A new method learns DAGs from data using permutation optimization.
problem Discovering latent DAGs from observational data.
method Optimizes over the Permutahedron to learn topological orderings and edges.
result Our method optimizes exact DAGs, is modular, and performs well on real-world data.
Develops a new method for efficient stochastic bilevel optimization.
problem Stochastic bilevel optimization problems in machine learning applications.
method Single-Timescale stochAstic BiLevEl optimization (STABLE) method.
result Achieves the same order of sample complexity as stochastic gradient descent for single-level optimization.
Enhanced ROOT-SGD optimizes stochastic optimization with diminishing stepsizes.
problem Improving statistical efficiency in stochastic optimization.
method Integrates a diminishing stepsize strategy into ROOT-SGD.
result Achieves optimal convergence rates with improved stability and precision.
Convex optimization models predict outputs from inputs via optimization problems.
problem Predicting outputs from inputs using convex optimization models.
method Proposed a heuristic for learning parameters of convex optimization models from datasets.
result Demonstrated the effectiveness of the proposed method on three model classes.
A new approach for efficient batch multiobjective optimization using Thompson sampling.
problem Inefficient batch multiobjective optimization due to expensive oracles and hard inner optimization.
method Proposes a Thompson sampling approach (qextttPOTS) that chooses Pareto optimal candidates sequentially. result Empirically superior performance compared to classical evolutionary approaches and MOBO.
New framework for decentralized optimization of upper-linearizable functions with improved regret and complexity.
problem Decentralized optimization of upper-linearizable functions with general constraints.
method Decentralized projection-free optimization with upper-linearizable function framework.
result Regret of O(T1−θ/2) with communication complexity of O(Tθ) and linear optimization calls of O(T2θ). New method optimizes portfolio weights as functions, outperforming traditional approaches.
problem Optimizing portfolio weights in mean-variance models.
method Functional optimization approach, treating weights as functions of past values.
result Gradient-ascent algorithms can solve functional optimization problems for mean-variance portfolio management.
Improved stability and generalization for blackbox learned optimizers.
problem Stability and generalization issues in blackbox learned optimizers.
method Investigation using dynamical systems, modifications to optimizer architecture and meta-training procedure.
result Improved stability and generalization of learned optimizers.
PSO improves G-optimal designs for up to 5 factors, reducing computation time.
problem Computing highly G-optimal designs for response surface models is computationally expensive. method Extended Particle Swarm Optimization (PSO) for optimal design problems.
result PSO generates improved G-optimal designs for up to 5 factors with comparable computational cost. We develop the first Bayesian Optimization algorithm, BLOSSOM, which selects between multiple alternative acquisition functions and traditional local optimization at each step. This is combined with a novel stopping condition based on expected regret. This pairing allows us to obtain the best characteristics of both lo…
This work analyzes and optimizes memory and compute costs of learned optimizers.
problem High memory and compute costs of learned optimizers.
method Identified and quantified design features of learned and hand-designed optimizers, constructed a more efficient learned optimizer.
result A learned optimizer that is faster and more memory efficient than previous work.
Bayesian optimization outperformed random search in machine learning hyperparameter tuning challenge.
problem Optimizing hyperparameters of machine learning models using derivative-free methods.
method Bayesian optimization vs. random search on real datasets.
result Bayesian optimization significantly outperformed random search in held-out objective functions.
Optimizes shapes on non-standard manifolds.
problem Optimization on non-standard infinite-dimensional manifolds.
method Develops gradient descent on weak Riemannian manifolds.
result Establishes foundational properties for optimization on various weak Riemannian manifolds.
This paper introduces multi-objective hyperparameter optimization in machine learning.
problem Optimizing machine learning pipelines for multiple objectives, not just accuracy.
method Survey of optimization strategies and applications in multi-objective hyperparameter optimization.
result The importance and utility of multi-objective hyperparameter optimization in applied machine learning.
New framework solves low-rank optimization problems to certifiable optimality.
problem Low-rank optimization problems with certifiable solutions.
method Mixed-Projection Conic Optimization framework using symmetric projection matrices and outer-approximation algorithms.
result Solves low-rank problems to certifiable optimality, outperforming existing methods.
Simulated annealing improves candidate optimization for multi-objective Bayesian optimization.
problem Efficient candidate optimization for multi-objective acquisition functions in Bayesian optimization.
method Simulated annealing-based approach for batch acquisition function optimization.
result Simulated annealing outperforms SLSQP in most multi-objective optimization problems, achieving higher hypervolume values and better convergence characteristics.
BILBO optimizes bilevel problems without repeated lower-level optimizations.
problem Challenges in bilevel optimization, especially in noisy, constrained, and derivative-free settings.
method BILevel Bayesian Optimization (BILBO) that optimizes both levels simultaneously, using confidence-bounds and function query selection.
result Theoretical and empirical evidence of BILBO's effectiveness on various problems.
GRU models with Adam optimizer outperform other combinations in stock market forecasting.
problem Comparing optimization techniques for time series forecasting in LSTM and GRU networks.
method Examined Adam and Nesterov Accelerated Gradient (NAG) on LSTM and GRU models for stock market forecasting.
result GRU models with Adam optimizer produced the lowest RMSE and outperformed other combinations.
Optimal best-arm identification with known number of optimal arms.
problem Identifying the best arm in a multi-armed bandit with multiple optimal arms under fixed confidence.
method Deriving a new information-theoretic lower bound and proposing a modified stopping rule.
result Achieving asymptotic instance-optimality with a new lower bound and new stopping rule.