Proposes glocal hypergradient estimation for hyperparameter optimization.
problem Combining reliability and efficiency in hyperparameter optimization.
method Uses Koopman operator theory to approximate global hypergradients from local ones.
result Achieves both reliability and efficiency in hyperparameter optimization.
Study on stochastic hypergradient computation for machine learning problems.
problem Efficient computation of hypergradients in machine learning models.
method Stochastic approximation schemes for hypergradient computation, focusing on empirical risk minimization.
result Bounds for the mean square error of hypergradient approximation under contraction assumptions.
Paper proposes AggITD for efficient federated hypergradient computation.
problem Computing hypergradient in federated settings is challenging due to distributed and nonlinear construction of global Hessian matrices.
method AggITD: a novel communication-efficient federated hypergradient estimator via aggregated iterative differentiation.
result AggITD achieves the same sample complexity as AID-based approaches but with fewer communication rounds, especially in heterogeneous data environments.
MARTHE optimizes learning rates online using hypergradient approximations.
problem Optimizing task-specific learning rates for better generalization.
method Online algorithm guided by hypergradient approximations, interpolating between RTHO and HD.
result Produces more stable learning rate schedules leading to better model generalization.
Study compares methods for computing hypergradients in machine learning problems.
problem Computing exact hypergradients in machine learning is difficult.
method Investigates reverse mode iterative differentiation and approximate implicit differentiation methods.
result Unified analysis provides iteration complexity bounds and hierarchy of methods.
Semiparametric method removes bias in functional bilevel gradient estimation.
problem First-order bias in plug-in hypergradient when lower-level problem is nonparametric.
method Semiparametric debiasing theory based on efficient influence function leads to cross-fitted orthogonal hypergradient estimator.
result Asymptotic normality and uniform control over outer parameter established for the estimator.
Framework solves bilevel optimization on manifolds.
problem Solving bilevel optimization problems with manifold constraints.
method Hypergradient estimation strategies on manifolds, convergence and complexity analyses.
result Efficacy demonstrated through various applications.
NHGD solves bilevel optimization problems with reduced computational time.
problem Solving bilevel optimization problems with high computational cost.
method Exploits statistical structure of inner optimization to use empirical Fisher matrix as Hessian surrogate, enabling parallel optimization and approximation.
result NHGD achieves error bounds and sample complexity guarantees matching state-of-the-art methods, with significantly reduced computational time.
One-pass optimisation for high-dimensional hyperparameters.
problem Efficient optimisation of hyperparameters in machine learning models.
method Approximate hypergradient-based optimisation for any continuous hyperparameter, requiring only one training episode.
result Competitive performance on various datasets without hyperparameter restarts.
ROBOT framework solves regression without correspondence for large data and complex models.
problem Regression without known correspondence in large datasets.
method ROBOT framework reformulates regression as a continuous optimization problem and uses hypergradient approach.
result ROBOT achieves better performance than existing methods in linear and nonlinear regression tasks.
New method finds stationary points in bilevel optimization problems.
problem Solving nonconvex-strongly-convex bilevel optimization problems.
method Restarted Accelerated HyperGradient Descent (RAHGD) method.
result Achieves best-known theoretical guarantees for finding stationary points in bilevel optimization.
EvoGrad improves efficiency in meta-learning and hyperparameter optimization.
problem Efficiently compute hypergradients for larger network architectures.
method Uses evolutionary techniques to estimate hypergradients without second-order derivatives or longer computational graphs.
result Significant improvements in efficiency, enabling scaling to bigger architectures.
FOP improves deep learning optimizers with minimal computational overhead.
problem Training deep learning models can be hindered by high correlations and different scaling in parameter space.
method FOP uses first-order information to learn a preconditioning matrix that improves convergence without the high computational cost of second-order methods.
result FOP improves performance of standard deep learning optimizers on visual classification and reinforcement learning tasks.
We introduce a general method for improving the convergence rate of gradient-based optimizers that is easy to implement and works well in practice. We demonstrate the effectiveness of the method in a range of optimization problems by applying it to stochastic gradient descent, stochastic gradient descent with Nesterov …
First-order method solves stochastic bilevel optimization with linear constraints.
problem Stochastic bilevel optimization with linear constraints and noise.
method Developed a novel framework using gradient-based techniques and smoothed penalty functions.
result Achieved finite-time convergence guarantees for (δ,ε)-Goldstein stationary points. Automatically optimizes hyperparameters of gradient-based machine learning algorithms.
problem Tuning hyperparameters of gradient-based machine learning algorithms is tedious and requires manual derivation of hypergradients.
method Automatically computes hypergradients with a simple modification to backpropagation, allowing for recursive application to hyper-hyperparameters.
result Optimizers become less sensitive to initial hyperparameter choices as they grow taller.
OSGM uses online learning to adapt stepsize for faster convergence.
problem Improving convergence rates of first-order methods.
method OSGM combines online learning and feedback functions to adjust stepsize.
result OSGM achieves convergence rates asymptotically no worse than optimal.
La-MAML improves fast online continual learning with a look-ahead approach.
problem Fast online continual learning with limited model capacity.
method Optimisation-based meta-learning with look-ahead and episodic memory.
result Superior performance on visual classification benchmarks.
New methods solve complex optimization problems in machine learning.
problem Challenges in stochastic bilevel optimization with constraints and high variables.
method Inexact bilevel stochastic gradient methods for constrained and unconstrained lower-level problems.
result Comprehensive convergence theory for both unconstrained and constrained cases.
New algorithm achieves linear speedup in non-i.i.d. federated bilevel learning.
problem Linear speedup in convergence for non-i.i.d. datasets in federated bilevel optimization.
method Proposes FedMBO with a novel client sampling scheme for non-i.i.d. datasets.
result Achieves a convergence rate of O(1/√(nK) + 1/K + √n/K³/²).
CB-RL solves complex decision-making problems with contextual information and exogenous events.
problem Optimal policy in strategic decision-making problems that depend on environmental configuration and exogenous events.
method Contextual Bilevel Reinforcement Learning (CB-RL) with a stochastic Hyper Policy Gradient Descent (HPGD) algorithm.
result Demonstrated convergence and performance of the HPGD algorithm for reward shaping and tax design.
New Hessian-free method improves bilevel optimization for meta-learning.
problem Efficiently solving bilevel optimization problems with limited second-order information.
method Proposes a new Hessian-free method that approximates the response Jacobian matrix via optimization path differences.
result Demonstrates superior performance on meta-learning tasks compared to baseline methods.
This work speeds up hyperparameter selection for non-smooth convex models using implicit differentiation.
problem Optimizing hyperparameters of non-smooth convex models.
method Implicit differentiation of proximal gradient and coordinate descent methods.
result Implicit differentiation can speed up hyperparameter optimization, especially for non-smooth problems.
SmoothFBO tackles non-stationary functional bilevel optimization.
problem Current FBO methods are limited to static offline settings and perform poorly in online, non-stationary scenarios.
method SmoothFBO introduces a time-smoothed stochastic hypergradient estimator with a window parameter to handle non-stationarity.
result SmoothFBO achieves sublinear regret and outperforms existing methods in non-stationary hyperparameter optimization and model-based reinforcement learning.
New method optimizes fairness in predictive models for continuous sensitive attributes.
problem Enforcing full statistical independence on continuous sensitive attributes is too restrictive.
method Functional bilevel optimization (FBO) and ITD algorithms.
result Achieves lowest or near-lowest fairness-accuracy regret on synthetic and real datasets.
A new method solves bilevel optimization problems without Hessian inversion.
problem Solving bilevel optimization problems in machine learning.
method Penalty method avoiding Hessian inversion.
result Asymptotically exact hypergradient and convergence under mild conditions.
The paper analyzes convergence rates of bilevel optimization algorithms and introduces a new stochastic algorithm.
problem Nonconvex-strongly-convex bilevel optimization problems in machine learning.
method Comprehensive convergence rate analysis for deterministic bilevel optimization using AID and ITD, and a novel stochastic algorithm stocBiO.
result Theoretical convergence rates for AID and ITD methods, and stocBiO's superior performance.
Fewer data weight updates lead to faster convergence in machine learning models.
problem Improving robustness of machine learning models through data mixing.
method Analyzing convergence behavior of data mixing with a finite number of inner steps.
result The optimal number of inner steps scales with the budget and type of gradients used.
Paper tackles bilevel optimization problems using penalty methods.
problem Unconstrained and constrained bilevel optimization problems with nonsmooth lower levels.
method Introduces first-order penalty methods and O(ε−4logε−1) and O(ε−7logε−1) operation complexities. result Establishes operation complexities for finding ε-KKT solutions. New method tackles bilevel optimization with polyhedral constraints.
problem Challenges in bilevel optimization with active-set changes and expensive Hessian inversions.
method Logarithmic barrier smoothing and proxy-gradient algorithm for differentiable approximation.
result Stationarity rates of O(K−2/3) in deterministic setting and O(K−2/5) under stochastic noise. A new method solves bilevel optimization problems in competitive Markov games.
problem Capturing competitive structures in RL with multiple interacting policies.
method Penalty-augmented Nikaido-Isoda descent-ascent (PANDA) method.
result PANDA converges to stationary points without convexity assumptions.
A new stochastic method tackles bi-level optimization problems in deep learning.
problem Bi-level optimization problems in deep learning, including hyperparameter optimization and meta learning.
method Turning a BLO problem into a stochastic optimization, using SGLD MCMC and a recurrent algorithm to compute MC-estimated hypergradient.
result Our method is more robust to suboptimal inner optimization and non-unique inner minima, leading to more reliable solutions.
New algorithm tackles stochastic bilevel optimization under relaxed smoothness conditions.
problem Optimal algorithms for stochastic bilevel optimization under relaxed smoothness conditions.
method Introduces a novel fully single-loop and Hessian-inversion-free algorithmic framework for stochastic bilevel optimization.
result Demonstrates state-of-the-art oracle complexity results for multi-objective robust bilevel optimization.
This thesis analyzes and improves convergence rates of bilevel optimization algorithms in machine learning.
problem Convergence analysis and algorithm design for bilevel optimization in machine learning.
method Comprehensive convergence rate analysis for both problem-based and algorithm-based bilevel optimization formulations.
result First lower bounds and matching upper bounds for bilevel optimization, and new stochastic algorithms with lower complexity.