New algorithm solves complex optimization problems efficiently.
problem Minimizing convex upper-level functions over optimal lower-level solutions.
method Reformulates bilevel problems into functionally constrained problems, achieving near-optimal rates.
result Achieves near-optimal rates for both smooth and nonsmooth problems.
Novel method for bilevel optimization with convex lower-level problem.
problem Minimizing a smooth objective over the optimal solution set of a convex constrained problem.
method Local cutting plane approximation of lower-level solution set combined with conditional gradient updates.
result Achieves optimal iteration complexity for the considered class of bilevel problems.
New framework solves dynamic bilevel optimization problems in reinforcement learning.
problem Dynamic objective functions in reinforcement learning and human feedback.
method Principled penalty-based methods for bilevel reinforcement learning.
result Demonstrated effectiveness of penalty-based algorithms in simulations.
Develops an algorithm for bilevel optimization with coupled constraints.
problem Challenges in bilevel optimization with coupled constraints.
method Primal-dual-assisted penalty approach and a fully first-order algorithm (BLOCC).
result Established rigorous convergence theory and demonstrated effectiveness on real-world applications.
Paper tackles Hessian/Jacobian-free stochastic bilevel optimization with O ( ε − 1.5 ) {O}(ε^{-1.5}) O ( ε − 1.5 ) complexity.
problem Nonconvex-strongly-convex bilevel optimization problem.
method FdeHBO optimizer with finite-difference Hessian/Jacobian-vector approximation and momentum.
result FdeHBO achieves O ( ε − 1.5 ) {O}(ε^{-1.5}) O ( ε − 1.5 ) iterations for ε ε ε -accurate stationary point. Paper resolves ambiguity in non-convex bilevel optimization problems.
problem Ambiguity in bilevel optimization with non-convex lower-level objectives.
method Introduces selection maps to define critical points and resolves ambiguity.
result Validates new analytical tools in Morse theory for implicit differentiation.
Semiparametric method removes bias in functional bilevel gradient estimation.
problem First-order bias in plug-in hypergradient when lower-level problem is nonparametric.
method Semiparametric debiasing theory based on efficient influence function leads to cross-fitted orthogonal hypergradient estimator.
result Asymptotic normality and uniform control over outer parameter established for the estimator.
New method creates coresets for deep neural networks efficiently.
problem Efficiently handling large data streams with limited resources.
method Cardinality-constrained bilevel optimization for deep neural networks.
result Demonstrated efficient generation of coresets for deep neural networks.
New method solves complex optimization problems faster.
problem Minimizing a convex smooth objective over the optimal solution set of another convex smooth problem.
method Uses a cutting plane approach to approximate the lower-level problem and an accelerated gradient method to update the upper-level objective.
result Shows that the method requires at most O ( max { 1 / ε f , 1 / ε g } ) \mathcal{O}(\max\{1/\sqrt{ε_{f}}, 1/ε_g\}) O ( max { 1/ ε f , 1/ ε g }) iterations to achieve ε f ε_f ε f -suboptimality and ε g ε_g ε g -infeasibility. New methods optimize complex optimization problems with improved efficiency.
problem Optimizing complex problems with a convex lower-level objective.
method Uses stochastic cutting planes and conditional gradient updates.
result Improves complexity for both convex and non-convex upper-level functions.
The paper compares unrolling and bilevel optimization for learning variational models.
problem Learning variational models in supervised learning.
method Analyzes unrolling and bilevel optimization approaches for variational models.
result Unrolling can be better than bilevel optimization, but performance depends on parameters.
New method solves complex bilevel optimization problems.
problem Challenging bilevel optimization problems in machine learning.
method Penalty-based bilevel gradient descent (PBGD) algorithm.
result PBGD algorithm converges for constrained bilevel problems.
A new framework for coreset selection in machine learning models.
problem Learning models under resource constraints.
method Formulates coreset selection as a bilevel optimization problem.
result Framework applies to any twice differentiable model, including neural networks.
This thesis analyzes and improves convergence rates of bilevel optimization algorithms in machine learning.
problem Convergence analysis and algorithm design for bilevel optimization in machine learning.
method Comprehensive convergence rate analysis for both problem-based and algorithm-based bilevel optimization formulations.
result First lower bounds and matching upper bounds for bilevel optimization, and new stochastic algorithms with lower complexity.
SimFBO simplifies FBO, making it more efficient and flexible.
problem Complex nested optimization in machine learning and edge computing.
method Proposes SimFBO, a simple and flexible FBO framework with improved communication efficiency.
result SimFBO and ShroFBO achieve linear convergence speedup and improved sample and communication complexities.
Framework solves bilevel optimization on manifolds.
problem Solving bilevel optimization problems with manifold constraints.
method Hypergradient estimation strategies on manifolds, convergence and complexity analyses.
result Efficacy demonstrated through various applications.
Develops a new method for efficient stochastic bilevel optimization.
problem Stochastic bilevel optimization problems in machine learning applications.
method Single-Timescale stochAstic BiLevEl optimization (STABLE) method.
result Achieves the same order of sample complexity as stochastic gradient descent for single-level optimization.
A new algorithm tackles bilevel optimization with multiple inner minima.
problem Challenges in bilevel optimization with multiple inner minima.
method Reformulated as constrained optimization, solved via primal-dual bilevel optimization (PDBO) algorithm.
result First non-asymptotic convergence guarantee for bilevel optimization with multiple inner minima.
New analysis improves understanding of bilevel optimization stability and generalization.
problem Understanding how well bilevel optimization algorithms generalize.
method Algorithmic stability arguments and generalization bounds for three bilevel minimax solvers.
result Precise trade-off between algorithmic stability, generalization gaps, and practical settings.
Optimizes bilevel empirical risk minimization with improved oracle calls.
problem Optimizing bilevel empirical risk minimization problems.
method Proposes a bilevel extension of the SARAH algorithm.
result Demonstrates improved oracle calls to achieve stationarity.
Paper tackles efficient SGD methods for constrained bilevel optimization.
problem Stochastic bilevel optimization with equality constraints.
method Alternating implicit projected SGD and its variants.
result Achieves sample complexity matching state-of-the-art for unconstrained problems.
The paper analyzes convergence rates of bilevel optimization algorithms and introduces a new stochastic algorithm.
problem Nonconvex-strongly-convex bilevel optimization problems in machine learning.
method Comprehensive convergence rate analysis for deterministic bilevel optimization using AID and ITD, and a novel stochastic algorithm stocBiO.
result Theoretical convergence rates for AID and ITD methods, and stocBiO's superior performance.
New Hessian-free method improves bilevel optimization for meta-learning.
problem Efficiently solving bilevel optimization problems with limited second-order information.
method Proposes a new Hessian-free method that approximates the response Jacobian matrix via optimization path differences.
result Demonstrates superior performance on meta-learning tasks compared to baseline methods.
Unified framework for decentralized bilevel optimization with various heterogeneity-correction strategies.
problem Decentralized bilevel optimization with neighborhood communications and data heterogeneity.
method SPARKLE: Single-loop Primal-dual Algorithm for decentralized bilevel optimization, incorporating various heterogeneity-correction techniques.
result Unified convergence analysis for SPARKLE with state-of-the-art convergence rates compared to existing algorithms.
This paper improves SAM by reformulating it as a bilevel optimization problem.
problem Improving Sharpness-Aware Minimization (SAM) for better performance.
method Reformulate SAM as a bilevel optimization problem using a 0-1 loss surrogate.
result BiSAM consistently results in improved performance compared to SAM and its variants.
New algorithms reduce bilevel optimization complexity to ε^(-1.5).
problem Efficiently solving bilevel optimization problems in machine learning.
method Proposed two new algorithms: one using momentum-based recursive iterations, the other using recursive gradient estimations.
result Achieved computational complexity of ε^(-1.5), significantly faster than previous methods.
SmoothFBO tackles non-stationary functional bilevel optimization.
problem Current FBO methods are limited to static offline settings and perform poorly in online, non-stationary scenarios.
method SmoothFBO introduces a time-smoothed stochastic hypergradient estimator with a window parameter to handle non-stationarity.
result SmoothFBO achieves sublinear regret and outperforms existing methods in non-stationary hyperparameter optimization and model-based reinforcement learning.
This paper proposes a gossip-based algorithm for distributed bilevel optimization over networks.
problem Distributed bilevel optimization over networks with limited communication.
method Gossip-based distributed bilevel learning algorithm.
result Achieves optimal sample complexities for general and strongly convex objectives.
A new method optimizes diffusion models for fine-tuning tasks efficiently.
problem Optimizing diffusion models for downstream tasks using nested bilevel structures.
method Formalizes the challenge as a generative bilevel optimization problem and introduces a first-order bilevel framework.
result Our method outperforms existing fine-tuning and hyperparameter search baselines.
New lower bounds for bilevel optimization with first-order oracles.
problem Complexity of bilevel optimization with first-order oracles.
method Development of hard instances and proof of lower bounds.
result Nontrivial lower bounds for first-order zero-respecting algorithms.
Paper proposes SMO for solving bilevel optimization problems efficiently.
problem Solving bilevel optimization problems with nonsmooth convex lower-level and nonconvex upper-level objectives.
method Sequential minimax optimization (SMO) method using modified augmented Lagrangian and penalty schemes.
result Improves operation complexity for finding ε \varepsilon ε -KKT solutions. Novel approach trains ASR models with less supervision using bilevel optimization.
problem Training acoustic models for ASR with minimal supervision.
method Bilevel optimization with unsupervised and supervised losses.
result Achieves superior performance compared to existing methods.
New algorithm tackles stochastic bilevel optimization under relaxed smoothness conditions.
problem Optimal algorithms for stochastic bilevel optimization under relaxed smoothness conditions.
method Introduces a novel fully single-loop and Hessian-inversion-free algorithmic framework for stochastic bilevel optimization.
result Demonstrates state-of-the-art oracle complexity results for multi-objective robust bilevel optimization.
Paper tackles bilevel optimization problems using penalty methods.
problem Unconstrained and constrained bilevel optimization problems with nonsmooth lower levels.
method Introduces first-order penalty methods and O ( ε − 4 log ε − 1 ) O(\varepsilon^{-4}\log\varepsilon^{-1}) O ( ε − 4 log ε − 1 ) and O ( ε − 7 log ε − 1 ) O(\varepsilon^{-7}\log\varepsilon^{-1}) O ( ε − 7 log ε − 1 ) operation complexities. result Establishes operation complexities for finding ε \varepsilon ε -KKT solutions. New methods solve complex optimization problems without strong convexity assumptions.
problem Complex bilevel optimization problems with minimax lower-level structures.
method Penalty-based first-order methods for bilevel minimax optimization.
result Achieves ε ε ε -KKT point with improved oracle complexity. BILBO optimizes bilevel problems without repeated lower-level optimizations.
problem Challenges in bilevel optimization, especially in noisy, constrained, and derivative-free settings.
method BILevel Bayesian Optimization (BILBO) that optimizes both levels simultaneously, using confidence-bounds and function query selection.
result Theoretical and empirical evidence of BILBO's effectiveness on various problems.
New approach to bilevel optimization for machine learning using functional methods.
problem Solving bilevel optimization problems in machine learning, especially with over-parameterized neural networks.
method Functional point of view, scalable and efficient algorithms for functional bilevel optimization.
result Demonstrates benefits of functional approach on instrumental regression and reinforcement learning tasks.
BOML unifies meta-learning methods into a common bilevel optimization framework.
problem Meta-learning methods with diverse modeling aspects.
method Modularized bilevel optimization library in Python.
result Unified solution for various meta-learning formulations.
This paper improves the convergence rates of bilevel optimization algorithms.
problem Improving the convergence rates of bilevel optimization algorithms.
method Provided lower complexity bounds and proposed an accelerated bilevel optimizer.
result AccBiO achieves optimal results under certain conditions.
A new method tackles bilevel optimization using Lanczos process for efficient hyper-gradient computation.
problem Efficiently solving large-scale bilevel optimization problems with gradient-based methods.
method Constructing low-dimensional approximate Krylov subspaces with the Lanczos process to approximate the Hessian inverse vector product.
result Demonstrates a O ( ε − 1 ) \mathcal{O}(ε^{-1}) O ( ε − 1 ) convergence rate and efficiency in synthetic and deep learning tasks. New methods solve complex optimization problems in machine learning.
problem Challenges in stochastic bilevel optimization with constraints and high variables.
method Inexact bilevel stochastic gradient methods for constrained and unconstrained lower-level problems.
result Comprehensive convergence theory for both unconstrained and constrained cases.
AdaSDBO solves decentralized bilevel optimization without problem parameters, achieving competitive performance.
problem Decentralized bilevel optimization problems without known parameters.
method AdaSDBO, a fully problem-parameter-free algorithm with adaptive stepsizes.
result AdaSDBO achieves a convergence rate of $\widetilde{\mathcal{O}}\left(\frac{1}{T}
ight)$ , matching state-of-the-art methods up to polylogarithmic factors.
Paper proposes AggITD for efficient federated hypergradient computation.
problem Computing hypergradient in federated settings is challenging due to distributed and nonlinear construction of global Hessian matrices.
method AggITD: a novel communication-efficient federated hypergradient estimator via aggregated iterative differentiation.
result AggITD achieves the same sample complexity as AID-based approaches but with fewer communication rounds, especially in heterogeneous data environments.
BADR framework optimizes fairness metrics efficiently.
problem Fairness-inefficient models in machine learning.
method Bilevel Adaptive Rescalarisation procedure.
result BADR framework recovers optimal Pareto-efficient models.
New method achieves optimal sample complexity without warm-start in bilevel optimization.
problem Optimizing smooth objective functions with fixed point constraints in meta-learning and equilibrium models.
method Fixed point iterations at lower-level and projected inexact gradient descent at upper-level.
result Achieves near optimal sample complexity O ( ε − 2 ) O(ε^{-2}) O ( ε − 2 ) and i l d e O ( ε − 1 ) ilde{O}(ε^{-1}) i l d e O ( ε − 1 ) samples. A new method solves bilevel optimization problems in competitive Markov games.
problem Capturing competitive structures in RL with multiple interacting policies.
method Penalty-augmented Nikaido-Isoda descent-ascent (PANDA) method.
result PANDA converges to stationary points without convexity assumptions.
A bilevel optimization method for reweighting data leads to suboptimal results.
problem Reweighting data for better model performance on a different distribution.
method Formalizes data reweighting as a bilevel optimization problem and analyzes the limitations of classical solvers.
result The final data weights are often very sparse, indicating suboptimal solutions.
Solving a bilevel optimization problem is at the core of several machine learning problems such as hyperparameter tuning, data denoising, meta- and few-shot learning, and training-data poisoning. Different from simultaneous or multi-objective optimization, the steepest descent direction for minimizing the upper-level c…