New algorithm improves understanding of decentralized SBO transient iteration complexity.
problem Limited understanding of how network topology, data heterogeneity, and nested structures affect SBO.
method D-SOBA framework with two variants: D-SOBA-SO and D-SOBA-FO, providing non-asymptotic convergence analysis and transient iteration complexity.
result First theoretical understanding of how network topology, data heterogeneity, and nested structures influence decentralized SBO.
Scalable tools for nested optimization in deep learning.
problem Solving nested optimization problems on a large scale in deep learning.
method Building scalable tools for bilevel optimization.
result Tools for nested optimization scale to deep learning setups.
Optimizes bilevel empirical risk minimization with improved oracle calls.
problem Optimizing bilevel empirical risk minimization problems.
method Proposes a bilevel extension of the SARAH algorithm.
result Demonstrates improved oracle calls to achieve stationarity.
A new method optimizes diffusion models for fine-tuning tasks efficiently.
problem Optimizing diffusion models for downstream tasks using nested bilevel structures.
method Formalizes the challenge as a generative bilevel optimization problem and introduces a first-order bilevel framework.
result Our method outperforms existing fine-tuning and hyperparameter search baselines.
Unified framework for decentralized bilevel optimization with various heterogeneity-correction strategies.
problem Decentralized bilevel optimization with neighborhood communications and data heterogeneity.
method SPARKLE: Single-loop Primal-dual Algorithm for decentralized bilevel optimization, incorporating various heterogeneity-correction techniques.
result Unified convergence analysis for SPARKLE with state-of-the-art convergence rates compared to existing algorithms.
Paper tackles efficient SGD methods for constrained bilevel optimization.
problem Stochastic bilevel optimization with equality constraints.
method Alternating implicit projected SGD and its variants.
result Achieves sample complexity matching state-of-the-art for unconstrained problems.
New Hessian-free method improves bilevel optimization for meta-learning.
problem Efficiently solving bilevel optimization problems with limited second-order information.
method Proposes a new Hessian-free method that approximates the response Jacobian matrix via optimization path differences.
result Demonstrates superior performance on meta-learning tasks compared to baseline methods.
New algorithms reduce bilevel optimization complexity to ε^(-1.5).
problem Efficiently solving bilevel optimization problems in machine learning.
method Proposed two new algorithms: one using momentum-based recursive iterations, the other using recursive gradient estimations.
result Achieved computational complexity of ε^(-1.5), significantly faster than previous methods.
Unified SGD method improves convergence for nested optimization problems.
problem Stochastic nested optimization problems.
method ALTERNATE dESCEN (ALSET) method leveraging hidden smoothness.
result Requires O(ε−2) samples to achieve an ε-stationary point. This paper proposes a gossip-based algorithm for distributed bilevel optimization over networks.
problem Distributed bilevel optimization over networks with limited communication.
method Gossip-based distributed bilevel learning algorithm.
result Achieves optimal sample complexities for general and strongly convex objectives.
Paper resolves ambiguity in non-convex bilevel optimization problems.
problem Ambiguity in bilevel optimization with non-convex lower-level objectives.
method Introduces selection maps to define critical points and resolves ambiguity.
result Validates new analytical tools in Morse theory for implicit differentiation.
SimFBO simplifies FBO, making it more efficient and flexible.
problem Complex nested optimization in machine learning and edge computing.
method Proposes SimFBO, a simple and flexible FBO framework with improved communication efficiency.
result SimFBO and ShroFBO achieve linear convergence speedup and improved sample and communication complexities.
Bilevel optimization has been recently revisited for designing and analyzing algorithms in hyperparameter tuning and meta learning tasks. However, due to its nested structure, evaluating exact gradients for high-dimensional problems is computationally challenging. One heuristic to circumvent this difficulty is to use t…
Efficiently optimizes hyperparameters for PDE and inverse problems using Gaussian processes.
problem Hyperparameter optimization for scientific computing and inference methods.
method Bilevel optimization with Gauss-Newton linearization for efficient hyperparameter updates.
result Significant improvements in accuracy and robustness compared to random initialization.
New method tackles inexact bilevel optimization for faster parameter learning.
problem Nested optimization problems in bilevel learning with computationally difficult exact solutions.
method Inexact derivative-free optimization algorithms for approximate lower-level solutions.
result Global convergence and worst-case complexity for the proposed approach.
New framework solves dynamic bilevel optimization problems in reinforcement learning.
problem Dynamic objective functions in reinforcement learning and human feedback.
method Principled penalty-based methods for bilevel reinforcement learning.
result Demonstrated effectiveness of penalty-based algorithms in simulations.
A new method solves bilevel optimization problems in competitive Markov games.
problem Capturing competitive structures in RL with multiple interacting policies.
method Penalty-augmented Nikaido-Isoda descent-ascent (PANDA) method.
result PANDA converges to stationary points without convexity assumptions.
New analysis improves understanding of bilevel optimization stability and generalization.
problem Understanding how well bilevel optimization algorithms generalize.
method Algorithmic stability arguments and generalization bounds for three bilevel minimax solvers.
result Precise trade-off between algorithmic stability, generalization gaps, and practical settings.
BILBO optimizes bilevel problems without repeated lower-level optimizations.
problem Challenges in bilevel optimization, especially in noisy, constrained, and derivative-free settings.
method BILevel Bayesian Optimization (BILBO) that optimizes both levels simultaneously, using confidence-bounds and function query selection.
result Theoretical and empirical evidence of BILBO's effectiveness on various problems.
Improved bilevel model solves complex machine learning problems faster and more accurately.
problem Complex hierarchical machine learning problems with inner and outer problem correlation.
method Proposed an improved bilevel model with theoretical guarantees and empirical validation.
result Empirical results show our model outperforms current bilevel models significantly.
Paper tackles Hessian/Jacobian-free stochastic bilevel optimization with O(ε−1.5) complexity.
problem Nonconvex-strongly-convex bilevel optimization problem.
method FdeHBO optimizer with finite-difference Hessian/Jacobian-vector approximation and momentum.
result FdeHBO achieves O(ε−1.5) iterations for ε-accurate stationary point. New method circumvents non-convexity in bilevel RL via hyper-gradient.
problem Non-convexity in lower-level RL problems in bilevel reinforcement learning.
method Characterizing hyper-gradient via fully first-order information, circumventing convexity assumption.
result Developed model-based and model-free algorithms with convergence rate O(ε−1). A new method tackles bilevel optimization using Lanczos process for efficient hyper-gradient computation.
problem Efficiently solving large-scale bilevel optimization problems with gradient-based methods.
method Constructing low-dimensional approximate Krylov subspaces with the Lanczos process to approximate the Hessian inverse vector product.
result Demonstrates a O(ε−1) convergence rate and efficiency in synthetic and deep learning tasks. New lower bounds for bilevel optimization with first-order oracles.
problem Complexity of bilevel optimization with first-order oracles.
method Development of hard instances and proof of lower bounds.
result Nontrivial lower bounds for first-order zero-respecting algorithms.
AdaSDBO solves decentralized bilevel optimization without problem parameters, achieving competitive performance.
problem Decentralized bilevel optimization problems without known parameters.
method AdaSDBO, a fully problem-parameter-free algorithm with adaptive stepsizes.
result AdaSDBO achieves a convergence rate of $\widetilde{\mathcal{O}}\left(\frac{1}{T}
ight)$, matching state-of-the-art methods up to polylogarithmic factors.
New methods solve complex optimization problems without strong convexity assumptions.
problem Complex bilevel optimization problems with minimax lower-level structures.
method Penalty-based first-order methods for bilevel minimax optimization.
result Achieves ε-KKT point with improved oracle complexity. Paper tackles bilevel optimization problems using penalty methods.
problem Unconstrained and constrained bilevel optimization problems with nonsmooth lower levels.
method Introduces first-order penalty methods and O(ε−4logε−1) and O(ε−7logε−1) operation complexities. result Establishes operation complexities for finding ε-KKT solutions. New method solves complex bilevel optimization problems.
problem Challenging bilevel optimization problems in machine learning.
method Penalty-based bilevel gradient descent (PBGD) algorithm.
result PBGD algorithm converges for constrained bilevel problems.
Paper tackles low sample and communication complexities in decentralized bilevel optimization.
problem Decentralized bilevel optimization problems with limited computation and communication capabilities.
method Proposes INTERACT and SVR-INTERACT algorithms to achieve low sample and communication complexities.
result Achieves both low sample and communication complexities for solving decentralized bilevel optimization problems.
This thesis analyzes and improves convergence rates of bilevel optimization algorithms in machine learning.
problem Convergence analysis and algorithm design for bilevel optimization in machine learning.
method Comprehensive convergence rate analysis for both problem-based and algorithm-based bilevel optimization formulations.
result First lower bounds and matching upper bounds for bilevel optimization, and new stochastic algorithms with lower complexity.
This paper proposes a method to reveal task relationships in multi-task learning models using sparse graphs.
problem Understanding the underlying task relationships in multi-task learning models.
method Proposes a bilevel formulation of multi-task learning that induces sparse graphs.
result The method improves interpretability of multi-task learning models without sacrificing generalization performance.
Framework solves bilevel optimization on manifolds.
problem Solving bilevel optimization problems with manifold constraints.
method Hypergradient estimation strategies on manifolds, convergence and complexity analyses.
result Efficacy demonstrated through various applications.
Semiparametric method removes bias in functional bilevel gradient estimation.
problem First-order bias in plug-in hypergradient when lower-level problem is nonparametric.
method Semiparametric debiasing theory based on efficient influence function leads to cross-fitted orthogonal hypergradient estimator.
result Asymptotic normality and uniform control over outer parameter established for the estimator.
A new algorithm tackles bilevel optimization with multiple inner minima.
problem Challenges in bilevel optimization with multiple inner minima.
method Reformulated as constrained optimization, solved via primal-dual bilevel optimization (PDBO) algorithm.
result First non-asymptotic convergence guarantee for bilevel optimization with multiple inner minima.
Develops a new method for efficient stochastic bilevel optimization.
problem Stochastic bilevel optimization problems in machine learning applications.
method Single-Timescale stochAstic BiLevEl optimization (STABLE) method.
result Achieves the same order of sample complexity as stochastic gradient descent for single-level optimization.
NHGD solves bilevel optimization problems with reduced computational time.
problem Solving bilevel optimization problems with high computational cost.
method Exploits statistical structure of inner optimization to use empirical Fisher matrix as Hessian surrogate, enabling parallel optimization and approximation.
result NHGD achieves error bounds and sample complexity guarantees matching state-of-the-art methods, with significantly reduced computational time.
New discrete cobordism category for nested manifolds and relations to algebraic structures.
problem Discrete cobordism category for nested manifolds.
method Stratified Morse theory, Cyl-objects, doubling construction, cylindrical bar construction.
result Relations between Cyl-objects and algebraic structures like Temperley-Lieb algebras.
Develops an algorithm for bilevel optimization with coupled constraints.
problem Challenges in bilevel optimization with coupled constraints.
method Primal-dual-assisted penalty approach and a fully first-order algorithm (BLOCC).
result Established rigorous convergence theory and demonstrated effectiveness on real-world applications.
Improved hypernetwork for efficient neural network hyperparameter tuning.
problem Efficiently optimizing hyperparameters in neural networks.
method Proposed Δ-STN architecture focusing on accurate best-response Jacobian approximation. result Significantly improved hyperparameter tuning accuracy and stability.
New algorithms eliminate stepsize tuning for bilevel optimization problems.
problem Bilevel optimization problems with unknown parameters and stepsizes.
method D-TFBO and S-TFBO algorithms with adaptive stepsizes.
result Achieve performance comparable to well-tuned approaches with theoretical guarantees.
The paper analyzes convergence rates of bilevel optimization algorithms and introduces a new stochastic algorithm.
problem Nonconvex-strongly-convex bilevel optimization problems in machine learning.
method Comprehensive convergence rate analysis for deterministic bilevel optimization using AID and ITD, and a novel stochastic algorithm stocBiO.
result Theoretical convergence rates for AID and ITD methods, and stocBiO's superior performance.
Study analyzes a new algorithm for complex optimization problems.
problem Stochastic bilevel optimisation problems in continuous-time models.
method Continuous-time, two-timescale stochastic approximation algorithm.
result Obtained weak convergence rate using central limit theorem.
SmoothFBO tackles non-stationary functional bilevel optimization.
problem Current FBO methods are limited to static offline settings and perform poorly in online, non-stationary scenarios.
method SmoothFBO introduces a time-smoothed stochastic hypergradient estimator with a window parameter to handle non-stationarity.
result SmoothFBO achieves sublinear regret and outperforms existing methods in non-stationary hyperparameter optimization and model-based reinforcement learning.
Novel approach trains ASR models with less supervision using bilevel optimization.
problem Training acoustic models for ASR with minimal supervision.
method Bilevel optimization with unsupervised and supervised losses.
result Achieves superior performance compared to existing methods.
New algorithm tackles stochastic bilevel optimization under relaxed smoothness conditions.
problem Optimal algorithms for stochastic bilevel optimization under relaxed smoothness conditions.
method Introduces a novel fully single-loop and Hessian-inversion-free algorithmic framework for stochastic bilevel optimization.
result Demonstrates state-of-the-art oracle complexity results for multi-objective robust bilevel optimization.
New approach to bilevel optimization for machine learning using functional methods.
problem Solving bilevel optimization problems in machine learning, especially with over-parameterized neural networks.
method Functional point of view, scalable and efficient algorithms for functional bilevel optimization.
result Demonstrates benefits of functional approach on instrumental regression and reinforcement learning tasks.
New methods solve complex optimization problems in machine learning.
problem Challenges in stochastic bilevel optimization with constraints and high variables.
method Inexact bilevel stochastic gradient methods for constrained and unconstrained lower-level problems.
result Comprehensive convergence theory for both unconstrained and constrained cases.
Paper integrates predictive and prescriptive tasks using bilevel optimization.
problem Combining prediction and decision-making in machine learning.
method Bilevel optimization for integrating predictive and prescriptive tasks.
result Improved decision-making through direct integration of predictions.