Improved bilevel model solves complex machine learning problems faster and more accurately.
problem Complex hierarchical machine learning problems with inner and outer problem correlation.
method Proposed an improved bilevel model with theoretical guarantees and empirical validation.
result Empirical results show our model outperforms current bilevel models significantly.
We introduce a framework based on bilevel programming that unifies gradient-based hyperparameter optimization and meta-learning. We show that an approximate version of the bilevel problem can be solved by taking into explicit account the optimization dynamics for the inner objective. Depending on the specific setting, …
This paper tackles Sinkhorn DRO by reformulating it as a bilevel program and proposes sampling-based algorithms.
problem Distributionally robust optimization with ambiguity sets defined via the Sinkhorn discrepancy.
method Primal perspective reformulation as a bilevel program, double-loop and single-loop sampling-based algorithms.
result Simultaneously obtain the optimal robust decision and the worst-case distribution.
CPP solves chance constrained optimization problems with a framework that combines samples and quantile lemma.
problem Chance constrained optimization problems with constraints on random variables.
method CPP framework using samples and quantile lemma to transform into deterministic problem.
result CPP provides a posteriori guarantees on constraint satisfaction and can handle different types of chance constraints.
In (Franceschi et al., 2018) we proposed a unified mathematical framework, grounded on bilevel programming, that encompasses gradient-based hyperparameter optimization and meta-learning. We formulated an approximate version of the problem where the inner objective is solved iteratively, and gave sufficient conditions e…
Efficiently solves heterogeneous QPs by reducing variables using instance-specific projections.
problem Solving high-dimensional quadratic programming problems efficiently.
method Data-driven framework with a graph neural network generating projections tailored to each QP instance.
result Produces high-quality solutions with reduced computation time, outperforming existing methods.
This paper proposes a method to reveal task relationships in multi-task learning models using sparse graphs.
problem Understanding the underlying task relationships in multi-task learning models.
method Proposes a bilevel formulation of multi-task learning that induces sparse graphs.
result The method improves interpretability of multi-task learning models without sacrificing generalization performance.
Optimal Volt/VAR control rules are designed using deep neural networks.
problem Designing optimal Volt/VAR control rules for distributed energy resources (DERs).
method Formulate optimal rule design as a bilevel program, then reformulate it as training a deep neural network (DNN). Use proximal gradient descent (PGD) iterations to emulate Volt/VAR dynamics.
result The proposed solution can be adapted to single/multi-phase feeders and achieves enhanced steady-state voltage profiles.
New method solves complex bilevel optimization problems.
problem Challenging bilevel optimization problems in machine learning.
method Penalty-based bilevel gradient descent (PBGD) algorithm.
result PBGD algorithm converges for constrained bilevel problems.
This thesis analyzes and improves convergence rates of bilevel optimization algorithms in machine learning.
problem Convergence analysis and algorithm design for bilevel optimization in machine learning.
method Comprehensive convergence rate analysis for both problem-based and algorithm-based bilevel optimization formulations.
result First lower bounds and matching upper bounds for bilevel optimization, and new stochastic algorithms with lower complexity.
New framework solves dynamic bilevel optimization problems in reinforcement learning.
problem Dynamic objective functions in reinforcement learning and human feedback.
method Principled penalty-based methods for bilevel reinforcement learning.
result Demonstrated effectiveness of penalty-based algorithms in simulations.
A new method solves sparse regularization problems efficiently and robustly.
problem Sparse regularization in machine learning problems.
method Iteratively reweighted least square (IRLS) with bilevel resolution and BFGS solver.
result Achieves top performances on various sparsity and regularization problems.
Framework solves bilevel optimization on manifolds.
problem Solving bilevel optimization problems with manifold constraints.
method Hypergradient estimation strategies on manifolds, convergence and complexity analyses.
result Efficacy demonstrated through various applications.
Semiparametric method removes bias in functional bilevel gradient estimation.
problem First-order bias in plug-in hypergradient when lower-level problem is nonparametric.
method Semiparametric debiasing theory based on efficient influence function leads to cross-fitted orthogonal hypergradient estimator.
result Asymptotic normality and uniform control over outer parameter established for the estimator.
A new algorithm tackles bilevel optimization with multiple inner minima.
problem Challenges in bilevel optimization with multiple inner minima.
method Reformulated as constrained optimization, solved via primal-dual bilevel optimization (PDBO) algorithm.
result First non-asymptotic convergence guarantee for bilevel optimization with multiple inner minima.
Develops a new method for efficient stochastic bilevel optimization.
problem Stochastic bilevel optimization problems in machine learning applications.
method Single-Timescale stochAstic BiLevEl optimization (STABLE) method.
result Achieves the same order of sample complexity as stochastic gradient descent for single-level optimization.
New Hessian-free method improves bilevel optimization for meta-learning.
problem Efficiently solving bilevel optimization problems with limited second-order information.
method Proposes a new Hessian-free method that approximates the response Jacobian matrix via optimization path differences.
result Demonstrates superior performance on meta-learning tasks compared to baseline methods.
New analysis improves understanding of bilevel optimization stability and generalization.
problem Understanding how well bilevel optimization algorithms generalize.
method Algorithmic stability arguments and generalization bounds for three bilevel minimax solvers.
result Precise trade-off between algorithmic stability, generalization gaps, and practical settings.
Optimizes bilevel empirical risk minimization with improved oracle calls.
problem Optimizing bilevel empirical risk minimization problems.
method Proposes a bilevel extension of the SARAH algorithm.
result Demonstrates improved oracle calls to achieve stationarity.
Unified framework for decentralized bilevel optimization with various heterogeneity-correction strategies.
problem Decentralized bilevel optimization with neighborhood communications and data heterogeneity.
method SPARKLE: Single-loop Primal-dual Algorithm for decentralized bilevel optimization, incorporating various heterogeneity-correction techniques.
result Unified convergence analysis for SPARKLE with state-of-the-art convergence rates compared to existing algorithms.
Develops an algorithm for bilevel optimization with coupled constraints.
problem Challenges in bilevel optimization with coupled constraints.
method Primal-dual-assisted penalty approach and a fully first-order algorithm (BLOCC).
result Established rigorous convergence theory and demonstrated effectiveness on real-world applications.
Paper tackles efficient SGD methods for constrained bilevel optimization.
problem Stochastic bilevel optimization with equality constraints.
method Alternating implicit projected SGD and its variants.
result Achieves sample complexity matching state-of-the-art for unconstrained problems.
A new method optimizes diffusion models for fine-tuning tasks efficiently.
problem Optimizing diffusion models for downstream tasks using nested bilevel structures.
method Formalizes the challenge as a generative bilevel optimization problem and introduces a first-order bilevel framework.
result Our method outperforms existing fine-tuning and hyperparameter search baselines.
The paper analyzes convergence rates of bilevel optimization algorithms and introduces a new stochastic algorithm.
problem Nonconvex-strongly-convex bilevel optimization problems in machine learning.
method Comprehensive convergence rate analysis for deterministic bilevel optimization using AID and ITD, and a novel stochastic algorithm stocBiO.
result Theoretical convergence rates for AID and ITD methods, and stocBiO's superior performance.
Study analyzes a new algorithm for complex optimization problems.
problem Stochastic bilevel optimisation problems in continuous-time models.
method Continuous-time, two-timescale stochastic approximation algorithm.
result Obtained weak convergence rate using central limit theorem.
New algorithms reduce bilevel optimization complexity to ε^(-1.5).
problem Efficiently solving bilevel optimization problems in machine learning.
method Proposed two new algorithms: one using momentum-based recursive iterations, the other using recursive gradient estimations.
result Achieved computational complexity of ε^(-1.5), significantly faster than previous methods.
Paper tackles Hessian/Jacobian-free stochastic bilevel optimization with O(ε−1.5) complexity.
problem Nonconvex-strongly-convex bilevel optimization problem.
method FdeHBO optimizer with finite-difference Hessian/Jacobian-vector approximation and momentum.
result FdeHBO achieves O(ε−1.5) iterations for ε-accurate stationary point. SmoothFBO tackles non-stationary functional bilevel optimization.
problem Current FBO methods are limited to static offline settings and perform poorly in online, non-stationary scenarios.
method SmoothFBO introduces a time-smoothed stochastic hypergradient estimator with a window parameter to handle non-stationarity.
result SmoothFBO achieves sublinear regret and outperforms existing methods in non-stationary hyperparameter optimization and model-based reinforcement learning.
New method circumvents non-convexity in bilevel RL via hyper-gradient.
problem Non-convexity in lower-level RL problems in bilevel reinforcement learning.
method Characterizing hyper-gradient via fully first-order information, circumventing convexity assumption.
result Developed model-based and model-free algorithms with convergence rate O(ε−1). This paper proposes a gossip-based algorithm for distributed bilevel optimization over networks.
problem Distributed bilevel optimization over networks with limited communication.
method Gossip-based distributed bilevel learning algorithm.
result Achieves optimal sample complexities for general and strongly convex objectives.
A new method tackles bilevel optimization using Lanczos process for efficient hyper-gradient computation.
problem Efficiently solving large-scale bilevel optimization problems with gradient-based methods.
method Constructing low-dimensional approximate Krylov subspaces with the Lanczos process to approximate the Hessian inverse vector product.
result Demonstrates a O(ε−1) convergence rate and efficiency in synthetic and deep learning tasks. Paper resolves ambiguity in non-convex bilevel optimization problems.
problem Ambiguity in bilevel optimization with non-convex lower-level objectives.
method Introduces selection maps to define critical points and resolves ambiguity.
result Validates new analytical tools in Morse theory for implicit differentiation.
Novel approach trains ASR models with less supervision using bilevel optimization.
problem Training acoustic models for ASR with minimal supervision.
method Bilevel optimization with unsupervised and supervised losses.
result Achieves superior performance compared to existing methods.
A model-based optimal experiment design (OED) of nonlinear systems is studied. OED represents a methodology for optimizing the geometry of the parametric joint-confidence regions (CRs), which are obtained in an a posteriori analysis of the least-squares parameter estimates. The optimal design is achieved by using the a…
New lower bounds for bilevel optimization with first-order oracles.
problem Complexity of bilevel optimization with first-order oracles.
method Development of hard instances and proof of lower bounds.
result Nontrivial lower bounds for first-order zero-respecting algorithms.
New algorithm tackles stochastic bilevel optimization under relaxed smoothness conditions.
problem Optimal algorithms for stochastic bilevel optimization under relaxed smoothness conditions.
method Introduces a novel fully single-loop and Hessian-inversion-free algorithmic framework for stochastic bilevel optimization.
result Demonstrates state-of-the-art oracle complexity results for multi-objective robust bilevel optimization.
Proposes a new method for predicting uncertain net electricity demand.
problem Uncertain net electricity demand and asymmetric cost structure.
method Bilevel program using clustering to tailor prescriptions.
result Substantial cost savings compared to standard methods.
New approach to bilevel optimization for machine learning using functional methods.
problem Solving bilevel optimization problems in machine learning, especially with over-parameterized neural networks.
method Functional point of view, scalable and efficient algorithms for functional bilevel optimization.
result Demonstrates benefits of functional approach on instrumental regression and reinforcement learning tasks.
New methods solve complex optimization problems in machine learning.
problem Challenges in stochastic bilevel optimization with constraints and high variables.
method Inexact bilevel stochastic gradient methods for constrained and unconstrained lower-level problems.
result Comprehensive convergence theory for both unconstrained and constrained cases.
New algorithm achieves linear speedup in non-i.i.d. federated bilevel learning.
problem Linear speedup in convergence for non-i.i.d. datasets in federated bilevel optimization.
method Proposes FedMBO with a novel client sampling scheme for non-i.i.d. datasets.
result Achieves a convergence rate of O(1/√(nK) + 1/K + √n/K³/²).
Novel method for bilevel optimization with convex lower-level problem.
problem Minimizing a smooth objective over the optimal solution set of a convex constrained problem.
method Local cutting plane approximation of lower-level solution set combined with conditional gradient updates.
result Achieves optimal iteration complexity for the considered class of bilevel problems.
Bilevel Continual Learning improves continual learning by transferring knowledge effectively.
problem Catastrophic forgetting and poor generalization in continual learning.
method Bilevel optimization and dual memory management strategies.
result BCL achieves effective knowledge transfer and alleviates catastrophic forgetting.
Paper proposes SMO for solving bilevel optimization problems efficiently.
problem Solving bilevel optimization problems with nonsmooth convex lower-level and nonconvex upper-level objectives.
method Sequential minimax optimization (SMO) method using modified augmented Lagrangian and penalty schemes.
result Improves operation complexity for finding ε-KKT solutions. New methods solve complex optimization problems without strong convexity assumptions.
problem Complex bilevel optimization problems with minimax lower-level structures.
method Penalty-based first-order methods for bilevel minimax optimization.
result Achieves ε-KKT point with improved oracle complexity. Paper tackles bilevel optimization problems using penalty methods.
problem Unconstrained and constrained bilevel optimization problems with nonsmooth lower levels.
method Introduces first-order penalty methods and O(ε−4logε−1) and O(ε−7logε−1) operation complexities. result Establishes operation complexities for finding ε-KKT solutions. Solving a bilevel optimization problem is at the core of several machine learning problems such as hyperparameter tuning, data denoising, meta- and few-shot learning, and training-data poisoning. Different from simultaneous or multi-objective optimization, the steepest descent direction for minimizing the upper-level c…
BOML unifies meta-learning methods into a common bilevel optimization framework.
problem Meta-learning methods with diverse modeling aspects.
method Modularized bilevel optimization library in Python.
result Unified solution for various meta-learning formulations.
AdaSDBO solves decentralized bilevel optimization without problem parameters, achieving competitive performance.
problem Decentralized bilevel optimization problems without known parameters.
method AdaSDBO, a fully problem-parameter-free algorithm with adaptive stepsizes.
result AdaSDBO achieves a convergence rate of $\widetilde{\mathcal{O}}\left(\frac{1}{T}
ight)$, matching state-of-the-art methods up to polylogarithmic factors.