Proposes glocal hypergradient estimation for hyperparameter optimization.
problem Combining reliability and efficiency in hyperparameter optimization.
method Uses Koopman operator theory to approximate global hypergradients from local ones.
result Achieves both reliability and efficiency in hyperparameter optimization.
Paper proposes AggITD for efficient federated hypergradient computation.
problem Computing hypergradient in federated settings is challenging due to distributed and nonlinear construction of global Hessian matrices.
method AggITD: a novel communication-efficient federated hypergradient estimator via aggregated iterative differentiation.
result AggITD achieves the same sample complexity as AID-based approaches but with fewer communication rounds, especially in heterogeneous data environments.
Study on stochastic hypergradient computation for machine learning problems.
problem Efficient computation of hypergradients in machine learning models.
method Stochastic approximation schemes for hypergradient computation, focusing on empirical risk minimization.
result Bounds for the mean square error of hypergradient approximation under contraction assumptions.
Study compares methods for computing hypergradients in machine learning problems.
problem Computing exact hypergradients in machine learning is difficult.
method Investigates reverse mode iterative differentiation and approximate implicit differentiation methods.
result Unified analysis provides iteration complexity bounds and hierarchy of methods.
NHGD solves bilevel optimization problems with reduced computational time.
problem Solving bilevel optimization problems with high computational cost.
method Exploits statistical structure of inner optimization to use empirical Fisher matrix as Hessian surrogate, enabling parallel optimization and approximation.
result NHGD achieves error bounds and sample complexity guarantees matching state-of-the-art methods, with significantly reduced computational time.
EvoGrad improves efficiency in meta-learning and hyperparameter optimization.
problem Efficiently compute hypergradients for larger network architectures.
method Uses evolutionary techniques to estimate hypergradients without second-order derivatives or longer computational graphs.
result Significant improvements in efficiency, enabling scaling to bigger architectures.
One-pass optimisation for high-dimensional hyperparameters.
problem Efficient optimisation of hyperparameters in machine learning models.
method Approximate hypergradient-based optimisation for any continuous hyperparameter, requiring only one training episode.
result Competitive performance on various datasets without hyperparameter restarts.
Semiparametric method removes bias in functional bilevel gradient estimation.
problem First-order bias in plug-in hypergradient when lower-level problem is nonparametric.
method Semiparametric debiasing theory based on efficient influence function leads to cross-fitted orthogonal hypergradient estimator.
result Asymptotic normality and uniform control over outer parameter established for the estimator.
We introduce a general method for improving the convergence rate of gradient-based optimizers that is easy to implement and works well in practice. We demonstrate the effectiveness of the method in a range of optimization problems by applying it to stochastic gradient descent, stochastic gradient descent with Nesterov …
ROBOT framework solves regression without correspondence for large data and complex models.
problem Regression without known correspondence in large datasets.
method ROBOT framework reformulates regression as a continuous optimization problem and uses hypergradient approach.
result ROBOT achieves better performance than existing methods in linear and nonlinear regression tasks.
We study the problem of fitting task-specific learning rate schedules from the perspective of hyperparameter optimization, aiming at good generalization. We describe the structure of the gradient of a validation error w.r.t. the learning rate schedule -- the hypergradient. Based on this, we introduce MARTHE, a novel on…
Framework solves bilevel optimization on manifolds.
problem Solving bilevel optimization problems with manifold constraints.
method Hypergradient estimation strategies on manifolds, convergence and complexity analyses.
result Efficacy demonstrated through various applications.
Working with any gradient-based machine learning algorithm involves the tedious task of tuning the optimizer's hyperparameters, such as its step size. Recent work has shown how the step size can itself be optimized alongside the model parameters by manually deriving expressions for "hypergradients" ahead of time. We sh…
First-order method solves stochastic bilevel optimization with linear constraints.
problem Stochastic bilevel optimization with linear constraints and noise.
method Developed a novel framework using gradient-based techniques and smoothed penalty functions.
result Achieved finite-time convergence guarantees for (δ,ε)-Goldstein stationary points. Standard gradient descent methods are susceptible to a range of issues that can impede training, such as high correlations and different scaling in parameter space.These difficulties can be addressed by second-order approaches that apply a pre-conditioning matrix to the gradient to improve convergence. Unfortunately, s…
New method finds stationary points in bilevel optimization problems.
problem Solving nonconvex-strongly-convex bilevel optimization problems.
method Restarted Accelerated HyperGradient Descent (RAHGD) method.
result Achieves best-known theoretical guarantees for finding stationary points in bilevel optimization.
New methods solve complex optimization problems in machine learning.
problem Challenges in stochastic bilevel optimization with constraints and high variables.
method Inexact bilevel stochastic gradient methods for constrained and unconstrained lower-level problems.
result Comprehensive convergence theory for both unconstrained and constrained cases.
OSGM uses online learning to adapt stepsize for faster convergence.
problem Improving convergence rates of first-order methods.
method OSGM combines online learning and feedback functions to adjust stepsize.
result OSGM achieves convergence rates asymptotically no worse than optimal.
La-MAML improves fast online continual learning with a look-ahead approach.
problem Fast online continual learning with limited model capacity.
method Optimisation-based meta-learning with look-ahead and episodic memory.
result Superior performance on visual classification benchmarks.
New Hessian-free method improves bilevel optimization for meta-learning.
problem Efficiently solving bilevel optimization problems with limited second-order information.
method Proposes a new Hessian-free method that approximates the response Jacobian matrix via optimization path differences.
result Demonstrates superior performance on meta-learning tasks compared to baseline methods.
This work speeds up hyperparameter selection for non-smooth convex models using implicit differentiation.
problem Optimizing hyperparameters of non-smooth convex models.
method Implicit differentiation of proximal gradient and coordinate descent methods.
result Implicit differentiation can speed up hyperparameter optimization, especially for non-smooth problems.
New algorithm achieves linear speedup in non-i.i.d. federated bilevel learning.
problem Linear speedup in convergence for non-i.i.d. datasets in federated bilevel optimization.
method Proposes FedMBO with a novel client sampling scheme for non-i.i.d. datasets.
result Achieves a convergence rate of O(1/√(nK) + 1/K + √n/K³/²).
CB-RL solves complex decision-making problems with contextual information and exogenous events.
problem Optimal policy in strategic decision-making problems that depend on environmental configuration and exogenous events.
method Contextual Bilevel Reinforcement Learning (CB-RL) with a stochastic Hyper Policy Gradient Descent (HPGD) algorithm.
result Demonstrated convergence and performance of the HPGD algorithm for reward shaping and tax design.
The paper analyzes convergence rates of bilevel optimization algorithms and introduces a new stochastic algorithm.
problem Nonconvex-strongly-convex bilevel optimization problems in machine learning.
method Comprehensive convergence rate analysis for deterministic bilevel optimization using AID and ITD, and a novel stochastic algorithm stocBiO.
result Theoretical convergence rates for AID and ITD methods, and stocBiO's superior performance.
A new method solves bilevel optimization problems without Hessian inversion.
problem Solving bilevel optimization problems in machine learning.
method Penalty method avoiding Hessian inversion.
result Asymptotically exact hypergradient and convergence under mild conditions.
Fewer data weight updates lead to faster convergence in machine learning models.
problem Improving robustness of machine learning models through data mixing.
method Analyzing convergence behavior of data mixing with a finite number of inner steps.
result The optimal number of inner steps scales with the budget and type of gradients used.
A new method solves bilevel optimization problems in competitive Markov games.
problem Capturing competitive structures in RL with multiple interacting policies.
method Penalty-augmented Nikaido-Isoda descent-ascent (PANDA) method.
result PANDA converges to stationary points without convexity assumptions.
SmoothFBO tackles non-stationary functional bilevel optimization.
problem Current FBO methods are limited to static offline settings and perform poorly in online, non-stationary scenarios.
method SmoothFBO introduces a time-smoothed stochastic hypergradient estimator with a window parameter to handle non-stationarity.
result SmoothFBO achieves sublinear regret and outperforms existing methods in non-stationary hyperparameter optimization and model-based reinforcement learning.
A new stochastic method tackles bi-level optimization problems in deep learning.
problem Bi-level optimization problems in deep learning, including hyperparameter optimization and meta learning.
method Turning a BLO problem into a stochastic optimization, using SGLD MCMC and a recurrent algorithm to compute MC-estimated hypergradient.
result Our method is more robust to suboptimal inner optimization and non-unique inner minima, leading to more reliable solutions.
New method optimizes fairness in predictive models for continuous sensitive attributes.
problem Enforcing full statistical independence on continuous sensitive attributes is too restrictive.
method Functional bilevel optimization (FBO) and ITD algorithms.
result Achieves lowest or near-lowest fairness-accuracy regret on synthetic and real datasets.
Paper tackles bilevel optimization problems using penalty methods.
problem Unconstrained and constrained bilevel optimization problems with nonsmooth lower levels.
method Introduces first-order penalty methods and O(ε−4logε−1) and O(ε−7logε−1) operation complexities. result Establishes operation complexities for finding ε-KKT solutions. New method tackles bilevel optimization with polyhedral constraints.
problem Challenges in bilevel optimization with active-set changes and expensive Hessian inversions.
method Logarithmic barrier smoothing and proxy-gradient algorithm for differentiable approximation.
result Stationarity rates of O(K−2/3) in deterministic setting and O(K−2/5) under stochastic noise. New algorithm tackles stochastic bilevel optimization under relaxed smoothness conditions.
problem Optimal algorithms for stochastic bilevel optimization under relaxed smoothness conditions.
method Introduces a novel fully single-loop and Hessian-inversion-free algorithmic framework for stochastic bilevel optimization.
result Demonstrates state-of-the-art oracle complexity results for multi-objective robust bilevel optimization.
This thesis analyzes and improves convergence rates of bilevel optimization algorithms in machine learning.
problem Convergence analysis and algorithm design for bilevel optimization in machine learning.
method Comprehensive convergence rate analysis for both problem-based and algorithm-based bilevel optimization formulations.
result First lower bounds and matching upper bounds for bilevel optimization, and new stochastic algorithms with lower complexity.
Quantum computing speeds up multi-period asset allocation.
problem High computational complexity in classic computing for multi-period asset allocation.
method Applied quantum computing to simulate multi-asset portfolio using historic data.
result Quantum computing offers significant advantages over classical computing in finance.
The paper analyzes the pricing of a new compute futures asset.
problem Uncertainty in AI adoption and pricing of compute capital.
method An asset-pricing framework for compute futures, including synthetic futures pricing.
result Preliminary evidence suggests a positive compute risk premium.
Quantum computing offers energy savings over classical computing.
problem Energy efficiency in computing services.
method Cournot competition model constrained by energy usage.
result Quantum computing firms can outperform classical counterparts in energy efficiency.
The paper introduces reservoir computing models for complex systems.
problem Modeling complex engineering systems using nonlinear autoregression.
method Introduces reservoir computing with output feedback as stationary and ergodic infinite-order nonlinear autoregressive models.
result Demonstrates versatility of classical and quantum reservoir computers in modeling synthetic and real data.
This work makes neural sequence models more efficient by controlling computation.
problem Fixed compute for all examples in neural networks.
method Conditional computation to adapt compute to example complexity.
result Conditional Computation Transformer (CCT) improves efficiency and performance.
Defines computable learning for binary classification over metric spaces.
problem Defines computable PAC learning for binary classification over computable metric spaces.
method Provides sufficient conditions for ERM learners to be computable and bounds the strong Weihrauch degree of an ERM learner.
result Gives a hypothesis class that does not admit any proper computable PAC learner with computable sample function.
Automatic computation speeds up crosscap number calculation for alternating knots.
problem Computing crosscap numbers for alternating knots efficiently.
method Introduced an automatic computation with complexity O(E3). result Crosscap numbers of alternating knots can be computed in O(E3) time. TKFT models computation via smooth vector fields, simulating functions in a single dynamical step.
problem Modeling computation in a single step.
method Established Topological Kleene Field Theory (TKFT) as a new model of computation.
result Any computable function can be simulated in a single go of a dynamical system.
Stochastic reservoir computing is shown to be a universal approximator.
problem Theoretical justification for using stochastic reservoirs in machine learning.
method Investigated stochastic reservoir computing using probabilities of reservoir states as readout.
result Stochastic reservoir computers are universal approximating classes.
Machine learning impacts computational math, offering new functions approximations.
problem Machine learning's black box nature hinders further progress in computational math.
method Analyzes machine learning's impact on computational math and vice versa.
result Integrating computational math with machine learning can enhance both fields.
Coded computation techniques provide robustness against straggling servers in distributed computing, with the following limitations: First, they increase decoding complexity. Second, they ignore computations carried out by straggling servers; and they are typically designed to recover the full gradient, and thus, canno…
Method for computing Khovanov homology of tangles.
problem Limited explicit computational studies of Khovanov homology for tangles.
method Arc reduction approach to compute Khovanov homology.
result Derived and computed Poincaré polynomials for simple and complex tangles.
With the rapid growth of the data volume and the fast increasing of the computational model complexity in the scenario of cloud computing, it becomes an important topic that how to handle users' requests by scheduling computational jobs and assigning the resources in data center. In order to have a better perception of…
This paper simplifies computing higher-order U-statistics efficiently.
problem The inefficiency of computing higher-order U-statistics in practice. method Decomposition, connection to Einstein summation, and treewidth-based complexity estimate.
result A new, more efficient algorithm to compute U-statistics.