Proposes efficient, modular method for implicit differentiation.
problem Implicit differentiation of optimization problems.
method Automatic implicit differentiation using autodiff and implicit function theorem.
result Automatic differentiation of optimization problems is made easier and more modular.
Improved MUSE boosts performance and reduces error in Bayesian inference.
problem Hierarchical Bayesian inference problems
method Implicit differentiation applied to MUSE algorithm
result Significant speedup and improved accuracy compared to Hamiltonian Monte Carlo
Efficient implicit differentiation for Lasso hyperparameter optimization.
problem Difficult hyperparameter optimization for Lasso-type models.
method Implicit differentiation algorithm tailored for Lasso-type problems, avoiding matrix inversion and solving linear systems.
result Outperforms standard methods in optimizing error on held-out data or Stein Unbiased Risk Estimator (SURE).
Abstract: Study of surface transitions and IDE inflections via contact geometry.
problem Understanding transitions on surfaces and implicit differential equations.
method Contact geometry and Legendrian properties of projections.
result List of unavoidable local phenomena on surfaces and IDE solutions.
For the implicit systems of first order ordinary differential equations on the plane there is presented the complete local classification of generic singularities of family of its phase curves up to smooth orbital equivalence. Besides the well known singularities of generic vector fields on the plane and the singularit…
A method to visualize multidimensional local subspaces using implicit differentiation.
problem Understanding the effect of multidimensional projection on local subspaces.
method Implicit function differentiation to analyze local subspaces shaped by multidimensional ellipses.
result Visualization of local subspaces provides insights into the global structure of data.
Improved set prediction model using multiset-equivariant operations and approximate implicit differentiation.
problem Existing set prediction models struggle with multisets and cannot represent certain functions.
method Introduced multiset-equivariance, improved DSPN with approximate implicit differentiation, and applied to CLEVR object property prediction.
result Significantly improved object property prediction on CLEVR dataset.
This work speeds up hyperparameter selection for non-smooth convex models using implicit differentiation.
problem Optimizing hyperparameters of non-smooth convex models.
method Implicit differentiation of proximal gradient and coordinate descent methods.
result Implicit differentiation can speed up hyperparameter optimization, especially for non-smooth problems.
A method for fair representation learning through bi-level optimization and implicit differentiation.
problem Ensuring fair predictors invariant across sub-groups.
method Bi-level optimization with inner-loop for invariant predictors, implicit path alignment for efficiency.
result Consistently better trade-off in prediction performance and fairness measurement.
SINDy-PI robustly identifies implicit dynamics from noisy data.
problem Accurately modeling nonlinear dynamics from noisy data.
method Parallel, implicit SINDy algorithm with multiple optimization algorithms and model selection.
result Significantly more noise robust than previous SINDy approaches.
Proposes a new reinforcement learning method to improve agent performance in control tasks.
problem Shortcomings of maximum likelihood estimation in model-based reinforcement learning.
method Directly optimizes expected returns using implicit differentiation of a Bellman optimality function.
result Empirical evidence shows improved performance in model misspecification regime.
Large batch training with DP-SGD reduces model performance due to implicit bias.
problem Large batch training with DP-SGD reduces model performance.
method The study analyzes the phenomenon of implicit bias in Noisy-SGD (DP-SGD without clipping) and its theoretical solutions for linear models.
result The implicit bias in large batch training with DP-SGD is amplified by additional noise, similar to SGD.
This paper is devoted to the characterization of differentially flat nonlinear systems in implicit representation, after elimination of the input variables, in the differential geometric framework of manifolds of jets of infinite order. We extend the notion of Lie-Bäcklund equivalence, introduced in Fliess et al. (1999…
Logic approach finds real singularities in differential equations.
problem Finding geometric singularities of implicit ODEs over the reals.
method Vessiot theory, parametric Gaussian elimination, heuristic simplification, real quantifier elimination.
result Effective computation of geometric singularities using logic methods.
New methods distill data for deep networks efficiently.
problem Reduction of training data cost and inconvenience.
method Generative teaching networks, gradient matching, Implicit Function Theorem.
result New methods are computationally more efficient and improve model performance.
Study efficient derivative computation for nondifferentiable maps in machine learning.
problem Efficiently compute derivatives of fixed-point of nondifferentiable contractions.
method Iterative Differentiation (ITD), Approximate Implicit Differentiation (AID), and New Stochastic Implicit Differentiation (NSID).
result Established convergence rates for ITD, AID, and NSID, matching or improving smooth setting rates.
Solves inverse problem for Maxwell equations using vector fields.
problem Inverse problem for Maxwell equations in vacuum.
method Abstract theory of implicit differential equations over pre-symplectic manifolds.
result Provides solution for Maxwell equations using vector fields.
A notion of implicit difference equation on a Lie groupoid is introduced and an algorithm for extracting the integrable part (backward or/and forward) is formulated. As an application, we prove that discrete Lagrangian dynamics on a Lie groupoid G may be described in terms of Lagrangian implicit difference equations …
A new Gaussian process regression method infers implicit manifold structure from data.
problem Scaling Gaussian process regression to high-dimensional data.
method Proposes a fully differentiable Gaussian process regression technique that infers implicit manifold structure from data.
result Improves predictive performance and calibration of standard Gaussian process regression in high-dimensional settings.
AuxiLearn combines auxiliary tasks into a single loss function.
problem Improving neural network performance on a main task using auxiliary tasks.
method Implicit differentiation to learn a network that combines auxiliary tasks into a single coherent objective function.
result AuxiLearn consistently outperforms competing methods in various tasks and domains.
The study analyzes implicit biases in neural networks using backward error analysis.
problem Analyzing implicit biases in multitask and continual learning settings.
method Backward error analysis to compute implicit training biases, deriving modified losses with three terms.
result The conflict term, measuring gradient alignment, is a new quantity in continual learning.
Efficiently integrates stiff ODEs with vectorized methods.
problem Stiff systems and sparse training data in ODEs.
method Implicit, vectorized time integration with adjoint method.
result Achieves speed ups of greater than 100x on modern GPUs.
We introduce a new algorithm for approximate inference that combines reparametrization, Markov chain Monte Carlo and variational methods. We construct a very flexible implicit variational distribution synthesized by an arbitrary Markov chain Monte Carlo operation and a deterministic transformation that can be optimized…
We propose an algorithm for inexpensive gradient-based hyperparameter optimization that combines the implicit function theorem (IFT) with efficient inverse Hessian approximations. We present results about the relationship between the IFT and differentiating through optimization, motivating our algorithm. We use the pro…
Deep equilibrium models converge globally without explicit computation.
problem Global convergence of deep learning models with implicit layers.
method Analysis of gradient dynamics and proof of convergence rate.
result Deep equilibrium models converge to global optimum at a linear rate.
Recently, there has been a growing interest in the problem of learning rich implicit models - those from which we can sample, but can not evaluate their density. These models apply some parametric function, such as a deep network, to a base measure, and are learned end-to-end using stochastic optimization. One strategy…
Noise in RNNs promotes flatter minima and more stable dynamics.
problem Understanding and optimizing the training of RNNs with noise.
method Formalizing RNNs as stochastic differential equations and analyzing the effect of noise in the hidden states.
result Noise injection in RNNs leads to flatter minima, more stable dynamics, and improved robustness.
A core capability of intelligent systems is the ability to quickly learn new tasks by drawing on prior experience. Gradient (or optimization) based meta-learning has recently emerged as an effective approach for few-shot learning. In this formulation, meta-parameters are learned in the outer loop, while task-specific m…
iDAD uses neural networks to quickly adapt experiments without likelihoods.
problem Performing adaptive experiments in real-time with implicit models.
method iDAD learns a design policy network upfront to make quick design decisions.
result iDAD can make design decisions in milliseconds, unlike traditional BOED approaches.
We study first-order optimization methods obtained by discretizing ordinary differential equations (ODEs) corresponding to Nesterov's accelerated gradient methods (NAGs) and Polyak's heavy-ball method. We consider three discretization schemes: an explicit Euler scheme, an implicit Euler scheme, and a symplectic scheme.…
We prove an implicit function theorem for functions on infinite-dimensional Banach manifolds, invariant under the (local) action of a finite dimensional Lie group. Motivated by some geometric variational problems, we consider group actions that are not necessarily differentiable everywhere, but only on some dense subse…
PANIS learns PDE surrogates for heterogeneous materials without solving the PDE.
problem Learning surrogates for parametrized PDEs in heterogeneous media.
method Physics-aware neural implicit solvers combining probabilistic learning and physics-informed discretization.
result Learned surrogates for effective solutions in heterogeneous materials without solving the reference problem.
Novel approach finds implicit regularisation in two-player games using BEA.
problem Understanding implicit regularisation in two-player games.
method Using backward error analysis to construct continuous-time flows with gradient-eligible vector fields.
result Identifies new implicit regularisation effects in two-player games.
Study shows how SGD's implicit regularization relates to ridge regression.
problem Least squares regression optimization with mini-batch SGD.
method Analyzes stochastic gradient flow as a continuous-time model of SGD.
result Bound on excess risk of SGD flow over ridge regression, revealing how parameters drive risk.
Paper presents an ADMM-based approach to efficiently integrate quadratic programming layers into neural networks.
problem Integrating quadratic programs into neural networks for optimization.
method An ADMM-based network layer architecture for solving quadratic programs efficiently.
result The ADMM layer is approximately an order of magnitude faster than existing methods for medium scaled problems.
We study the problem to provide a triangular form based on implicit differential equations for non-linear multi-input systems with respect to the flatness property. Furthermore, we suggest a constructive method for the transformation of a given system into that special triangular shape, if possible. The well known Brun…
Adam's hyperparameters implicitly regularize solutions, penalizing or impeding loss gradients' norms.
problem Implicit regularization in Adam's hyperparameters and training stage.
method Backward error analysis and ODE approximations to study Adam's behavior.
result Adam's implicit regularization depends on hyperparameters and training stage, involving different norms.
We propose a novel approach to approximate complex high-dimensional posteriors using neural samplers.
problem Accurately capturing complex, multimodal, and correlated posteriors in high-dimensional spaces.
method Implicit variational inference with neural samplers and local linearisation bounds.
result Our method enables implicit distributions over tens of millions of latent variables, recovering correlations across layers in large Bayesian neural networks.
Study compares methods for computing hypergradients in machine learning problems.
problem Computing exact hypergradients in machine learning is difficult.
method Investigates reverse mode iterative differentiation and approximate implicit differentiation methods.
result Unified analysis provides iteration complexity bounds and hierarchy of methods.
The paper details local forms of morphisms in colored supermanifolds.
problem Understanding local forms of morphisms in colored supermanifolds.
method Detailed account of Z2n-differential calculus and local theorems. result Detailed insights into local forms of morphisms in colored supermanifolds.
GNIs induce asymmetric heavy-tailed noise in SGD, affecting network performance.
problem The effect of Gaussian noise injections on SGD dynamics and network performance.
method Developed a Langevin-like SDE driven by asymmetric heavy-tailed noise to model the modified SGD dynamics.
result GNIs induce an implicit bias that varies with noise heaviness and asymmetry, affecting network performance.
We describe a set of Gaussian Process based approaches that can be used to solve non-linear Ordinary Differential Equations. We suggest an explicit probabilistic solver and two implicit methods, one analogous to Picard iteration and the other to gradient matching. All methods have greater accuracy than previously sugge…
This paper deals with the efficient numerical solution of the two-dimensional partial integro-differential complementarity problem (PIDCP) that holds for the value of American-style options under the two-asset Merton jump-diffusion model. We consider the adaptation of various operator splitting schemes of both the impl…
Paper resolves ambiguity in non-convex bilevel optimization problems.
problem Ambiguity in bilevel optimization with non-convex lower-level objectives.
method Introduces selection maps to define critical points and resolves ambiguity.
result Validates new analytical tools in Morse theory for implicit differentiation.
The paper analyzes the implicit bias of SGD near loss manifold and provides new insights.
problem Understanding the implicit bias of SGD near loss manifolds in overparametrized models.
method Adapting ideas from Katzenberger (1991) to analyze SGD dynamics using a stochastic differential equation (SDE).
result SGD with label noise locally decreases the sharpness of loss, leading to a global analysis of implicit bias.
By providing a simple and efficient way of computing low-variance gradients of continuous random variables, the reparameterization trick has become the technique of choice for training a variety of latent variable models. However, it is not applicable to a number of important continuous distributions. We introduce an a…
Gradients of neural networks can be computed efficiently for any architecture, but some applications require differential operators with higher time complexity. We describe a family of restricted neural network architectures that allow efficient computation of a family of differential operators involving dimension-wise…
MDEQ models learn multi-resolution features efficiently.
problem Large-scale, hierarchical pattern recognition.
method Implicit differentiation, multiscale deep equilibrium model.
result MDEQs achieve performance on par with recent models.