Global implicit function theorem for Fréchet spaces, solving derivative loss problems.
problem Solving initial value problems with derivative loss in Fréchet spaces.
method Global implicit function theorems for Keller's Cc1-mappings in Fréchet spaces, applied through submersions and transversality. result Global existence and uniqueness of solutions to initial value problems with derivative loss.
We prove implicit function theorems for mappings on topological vector spaces over valued fields. In the real and complex cases, we obtain implicit function theorems for mappings from arbitrary (not necessarily locally convex) topological vector spaces to Banach spaces.
In this paper, we revisit implicit regularization from the ground up using notions from dynamical systems and invariant subspaces of Morse functions. The key contributions are a new criterion for implicit regularization---a leading contender to explain the generalization power of deep models such as neural networks---a…
Proposes efficient, modular method for implicit differentiation.
problem Implicit differentiation of optimization problems.
method Automatic implicit differentiation using autodiff and implicit function theorem.
result Automatic differentiation of optimization problems is made easier and more modular.
New method for high-fidelity shape representations from raw data.
problem Creating accurate shape representations from raw data.
method A simple loss function encouraging neural network to vanish on input point cloud and have unit norm gradient.
result Our method produces high-fidelity, smooth, and natural zero level set surfaces.
In this paper, we describe the "implicit autoencoder" (IAE), a generative autoencoder in which both the generative path and the recognition path are parametrized by implicit distributions. We use two generative adversarial networks to define the reconstruction and the regularization cost functions of the implicit autoe…
SINDy-PI robustly identifies implicit dynamics from noisy data.
problem Accurately modeling nonlinear dynamics from noisy data.
method Parallel, implicit SINDy algorithm with multiple optimization algorithms and model selection.
result Significantly more noise robust than previous SINDy approaches.
This paper shows how to train only the implicit layer of overparameterized implicit neural networks.
problem Understanding how the implicit layer contributes to the training of overparameterized implicit neural networks.
method Restricting training to only the implicit layer and analyzing the generalization error for ReLU-activated networks.
result Global convergence is guaranteed even if only the implicit layer is trained, and gradient flow with proper random initialization can achieve small generalization errors.
Proposes a parametric modal regression method using the implicit function theorem.
problem Finding conditional modes for multi-modal conditional distributions.
method Uses the implicit function theorem to develop an objective function for learning a joint function over inputs and targets.
result Empirically demonstrates scalability and effectiveness in learning multi-valued functions and high-dimensional inputs.
This paper analyzes implicit bias in Deep Linear Discriminant Analysis.
problem The implicit bias of Deep Linear Discriminant Analysis.
method Analyzing gradient flow on a L-layer diagonal linear network.
result Under balanced initialization, the network transforms additive updates into multiplicative updates, conserving the (2/L) quasi-norm.
Gradient matching method estimates implicit regularization in complex deep learning systems.
problem Estimating implicit regularization in modern deep learning systems with complex modifications.
method Gradient matching methods to empirically estimate implicit regularization.
result Empirical estimation of implicit regularization in arbitrary networks, including dropout.
Neural networks learn the support of the target function through SGD's implicit regularization effect.
problem Learning the support of the target function in neural networks.
method Investigation of mini-batch SGD's ability to learn the support in the first layer of a neural network.
result Mini-batch SGD effectively learns the support in the first layer by shrinking irrelevant weights, while vanilla GD requires an explicit regularization term.
Implicit probabilistic models are models defined naturally in terms of a sampling procedure and often induces a likelihood function that cannot be expressed explicitly. We develop a simple method for estimating parameters in implicit models that does not require knowledge of the form of the likelihood function or any d…
Gradient descent on ReLU networks with square loss implicitly favors balanced weights.
problem Understanding implicit regularization in nonlinear neural networks with regression losses.
method Analyzing gradient descent dynamics on ReLU networks with square loss.
result It is impossible to characterize the implicit regularization of ReLU networks with square loss by any explicit function of model parameters.
A method to visualize multidimensional local subspaces using implicit differentiation.
problem Understanding the effect of multidimensional projection on local subspaces.
method Implicit function differentiation to analyze local subspaces shaped by multidimensional ellipses.
result Visualization of local subspaces provides insights into the global structure of data.
Proposes a new reinforcement learning method to improve agent performance in control tasks.
problem Shortcomings of maximum likelihood estimation in model-based reinforcement learning.
method Directly optimizes expected returns using implicit differentiation of a Bellman optimality function.
result Empirical evidence shows improved performance in model misspecification regime.
Gradient descent biases towards stable rank networks for nearly-orthogonal data.
problem Understanding implicit bias in non-smooth neural networks trained by gradient descent.
method Analysis of two-layer ReLU and leaky ReLU networks trained by gradient descent on nearly-orthogonal data.
result Gradient descent biases towards networks with stable rank and uniform margin for nearly-orthogonal data.
Improved set prediction model using multiset-equivariant operations and approximate implicit differentiation.
problem Existing set prediction models struggle with multisets and cannot represent certain functions.
method Introduced multiset-equivariance, improved DSPN with approximate implicit differentiation, and applied to CLEVR object property prediction.
result Significantly improved object property prediction on CLEVR dataset.
In this paper, we make a generalization of Routh's reduction method for Lagrangian systems with symmetry to the case where not any regularity condition is imposed on the Lagrangian. First, we show how implicit Lagrange-Routh equations can be obtained from the Hamilton-Pontryagin principle, by making use of an anholonom…
Deep equilibrium models converge globally without explicit computation.
problem Global convergence of deep learning models with implicit layers.
method Analysis of gradient dynamics and proof of convergence rate.
result Deep equilibrium models converge to global optimum at a linear rate.
We prove an implicit function theorem for functions on infinite-dimensional Banach manifolds, invariant under the (local) action of a finite dimensional Lie group. Motivated by some geometric variational problems, we consider group actions that are not necessarily differentiable everywhere, but only on some dense subse…
The paper studies geometric representations of submanifolds using complex-valued functions.
problem Exploring the geometry of codimension-2 submanifolds.
method Implicitly representing submanifolds by complex-valued functions and showing a prequantum bundle structure.
result The space of implicit representations admits a prequantum bundle structure over the space of submanifolds.
We introduce the implicit processes (IPs), a stochastic process that places implicitly defined multivariate distributions over any finite collections of random variables. IPs are therefore highly flexible implicit priors over functions, with examples including data simulators, Bayesian neural networks and non-linear tr…
A new method for optimizing non-decomposable metrics with constraints.
problem Optimizing complex machine learning objectives with thresholded constraints.
method Formulate rate-constrained optimization using the Implicit Function theorem and solve with gradient-based methods.
result Demonstrated effectiveness over existing methods on benchmark datasets.
This paper proves SGD converges to global minimum for over-parameterized ReLU networks.
problem Theoretical understanding of implicit neural networks is limited.
method Gradient flow analysis of ReLU activated implicit neural networks.
result Randomly initialized gradient descent converges to global minimum at a linear rate for square loss function in over-parameterized ReLU networks.
Large learning rates lead to various implicit biases in nonconvex optimization.
problem Understanding the conditions under which large learning rates yield edge of stability, balancing, and catapult phenomena.
method Developed a global convergence theory for nonconvex functions without globally Lipschitz continuous gradient, focusing on functions with good regularity.
result These implicit biases are more likely to occur in functions with good regularity, and large learning rates favor flatter regions.
IDAC improves reinforcement learning efficiency by modeling implicit distributions.
problem Improving sample efficiency in reinforcement learning algorithms.
method IDAC uses two DGNs for a distributional critic and a semi-implicit actor to model implicit policy distributions.
result IDAC outperforms state-of-the-art algorithms on OpenAI Gym environments.
Stochastic recurrent neural networks with latent random variables of complex dependency structures have shown to be more successful in modeling sequential data than deterministic deep models. However, the majority of existing methods have limited expressive power due to the Gaussian assumption of latent variables. In t…
The paper studies implicit regularization in over-parameterized models for high-dimensional data.
problem Understanding implicit regularization in over-parameterized models for high-dimensional data.
method The paper designs regularization-free algorithms for the high-dimensional single index model and provides theoretical guarantees for the induced implicit regularization phenomenon.
result The proposed methods achieve minimax optimal statistical rates of convergence and outperform classical methods with explicit regularization.
DVIP improves on IP-based methods by using IPs as priors over latent functions.
problem Limited expressiveness of IP-based models, especially in function space.
method Proposes DVIP, a multi-layer generalization of IPs, and scalable variational inference.
result DVIP outperforms previous IP-based methods and deep GPs in regression and classification tasks.
Gradient descent training of neural networks leads to solutions close to natural cubic splines.
problem Understanding the implicit bias of gradient descent in neural networks.
method Analysis of gradient descent training for wide neural networks, focusing on the curvature penalty and initialization schemes.
result The solutions of gradient descent training are polyharmonic splines for certain initialization schemes.
Mirror flow in shallow neural networks shows similar implicit bias to gradient flow, with key differences in curvature penalties.
problem Analyzing implicit bias in shallow neural networks with mirror flow.
method Characterization through variational problems and scaled potentials.
result Mirror flow with scaled potentials induces a rich class of biases not captured by RKHS norms.
SGD with large learning rates can achieve better test accuracy than expected.
problem SGD with large learning rates often outperforms expected convergence bounds.
method Proved that SGD with small learning rates stays close to gradient flow path on modified loss.
result Explicitly adding an implicit regularizer to the loss improves test accuracy.
The paper improves convergence for linear systems using entropic mirror descent with Polyak stepsizes.
problem Convergence analysis for linear systems with unbounded domain.
method Entropic mirror descent with Polyak stepsizes, sublinear and linear convergence results.
result Generalized convergence result for arbitrary convex functions.
We study the generalization properties of stochastic gradient methods for learning with convex loss functions and linearly parameterized functions. We show that, in the absence of penalizations or constraints, the stability and approximation properties of the algorithm can be controlled by tuning either the step-size o…
Implicit generative models are difficult to train as no explicit density functions are defined. Generative adversarial nets (GANs) present a minimax framework to train such models, which however can suffer from mode collapse due to the nature of the JS-divergence. This paper presents a learning by teaching (LBT) approa…
Novel framework optimizes experiments for implicit models using mutual information.
problem Optimizing experiments for intractable implicit models.
method Sequential Bayesian Experimental Design using Mutual Information.
result Framework efficiently estimates parameters with few iterations.
Improved score matching methods for estimating score functions and Hessians without high dimensionality.
problem Estimating score functions and Hessians efficiently in high-dimensional data.
method Implicit score matching and denoising score matching, leveraging Gagliardo-Nirenberg inequalities.
result Achieves convergence rates similar to denoising score matching and estimates Hessians without dimensionality issues.
Dual optimization connects ERM-fDR to normalization function.
problem Empirical risk minimization with f-divergence regularization.
method Dual formulation, Legendre-Fenchel transform, implicit function theorem, nonlinear ODE.
result Computational method to calculate normalization function efficiently.
Semi-implicit variational inference (SIVI) is introduced to expand the commonly used analytic variational distribution family, by mixing the variational parameter with a flexible distribution. This mixing distribution can assume any density function, explicit or not, as long as independent random samples can be generat…
JoVA combines two VAEs to learn user and item representations for better recommendation.
problem Collaborative filtering with implicit feedback.
method Joint Variational Autoencoders (JoVA) with a hinge-based pairwise loss function (JoVA-Hinge).
result JoVA-Hinge outperforms state-of-the-art methods in top-k recommendation.
The paper analyzes optimal implicit bias in linear regression for over-parameterized models.
problem Finding the best generalization performance in over-parameterized linear regression.
method Asymptotic analysis of generalization performance for convex functions/potentials.
result Optimal implicit bias that achieves the best generalization error under certain conditions.
New framework improves robustness of implicit neural networks.
problem Ill-posedness and convergence instability in implicit neural networks.
method NEMON framework based on contraction theory for ℓ∞ norm, including well-posedness condition, average iteration, and input-output Lipschitz constant regularization. result Improved accuracy and robustness of implicit models with smaller input-output Lipschitz bounds.
In reinforcement learning, the TD(λ) algorithm is a fundamental policy evaluation method with an efficient online implementation that is suitable for large-scale problems. One practical drawback of TD(λ) is its sensitivity to the choice of the step-size. It is an empirically well-known fact that a large step-size l…
In his book with Alan Jolis, Vers un monde sans pauvreté (1997) Yunus gives the example of a microcredit loan of 1000BDT reimbursed via 50 weekly settlements of 22BDT and correctly claims that this corresponds to the annual interest rate of 20%. But this is without taking into account that if the borrower has good reas…
Recently, there has been a growing interest in the problem of learning rich implicit models - those from which we can sample, but can not evaluate their density. These models apply some parametric function, such as a deep network, to a base measure, and are learned end-to-end using stochastic optimization. One strategy…
Unified framework approximates gradient descent's implicit bias in high dimensions.
problem Understanding gradient descent's behavior in overparameterized settings with convex losses.
method Unified framework for convex losses, including sensitivity analysis.
result Approximation of minimum-norm interpolation in high dimensions.
New methods distill data for deep networks efficiently.
problem Reduction of training data cost and inconvenience.
method Generative teaching networks, gradient matching, Implicit Function Theorem.
result New methods are computationally more efficient and improve model performance.