Gradient matching method estimates implicit regularization in complex deep learning systems.
problem Estimating implicit regularization in modern deep learning systems with complex modifications.
method Gradient matching methods to empirically estimate implicit regularization.
result Empirical estimation of implicit regularization in arbitrary networks, including dropout.
In this paper, we describe the "implicit autoencoder" (IAE), a generative autoencoder in which both the generative path and the recognition path are parametrized by implicit distributions. We use two generative adversarial networks to define the reconstruction and the regularization cost functions of the implicit autoe…
The study analyzes implicit biases in neural networks using backward error analysis.
problem Analyzing implicit biases in multitask and continual learning settings.
method Backward error analysis to compute implicit training biases, deriving modified losses with three terms.
result The conflict term, measuring gradient alignment, is a new quantity in continual learning.
New method learns robot skills from data, matching or outperforming existing methods.
problem Learning robot skills from fixed datasets.
method Offline Reinforcement Learning via Supervised Learning using implicit models.
result Implicit models can match or outperform explicit models in acquiring robotic skills.
Gradient descent converges to a global minimum in nonlinear ReLU implicit networks with linear width.
problem Understanding convergence of gradient methods in nonlinear, infinitely deep ReLU networks.
method Introduced a scaling constant to ensure well-posedness of the equilibrium equation, proving convergence to a global minimum for linear width networks.
result Gradient descent converges to a global minimum at a linear rate for nonlinear ReLU implicit networks with linear width.
New proof shows norms can't explain deep learning's implicit regularization.
problem Understanding the implicit regularization in deep learning.
method Mathematical proof on matrix factorization problems.
result Implicit regularization drives norms towards infinity, suggesting rank minimization is key.
The paper analyzes implicit regularization in tensor factorization using neural networks.
problem Understanding implicit regularization in tensor factorization.
method Dynamical systems perspective and gradient descent analysis.
result Gradient descent induces a form of greedy low tensor rank search.
Deep tensor factorization benefits from implicit regularization with polynomial growth.
problem Tensor factorization's implicit regularization effect in deep networks is not well understood.
method Investigated the implicit regularization in deep tensor factorization, showing polynomial growth.
result Implicit regularization in deep tensor factorization grows polynomially with depth, improving estimation accuracy and convergence.
New TD method stabilizes average-reward learning.
problem Stability issues in average-reward TD learning.
method Implicit fixed point update for average-reward TD(λ). result Improved numerical stability and broader step-size range.
Survey on why deep learning works despite having more parameters than data.
problem Understanding why deep learning algorithms generalize well despite having more parameters than training data.
method Explains the concept of implicit bias and reviews recent research findings.
result Implicit bias is a key factor in deep learning's ability to generalize.
IDAC improves reinforcement learning efficiency by modeling implicit distributions.
problem Improving sample efficiency in reinforcement learning algorithms.
method IDAC uses two DGNs for a distributional critic and a semi-implicit actor to model implicit policy distributions.
result IDAC outperforms state-of-the-art algorithms on OpenAI Gym environments.
The paper explains implicit regularization in hierarchical tensor factorization and deep CNNs.
problem Understanding implicit regularization in complex neural network architectures.
method Theoretical analysis using dynamical systems to overcome challenges in hierarchy.
result Established implicit regularization towards low hierarchical tensor rank, equivalent to locality in CNNs.
Study shows SGD's generalization is not explained by implicit bias.
problem Explaining the generalization ability of overparameterized learning algorithms.
method Revisited Stochastic Convex Optimization with SGD, demonstrating limitations of implicit bias.
result No distribution-independent or distribution-dependent implicit regularizer can explain SGD's generalization.
New insights into network generalization show learning rate affects both norm and sharpness.
problem Understanding the generalization of overparameterized networks.
method Empirical analysis and theoretical proof of the trade-off between norm and sharpness.
result Learning rate influences both norm and sharpness, neither alone minimizes generalization error.
This paper analyzes implicit bias in Deep Linear Discriminant Analysis.
problem The implicit bias of Deep Linear Discriminant Analysis.
method Analyzing gradient flow on a L-layer diagonal linear network.
result Under balanced initialization, the network transforms additive updates into multiplicative updates, conserving the (2/L) quasi-norm.
Proposes efficient, modular method for implicit differentiation.
problem Implicit differentiation of optimization problems.
method Automatic implicit differentiation using autodiff and implicit function theorem.
result Automatic differentiation of optimization problems is made easier and more modular.
Sayer uses implicit feedback to optimize system policies.
problem Leveraging implicit feedback to improve system policies is difficult due to bias and incompleteness.
method Sayer combines randomized exploration and unbiased counterfactual estimators to evaluate and train new policies using implicit feedback.
result Sayer can accurately evaluate and train new policies that outperform existing ones.
Implicit deep learning prediction rules generalize the recursive rules of feedforward neural networks. Such rules are based on the solution of a fixed-point equation involving a single vector of hidden features, which is thus only implicitly defined. The implicit framework greatly simplifies the notation of deep learni…
A method for fair representation learning through bi-level optimization and implicit differentiation.
problem Ensuring fair predictors invariant across sub-groups.
method Bi-level optimization with inner-loop for invariant predictors, implicit path alignment for efficiency.
result Consistently better trade-off in prediction performance and fairness measurement.
Deep equilibrium models converge globally without explicit computation.
problem Global convergence of deep learning models with implicit layers.
method Analysis of gradient dynamics and proof of convergence rate.
result Deep equilibrium models converge to global optimum at a linear rate.
This paper shows how to train only the implicit layer of overparameterized implicit neural networks.
problem Understanding how the implicit layer contributes to the training of overparameterized implicit neural networks.
method Restricting training to only the implicit layer and analyzing the generalization error for ReLU-activated networks.
result Global convergence is guaranteed even if only the implicit layer is trained, and gradient flow with proper random initialization can achieve small generalization errors.
Implicit discourse relation classification is of great challenge due to the lack of connectives as strong linguistic cues, which motivates the use of annotated implicit connectives to improve the recognition. We propose a feature imitation framework in which an implicit relation network is driven to learn from another …
This paper proves SGD converges to global minimum for over-parameterized ReLU networks.
problem Theoretical understanding of implicit neural networks is limited.
method Gradient flow analysis of ReLU activated implicit neural networks.
result Randomly initialized gradient descent converges to global minimum at a linear rate for square loss function in over-parameterized ReLU networks.
Implicit Q-learning and SARSA adjust step-sizes automatically, improving stability and performance.
problem Numerical instability and slow progress in Q-learning and SARSA due to step-size calibration.
method Reformulate iterative updates as fixed-point equations, scaling step-sizes inversely with feature norms.
result Implicit methods maintain stability over broader step-size ranges and achieve comparable convergence rates.
Implicit generative models are difficult to train as no explicit density functions are defined. Generative adversarial nets (GANs) present a minimax framework to train such models, which however can suffer from mode collapse due to the nature of the JS-divergence. This paper presents a learning by teaching (LBT) approa…
Machine learning improves implicit solvent models for molecular dynamics.
problem Accurate modeling of solvent effects for biological molecules is challenging.
method Leveraging machine learning and multi-scale coarse graining, ISSNet models implicit solvent potentials.
result ISSNet models outperform traditional methods in reproducing protein thermodynamics.
Geometric framework explains and controls implicit bias in machine learning.
problem Understanding and controlling the selection of solutions in overparameterized models.
method Developed a theoretical and constructive framework based on geometric corrections induced by gradient noise and continuous symmetries of the loss.
result Computed the induced bias across various architectures and enabled inverse design to shape the bias.
New algorithm expands FTRL framework with improved worst-case regret bounds.
problem Online learning with improved worst-case regret bounds.
method Generalized implicit Follow-The-Regularized-Leader (FTRL) algorithm.
result Unified framework for designing updates improving worst-case regret bounds.
FA algorithm provides convergence guarantees for deep linear networks.
problem Training efficiency and convergence of deep neural networks.
method Theoretical analysis of Feedback Alignment (FA) algorithm for deep linear networks.
result Certain initializations lead to implicit anti-regularization, affecting learning effectiveness.
Neural networks learn the support of the target function through SGD's implicit regularization effect.
problem Learning the support of the target function in neural networks.
method Investigation of mini-batch SGD's ability to learn the support in the first layer of a neural network.
result Mini-batch SGD effectively learns the support in the first layer by shrinking irrelevant weights, while vanilla GD requires an explicit regularization term.
We extend the existing framework of semi-implicit variational inference (SIVI) and introduce doubly semi-implicit variational inference (DSIVI), a way to perform variational inference and learning when both the approximate posterior and the prior distribution are semi-implicit. In other words, DSIVI performs inference …
FAWMF adapts weights for implicit feedback recommendation efficiently.
problem Challenges in treating unobserved data as negative in implicit feedback recommendation.
method FAWMF uses a variational auto-encoder with a parameterized neural network to adaptively assign personalized data confidence weights, and fBGD for efficient learning.
result FAWMF and fBGD outperform existing methods in real-world datasets.
SGD with large learning rates can achieve better test accuracy than expected.
problem SGD with large learning rates often outperforms expected convergence bounds.
method Proved that SGD with small learning rates stays close to gradient flow path on modified loss.
result Explicitly adding an implicit regularizer to the loss improves test accuracy.
Novel approach finds implicit regularisation in two-player games using BEA.
problem Understanding implicit regularisation in two-player games.
method Using backward error analysis to construct continuous-time flows with gradient-eligible vector fields.
result Identifies new implicit regularisation effects in two-player games.
User preferences for items can be inferred from either explicit feedback, such as item ratings, or implicit feedback, such as rental histories. Research in collaborative filtering has concentrated on explicit feedback, resulting in the development of accurate and scalable models. However, since explicit feedback is oft…
The paper challenges the belief that more inner iterations at test time improve performance in implicit deep learning.
problem The performance improvement of implicit deep learning models with increased inner iterations at test time.
method Theoretical analysis of a simple setting, validation on implicit deep learning problems.
result Overparametrization plays a key role; increasing the number of iterations at test time does not improve performance for overparametrized networks.
Efforts to understand the generalization mystery in deep learning have led to the belief that gradient-based optimization induces a form of implicit regularization, a bias towards models of low "complexity." We study the implicit regularization of gradient descent over deep linear neural networks for matrix completion …
We propose an adversarial training procedure for learning a causal implicit generative model for a given causal graph. We show that adversarial training can be used to learn a generative model with true observational and interventional distributions if the generator architecture is consistent with the given causal grap…
XLVINs improve data efficiency in implicit planning by leveraging latent space.
problem Improving data efficiency in implicit planning algorithms.
method XLVINs use a high-dimensional latent space to perform planning computations, breaking the algorithmic bottleneck.
result XLVINs significantly improve data efficiency across various settings compared to value iteration-based implicit planners and model-free baselines.
Proposes a new reinforcement learning method to improve agent performance in control tasks.
problem Shortcomings of maximum likelihood estimation in model-based reinforcement learning.
method Directly optimizes expected returns using implicit differentiation of a Bellman optimality function.
result Empirical evidence shows improved performance in model misspecification regime.
This work shows how exploiting gradient alignment can improve distributed and federated learning performance.
problem Misalignment of gradients across clients in distributed and federated learning.
method Utilizing implicit regularization through a novel GradAlign algorithm that induces gradient alignment with large mini-batches.
result Improvements in test accuracies and generalization performance.
EnCF improves data assimilation for implicit, non-smooth observations.
problem Data assimilation challenges with implicit, many-to-one observations.
method EnCF uses a stochastic controlled flow to update forecast distributions.
result EnCF outperforms Kalman filters for non-Gaussian, implicit observations.
New variational approach to deep learning via gradient descent.
problem Non-robustness and poor out-of-distribution generalization in deep learning.
method Regularize variational neural networks using gradient descent's implicit bias.
result Strong in- and out-of-distribution performance achieved without additional hyperparameter tuning.
We introduce the implicit processes (IPs), a stochastic process that places implicitly defined multivariate distributions over any finite collections of random variables. IPs are therefore highly flexible implicit priors over functions, with examples including data simulators, Bayesian neural networks and non-linear tr…
Implicit models, which allow for the generation of samples but not for point-wise evaluation of probabilities, are omnipresent in real-world problems tackled by machine learning and a hot topic of current research. Some examples include data simulators that are widely used in engineering and scientific research, genera…
A core capability of intelligent systems is the ability to quickly learn new tasks by drawing on prior experience. Gradient (or optimization) based meta-learning has recently emerged as an effective approach for few-shot learning. In this formulation, meta-parameters are learned in the outer loop, while task-specific m…
The paper learns compact implicit surface maps from streaming data using an ensemble of sparse Gaussian processes.
problem Creating compact and accurate implicit surface maps from streaming range data.
method An ensemble of sparse Gaussian process experts, incrementally adjusted, trades-off between model complexity and prediction error.
result The approach learns compact and accurate implicit surface models comparable to or better than exact GP regression with subsampled data.
Implicit models can match or exceed explicit models with more test-time compute.
problem Understanding the expressive power and scaling of implicit models.
method Nonparametric analysis of expressive power, mathematical characterization of implicit operators, and test-time scaling experiments.
result Implicit models can progressively express more complex mappings through iteration, matching a richer function class with test-time compute.