Continuous semi-implicit models enable faster training and better performance in generative modeling.
problem Slow convergence in hierarchical semi-implicit models during training.
method CoSIM, a continuous semi-implicit model that incorporates a continuous transition kernel for efficient training.
result CoSIM achieves superior performance on image generation tasks compared to existing methods.
Implicit probabilistic models are a flexible class of models defined by a simulation process for data. They form the basis for theories which encompass our understanding of the physical world. Despite this fundamental nature, the use of implicit models remains limited due to challenges in specifying complex latent stru…
In this paper, we revisit implicit regularization from the ground up using notions from dynamical systems and invariant subspaces of Morse functions. The key contributions are a new criterion for implicit regularization---a leading contender to explain the generalization power of deep models such as neural networks---a…
We propose an adversarial training procedure for learning a causal implicit generative model for a given causal graph. We show that adversarial training can be used to learn a generative model with true observational and interventional distributions if the generator architecture is consistent with the given causal grap…
Implicit generative models are difficult to train as no explicit density functions are defined. Generative adversarial nets (GANs) present a minimax framework to train such models, which however can suffer from mode collapse due to the nature of the JS-divergence. This paper presents a learning by teaching (LBT) approa…
To combine explicit and implicit generative models, we introduce semi-implicit generator (SIG) as a flexible hierarchical model that can be trained in the maximum likelihood framework. Both theoretically and experimentally, we demonstrate that SIG can generate high quality samples especially when dealing with multi-mod…
This paper shows how to train only the implicit layer of overparameterized implicit neural networks.
problem Understanding how the implicit layer contributes to the training of overparameterized implicit neural networks.
method Restricting training to only the implicit layer and analyzing the generalization error for ReLU-activated networks.
result Global convergence is guaranteed even if only the implicit layer is trained, and gradient flow with proper random initialization can achieve small generalization errors.
Learning algorithms for implicit generative models can optimize a variety of criteria that measure how the data distribution differs from the implicit model distribution, including the Wasserstein distance, the Energy distance, and the Maximum Mean Discrepancy criterion. A careful look at the geometries induced by thes…
Semi-implicit graph variational auto-encoder (SIG-VAE) is proposed to expand the flexibility of variational graph auto-encoders (VGAE) to model graph data. SIG-VAE employs a hierarchical variational framework to enable neighboring node sharing for better generative modeling of graph dependency structure, together with …
New insights into network generalization show learning rate affects both norm and sharpness.
problem Understanding the generalization of overparameterized networks.
method Empirical analysis and theoretical proof of the trade-off between norm and sharpness.
result Learning rate influences both norm and sharpness, neither alone minimizes generalization error.
Gradient descent on ReLU networks with square loss implicitly favors balanced weights.
problem Understanding implicit regularization in nonlinear neural networks with regression losses.
method Analyzing gradient descent dynamics on ReLU networks with square loss.
result It is impossible to characterize the implicit regularization of ReLU networks with square loss by any explicit function of model parameters.
Proposes LDIDPs for efficient sequential data generation from latent dynamical models.
problem Challenges in generating high-fidelity sequential samples from latent dynamical models.
method Utilizes implicit diffusion processes to sample from latent dynamical processes.
result Demonstrates accurate learning of dynamics and efficient generation of high-quality sequential data.
Paper proposes a new method for probabilistic electricity price forecasting.
problem Accurate estimation of forecast uncertainties for optimal decision making.
method Implicit generative ensemble post-processing using an ensemble of point forecasting models.
result Method outperforms well-established model combination benchmarks.
Study shows SGD's generalization is not explained by implicit bias.
problem Explaining the generalization ability of overparameterized learning algorithms.
method Revisited Stochastic Convex Optimization with SGD, demonstrating limitations of implicit bias.
result No distribution-independent or distribution-dependent implicit regularizer can explain SGD's generalization.
Efforts to understand the generalization mystery in deep learning have led to the belief that gradient-based optimization induces a form of implicit regularization, a bias towards models of low "complexity." We study the implicit regularization of gradient descent over deep linear neural networks for matrix completion …
Study shows how bias in optimization affects robustness in adversarial settings.
problem Understanding and mitigating implicit bias in adversarially robust models.
method Analyzes the implicit bias in robust empirical risk minimization and its impact on generalization.
result Implicit bias in optimization can significantly affect robust generalization.
Novel framework optimizes experiments for implicit models using mutual information.
problem Optimizing experiments for intractable implicit models.
method Sequential Bayesian Experimental Design using Mutual Information.
result Framework efficiently estimates parameters with few iterations.
SSMs can be poisoned with clean labels, leading to generalization failure.
problem The implicit bias of SSMs can be manipulated by including special training examples with clean labels.
method Formal proof and empirical demonstration of the phenomenon.
result SSMs can fail to generalize even with clean labels, due to the inclusion of special training examples.
IDAC improves reinforcement learning efficiency by modeling implicit distributions.
problem Improving sample efficiency in reinforcement learning algorithms.
method IDAC uses two DGNs for a distributional critic and a semi-implicit actor to model implicit policy distributions.
result IDAC outperforms state-of-the-art algorithms on OpenAI Gym environments.
Stochastic recurrent neural networks with latent random variables of complex dependency structures have shown to be more successful in modeling sequential data than deterministic deep models. However, the majority of existing methods have limited expressive power due to the Gaussian assumption of latent variables. In t…
New method trains generative models without discriminators, improving stability and accuracy.
problem Training implicit generative models with adversarial discriminators leads to instability and mode-dropping.
method Invariant statistical loss function, avoiding discriminators.
result Successfully trains generative models for various complex distributions without mode-dropping.
In this paper, we describe the "implicit autoencoder" (IAE), a generative autoencoder in which both the generative path and the recognition path are parametrized by implicit distributions. We use two generative adversarial networks to define the reconstruction and the regularization cost functions of the implicit autoe…
The paper analyzes optimal implicit bias in linear regression for over-parameterized models.
problem Finding the best generalization performance in over-parameterized linear regression.
method Asymptotic analysis of generalization performance for convex functions/potentials.
result Optimal implicit bias that achieves the best generalization error under certain conditions.
Dual-ISL improves implicit generative model training with convex optimization and explicit density approximation.
problem Training implicit generative models with robust and practical likelihood-free objectives.
method Introduces dual-ISL, a novel likelihood-free objective using a convex divergence derived from the invariant statistical loss (ISL) framework.
result Dual-ISL yields a convex optimization problem in the space of model densities, providing explicit density approximation and improved training stability.
New framework improves robustness of implicit neural networks.
problem Ill-posedness and convergence instability in implicit neural networks.
method NEMON framework based on contraction theory for ℓ∞ norm, including well-posedness condition, average iteration, and input-output Lipschitz constant regularization. result Improved accuracy and robustness of implicit models with smaller input-output Lipschitz bounds.
This paper measures the intensity of implicit government guarantees using PMC index model.
problem Excessive local government debt due to implicit government guarantees.
method Text mining of policy documents related to municipal investment bonds, PMC index model.
result Recent policies have reduced the intensity of implicit government guarantees.
New method uses neural networks to optimize experimental designs for complex models.
problem Designing experiments for complex, intractable models with high computational cost.
method Neural mutual information estimation for mutual information maximization.
result Optimal experimental designs and posterior inference can be jointly determined.
Implicit models, which allow for the generation of samples but not for point-wise evaluation of probabilities, are omnipresent in real-world problems tackled by machine learning and a hot topic of current research. Some examples include data simulators that are widely used in engineering and scientific research, genera…
New variational approach to deep learning via gradient descent.
problem Non-robustness and poor out-of-distribution generalization in deep learning.
method Regularize variational neural networks using gradient descent's implicit bias.
result Strong in- and out-of-distribution performance achieved without additional hyperparameter tuning.
Machine learning improves implicit solvent models for molecular dynamics.
problem Accurate modeling of solvent effects for biological molecules is challenging.
method Leveraging machine learning and multi-scale coarse graining, ISSNet models implicit solvent potentials.
result ISSNet models outperform traditional methods in reproducing protein thermodynamics.
The study provides a generalization bound for a family of implicit networks.
problem Theoretical understanding of implicit networks' generalization is limited.
method A generalization bound is derived for a family of implicit networks using a covering number argument for Rademacher complexity.
result A theoretical generalization bound is established for implicit networks.
Recent works propose using the discriminator of a GAN to filter out unrealistic samples of the generator. We generalize these ideas by introducing the implicit Metropolis-Hastings algorithm. For any implicit probabilistic model and a target distribution represented by a set of samples, implicit Metropolis-Hastings oper…
SIP corrects model bias in Bayesian ML.
problem Model selection biases predictions in Bayesian ML.
method Sparse Implicit Processes (SIP) for flexible, trainable predictions.
result SIP provides better predictive distributions than initial models.
Progress in probabilistic generative models has accelerated, developing richer models with neural architectures, implicit densities, and with scalable algorithms for their Bayesian inference. However, there has been limited progress in models that capture causal relationships, for example, how individual genetic factor…
For the implicit systems of first order ordinary differential equations on the plane there is presented the complete local classification of generic singularities of family of its phase curves up to smooth orbital equivalence. Besides the well known singularities of generic vector fields on the plane and the singularit…
Modern implicit generative models such as generative adversarial networks (GANs) are generally known to suffer from issues such as instability, uninterpretability, and difficulty in assessing their performance. If we see these implicit models as dynamical systems, some of these issues are caused by being unable to cont…
Modern implicit generative models such as generative adversarial networks (GANs) are generally known to suffer from issues such as instability, uninterpretability, and difficulty in assessing their performance. If we see these implicit models as dynamical systems, some of these issues are caused by being unable to cont…
Deep equilibrium models converge globally without explicit computation.
problem Global convergence of deep learning models with implicit layers.
method Analysis of gradient dynamics and proof of convergence rate.
result Deep equilibrium models converge to global optimum at a linear rate.
VRSMD improves SMD convergence and has implicit regularization.
problem Efficiently estimating models with large datasets.
method Variance reduction in stochastic mirror descent.
result VRSMD converges to the minimum mirror interpolant.
SAM improves generalization in overparameterized models, but its behavior in tensorized models is less understood.
problem Understanding the implicit regularization of SAM in tensorized models.
method Scale-invariance analysis and gradient flow analysis to derive Norm Deviation as a measure of core norm imbalance, and propose Deviation-Aware Scaling (DAS).
result DAS achieves competitive or improved performance over SAM, while offering reduced computational overhead.
Sayer uses implicit feedback to optimize system policies.
problem Leveraging implicit feedback to improve system policies is difficult due to bias and incompleteness.
method Sayer combines randomized exploration and unbiased counterfactual estimators to evaluate and train new policies using implicit feedback.
result Sayer can accurately evaluate and train new policies that outperform existing ones.
The paper explains implicit regularization in hierarchical tensor factorization and deep CNNs.
problem Understanding implicit regularization in complex neural network architectures.
method Theoretical analysis using dynamical systems to overcome challenges in hierarchy.
result Established implicit regularization towards low hierarchical tensor rank, equivalent to locality in CNNs.
Gradient descent converges to a global minimum in nonlinear ReLU implicit networks with linear width.
problem Understanding convergence of gradient methods in nonlinear, infinitely deep ReLU networks.
method Introduced a scaling constant to ensure well-posedness of the equilibrium equation, proving convergence to a global minimum for linear width networks.
result Gradient descent converges to a global minimum at a linear rate for nonlinear ReLU implicit networks with linear width.
Survey on why deep learning works despite having more parameters than data.
problem Understanding why deep learning algorithms generalize well despite having more parameters than training data.
method Explains the concept of implicit bias and reviews recent research findings.
result Implicit bias is a key factor in deep learning's ability to generalize.
New algorithm expands FTRL framework with improved worst-case regret bounds.
problem Online learning with improved worst-case regret bounds.
method Generalized implicit Follow-The-Regularized-Leader (FTRL) algorithm.
result Unified framework for designing updates improving worst-case regret bounds.
The paper analyzes implicit regularization in tensor factorization using neural networks.
problem Understanding implicit regularization in tensor factorization.
method Dynamical systems perspective and gradient descent analysis.
result Gradient descent induces a form of greedy low tensor rank search.
Gradient descent with early stopping achieves optimal sparse recovery.
problem Sparse regression with gradient descent and early stopping.
method Gradient descent on depth-N networks with early stopping.
result Implicit sparse regularization occurs with early stopping for general depth N.
Language models perform worse with implicit reward models than explicit ones.
problem Understanding why implicit reward models generalize worse than explicit ones.
method Investigated the root cause of the generalization gap between IM-RMs and EX-RMs.
result Implicit reward models rely more on superficial token-level cues, leading to worse generalization.