Deep equilibrium models converge globally without explicit computation.
problem Global convergence of deep learning models with implicit layers.
method Analysis of gradient dynamics and proof of convergence rate.
result Deep equilibrium models converge to global optimum at a linear rate.
RevDEQs improve performance on tasks with exact gradients and fewer function evaluations.
problem Inexact gradient calculation in DEQs leads to unstable training and requires regularisation or many function evaluations.
method Introduce Reversible Deep Equilibrium Models (RevDEQs) that allow for exact gradient calculation, no regularisation, and far fewer function evaluations.
result RevDEQs significantly improve performance on language modelling and image classification tasks.
Wide neural networks converge to Gaussian processes, improving generalization.
problem Understanding the generalization of wide neural networks, especially deep equilibrium models.
method Investigation of deep equilibrium models (DEQs) with infinite-depth layers, focusing on their convergence to Gaussian processes as width and depth approach infinity.
result Wide DEQs converge to Gaussian processes, maintaining generalization performance.
New approaches improve adversarial robustness of DEQs.
problem Adversarial vulnerability of DEQs.
method Developed approaches to estimate intermediate gradients and integrate them into attacking pipelines.
result Demonstrated adversarial robustness of DEQs competitive with deep networks.
Deep equilibrium models estimate latent variables from data.
problem Estimating latent variables from data.
method Generalized exponential family models, deep equilibrium networks.
result Deep equilibrium models solve MAP estimates for latent and transformation parameters.
Deep neural networks show some layers better align with data than others.
problem Understanding why some layers in deep neural networks better align with data.
method Introducing the Equilibrium Hypothesis to connect alignment pattern to signal propagation.
result The Equilibrium Hypothesis explains the ascent-descent pattern of alignment in deep neural networks.
Deep CapsNet improves sign language recognition from wearable IMUs.
problem Continuous recognition of sign language from wearable devices.
method Custom CapsNet architecture using deep capsule networks and game theory.
result Improved accuracy of 94% and 92.50% for 3 and 5 routings respectively, compared to 87.99% for CNN.
We present a new approach to modeling sequential data: the deep equilibrium model (DEQ). Motivated by an observation that the hidden layers of many existing deep sequence models converge towards some fixed point, we propose the DEQ approach that directly finds these equilibrium points via root-finding. Such a method is…
New method stabilizes DEQ models by regularizing Jacobian of fixed-point equations.
problem Stability and performance of DEQ models.
method Jacobian regularization to stabilize DEQ models.
result Significant stabilization of fixed-point convergence in DEQ models.
Generative operators solve many convex problems with minimal parameters.
problem Worst-case parameter bounds limit the practical use of neural operators.
method Developed generative equilibrium operators (GEOs) using realizable finite-dimensional layers.
result GEOs can uniformly approximate solutions to convex optimization problems with logarithmic growth in parameters.
Improves adversarial robustness of DEQ models by regulating neural dynamics.
problem Limited adversarial robustness of DEQ models.
method Interprets DEQs as neural dynamics, uses entropy reduction and random intermediate states.
result Significantly increases adversarial robustness of DEQ models.
We propose a novel approach to addressing the vanishing (or exploding) gradient problem in deep neural networks. We construct a new architecture for deep neural networks where all layers (except the output layer) of the network are a combination of rotation, permutation, diagonal, and activation sublayers which are all…
Two-layer model studies reinsurance contracts and competition between insurer and reinsurers.
problem Modeling and analyzing reinsurance contracts and competition between insurer and reinsurers.
method Two-layer stochastic game model with insurer negotiating with reinsurers, and reinsurers competing for business.
result Existence and uniqueness of equilibrium strategies for the insurer and reinsurers, characterized in semiclosed form.
This paper proves SGD converges to global minimum for over-parameterized ReLU networks.
problem Theoretical understanding of implicit neural networks is limited.
method Gradient flow analysis of ReLU activated implicit neural networks.
result Randomly initialized gradient descent converges to global minimum at a linear rate for square loss function in over-parameterized ReLU networks.
Stochastic normalizing flows improve lattice field theory simulations.
problem Efficiently sample lattice field theories.
method Combining neural-network layers with Monte Carlo updates.
result Stochastic normalizing flows are equivalent to out-of-equilibrium simulations.
A multi-layer deep Gaussian process (DGP) model is a hierarchical composition of GP models with a greater expressive power. Exact DGP inference is intractable, which has motivated the recent development of deterministic and stochastic approximation methods. Unfortunately, the deterministic approximation methods yield a…
The paper analyzes reinsurance strategies in a competitive multi-agent system.
problem Strategic interactions and competitive behavior in multi-layer reinsurance chains.
method Stochastic differential games and non-zero-sum game models to characterize strategic interactions. Dynamic programming and game theory to derive equilibrium strategies.
result Intensified competition reduces safety loadings in reinsurance contracts.
Gradient descent converges to a global minimum in nonlinear ReLU implicit networks with linear width.
problem Understanding convergence of gradient methods in nonlinear, infinitely deep ReLU networks.
method Introduced a scaling constant to ensure well-posedness of the equilibrium equation, proving convergence to a global minimum for linear width networks.
result Gradient descent converges to a global minimum at a linear rate for nonlinear ReLU implicit networks with linear width.
DEQs converge to optimal solutions with mild over-parameterization.
problem Training over-parameterized deep equilibrium models.
method Solves equilibrium point directly, uses gradient descent, and analyzes convergence via linear rate.
result Gradient descent converges to a globally optimal solution at a linear rate for quadratic loss.
DDEQs extend DEQs to discrete measure inputs using Wasserstein gradient flows.
problem Applying DEQs to discrete measure inputs like sets or point clouds.
method Wasserstein gradient flows for finding fixed points of discrete measures under permutation-invariance.
result DDEQs can compete with state-of-the-art models in tasks like point cloud classification and completion.
This paper provides a general framework for modeling financial contagion in a system with obligations in multiple illiquid assets (e.g., currencies). In so doing, we develop a multi-layered financial network that extends the single network of Eisenberg and Noe (2001). In particular, we develop a financial contagion mod…
This paper tackles traffic volume estimation challenges with a deep learning method.
problem Underdetermined and non-equilibrium traffic flows.
method Graph-based deep learning method with adaptive attention mechanisms.
result The proposed model achieves high accuracy even with low sensor coverage.
DREAM learns optimal strategies in imperfect games without needing a simulator.
problem Learning optimal strategies in imperfect-information games with multiple agents.
method DREAM is a deep reinforcement learning algorithm that converges to Nash Equilibria and coarse correlated equilibria.
result DREAM achieves state-of-the-art performance in benchmark games and is competitive with simulator-based algorithms.
Deep fictitious play converges to Nash equilibrium in stochastic differential games.
problem Finding Nash equilibrium in large stochastic differential games.
method Decouples the game into sub-optimization problems and solves each player's optimal strategy with deep BSDE method.
result Deep fictitious play converges to the true Nash equilibrium.
Study of insurance market equilibria with risk-averse policyholders.
problem Analyzing optimal insurance contracts in a monopoly market with risk-averse policyholders.
method Modeling Stackelberg equilibria with a profit-maximizing insurer and a risk-averse policyholder.
result Equilibrium contracts exhibit a layer-type structure, providing full insurance over pessimistic loss layers and no coverage over optimistic ones.
The paper analyzes strategic interactions in a multi-agent reinsurance chain using game theory.
problem Strategic behavior and competition among insurers and reinsurers in a multi-layer reinsurance chain.
method Employed Stackelberg differential games and non-zero-sum game models to characterize strategic interactions. Used dynamic programming and game theory to derive equilibrium strategies for investment and reinsurance.
result Intensified competition leads to reduced safety loadings in reinsurance contracts.
Deep learning models replicate Kyle's model's market equilibrium.
problem Understanding market equilibria in asymmetric information settings.
method Using deep neural networks to model agents in Kyle's single period model.
result Trained networks' behavior converges to Kyle's predicted equilibrium.
This paper extends the convergence rate of DEQs with ReLU to any general activation.
problem Proving global convergence rate for DEQs with general activations.
method Developed a novel population Gram matrix and new form of dual activation with Hermite polynomial expansion.
result Gradient descent converges to a globally optimal solution at a linear rate for DEQs with general activations.
Computing equilibrium states in condensed-matter many-body systems, such as solvated proteins, is a long-standing challenge. Lacking methods for generating statistically independent equilibrium samples in "one shot", vast computational effort is invested for simulating these system in small steps, e.g., using Molecular…
Deep learning has the potential to revolutionize quantum chemistry as it is ideally suited to learn representations for structured data and speed up the exploration of chemical space. While convolutional neural networks have proven to be the first choice for images, audio and video data, the atoms in molecules are not …
Gradient descent dynamics studied for DEQs in linear and single-index models.
problem Understanding gradient descent dynamics for DEQs.
method Rigorously studied gradient descent dynamics for DEQs in linear and single-index models.
result Gradient descent converges to a global minimizer for linear DEQs and single-index models.
FNO-DEQ solves steady-state PDEs as fixed points, outperforming traditional FNOs.
problem Lack of understanding in designing neural network architectures for PDEs.
method Proposes FNO-DEQ, a deep equilibrium architecture that solves steady-state PDEs as fixed points.
result FNO-DEQ outperforms FNO-based architectures in predicting solutions to steady-state PDEs.
A central problem in machine learning involves modeling complex data-sets using highly flexible families of probability distributions in which learning, sampling, inference, and evaluation are still analytically or computationally tractable. Here, we develop an approach that simultaneously achieves both flexibility and…
Deep-MacroFin uses neural networks to solve complex economic models efficiently.
problem Solving high-dimensional partial differential equations in continuous time economics.
method Leverages deep learning, specifically Multi-Layer Perceptrons and Kolmogorov-Arnold Networks, optimized with HJB equations.
result Offers a more efficient solution (5imes less memory, 40imes fewer FLOPs) for 50D economic models. New model outperforms Neural ODEs while being more efficient.
problem Stable convergence and existence guarantees for implicit-depth models.
method Developed Monotone Operator Equilibrium Network (monDEQ) based on monotone operator theory.
result MonDEQ models outperform Neural ODEs and are more computationally efficient.
Discovery of atomistic systems with desirable properties is a major challenge in chemistry and material science. Here we introduce a novel, autoregressive, convolutional deep neural network architecture that generates molecular equilibrium structures by sequentially placing atoms in three-dimensional space. The model e…
HomoODE connects DEQs and Neural ODEs via homotopy continuation, improving accuracy and memory efficiency.
problem Connecting DEQs and Neural ODEs for better model performance and efficiency.
method Established a connection between DEQs and Neural ODEs using homotopy continuation, proposing HomoODE.
result HomoODE outperforms existing implicit models in accuracy and memory consumption.
A new method uses deep learning to predict rare events in complex systems.
problem Predicting rare and extreme events in non-equilibrium systems.
method A deep learning approach that minimizes the geometrical action.
result The method accurately predicts rare events in various complex systems.
New theory predicts deep neural networks can operate in an extended critical regime without fine-tuning.
problem Understanding the dynamics and computational principles of deep neural networks.
method Combining theories of heavy-tailed random matrices and non-equilibrium statistical physics.
result Deep neural networks can operate in an extended critical regime without fine-tuning parameters.
Deep learning solves and estimates complex financial models.
problem Estimating and solving continuous-time financial models.
method Uses deep learning to solve and estimate models simultaneously.
result Demonstrates advantages like generality and large state space handling.
This paper introduces Non-Autonomous Input-Output Stable Network(NAIS-Net), a very deep architecture where each stacked processing block is derived from a time-invariant non-autonomous dynamical system. Non-autonomy is implemented by skip connections from the block input to each of the unrolled processing stages and al…
End-to-end model predicts multiagent trajectories using game theory and neural nets.
problem Predicting trajectories of interacting agents in complex scenarios.
method Hybrid neural net with game-theoretic reasoning, using implicit layers to map preferences to Nash equilibria.
result Trains an interpretable model that predicts future trajectories and transfers to decision making.
Proposes a deep learning method for solving complex financial games with delays.
problem Financial modeling with multi-agent interactions and delayed effects.
method Parameterizes controls using recurrent neural networks and trains them with modified fictitious play.
result Demonstrates effectiveness on finance problems with known solutions and new problems with derived Nash equilibria.
Improved sampling for gauge theory with SNFs.
problem Sampling from target probability distributions in gauge theory.
method Stochastic Normalizing Flows (SNFs) for SU(3) lattice gauge theory. result Promising scaling properties of SNFs with degrees of freedom.
MDEQ models learn multi-resolution features efficiently.
problem Large-scale, hierarchical pattern recognition.
method Implicit differentiation, multiscale deep equilibrium model.
result MDEQs achieve performance on par with recent models.
We propose a deep neural network-based algorithm to identify the Markovian Nash equilibrium of general large N-player stochastic differential games. Following the idea of fictitious play, we recast the N-player game into N decoupled decision problems (one for each player) and solve them iteratively. The individua…
Deep RL solves complex economic models with heterogeneous agents.
problem Solving models with heterogeneous economic actors is difficult.
method Reinforcement Learning techniques for solving general equilibrium models.
result Successfully captures economic behaviors induced by age-based health risks.
In this paper, we apply the idea of fictitious play to design deep neural networks (DNNs), and develop deep learning theory and algorithms for computing the Nash equilibrium of asymmetric N-player non-zero-sum stochastic differential games, for which we refer as \emph{deep fictitious play}, a multi-stage learning pro…