Monotonic differentiable sorting networks improve upon previous methods.
problem Non-monotonicity in differentiable sorting networks.
method Relaxation of conditional swap operations using sigmoid functions to ensure monotonicity.
result Monotonic differentiable sorting networks improve upon previous methods.
Paper proposes a novel method to estimate differential networks using additional knowledge.
problem Estimating differential statistical dependency networks in high-dimensional data with limited samples.
method Integrates various sources of knowledge beyond data samples to improve differential network estimation.
result Achieves sharp asymptotic convergence rate and improved differential network estimation.
Efficient neural networks compute various differential operators cheaply.
problem Efficient computation of higher time complexity differential operators.
method Restricted neural network architectures with diagonal and hollow Jacobian matrices, allowing efficient extraction of dimension-wise derivatives.
result Demonstrated efficient computation of differential operators for various applications.
Differential equations explain neural network dynamics.
problem Understanding neural networks using differential equations.
method Analyzing properties of differential equations as machine learning models.
result Neural networks can be modeled by differential equations.
DiffEqFlux.jl integrates neural networks with differential equations.
problem Combining machine learning and differential equations for modeling complex systems.
method Fusing neural networks and differential equations using DiffEqFlux.jl.
result Demonstrates the integration of differential equations into neural networks and vice versa.
This work integrates differentiation and integration in Physics-Informed Neural Networks.
problem Solving integro-differential equations and computing integral transforms.
method Augmenting Physics-Informed Neural Networks with automatic integration.
result Solving complex integral transforms and integro-differential equations.
Neural networks can approximate complex stochastic equations well.
problem Approximating general stochastic differential equations.
method Identified neural network classes approximating continuous functions.
result Neural stochastic differential equations can approximate general stochastic differential equations arbitrarily well.
Pruning neural networks adds differential privacy noise, preserving data utility.
problem Achieving differential privacy in neural networks without sacrificing data utility.
method Proving equivalence between pruning and adding differential privacy noise to hidden-layer activations.
result Pruning can be a more effective alternative to adding differential privacy noise for neural networks.
New neural network uses differential equations for adaptable activation functions.
problem Fixed activation functions limit neural network performance and size.
method Introduces differential equation units (DEUs) that learn nonlinear activation functions.
result DEUs enable more compact networks with comparable performance.
GRAND ensures node-level differential privacy for network data.
problem Lack of node-level differential privacy for network data.
method Proposes GRAND, the first mechanism for releasing networks with node-level differential privacy and preserving structural properties.
result GRAND releases networks while ensuring node-level differential privacy and preserving structural properties.
Study shows AD for neural nets with machine-representable numbers can be incorrect.
problem Correctness of AD for neural nets with machine-representable numbers.
method Analyzed two sets of parameters: incorrect and non-differentiable. Proved bounds and conditions for AD correctness.
result AD can be incorrect for machine-representable numbers, but provides a Clarke subderivative on non-differentiable set.
Elvet solves differential equations and variational problems with neural networks.
problem Solving complex differential and variational equations with arbitrary conditions.
method Machine learning, specifically neural networks, to represent and solve equations.
result Elvet can solve a wide range of differential and variational problems.
NeuPDE uses neural networks to model time-dependent data using differential equations.
problem Modeling time-dependent data from dynamic datasets.
method Neural network approach with both shallow multilayer perceptrons and nonlinear differential terms.
result Demonstrated on various dynamical systems, NeuPDE outperforms other methods.
Study reveals properties of local minima in ReLU networks.
problem Understanding the loss landscape of neural networks.
method Theoretical analysis of one-hidden-layer ReLU networks.
result All differentiable local minima are global within certain regions.
DEQGAN uses GANs to solve differential equations without supervision.
problem Solving differential equations with neural networks.
method Generative Adversarial Networks (GANs) to learn the loss function.
result DEQGAN achieves lower mean squared errors and competitive solution accuracy compared to traditional methods.
New framework explains neural network bias in solving differential equations.
problem Understanding and controlling the bias in PINNs for differential equations.
method Deriving an integro-differential equation from PINNs and GPR equivalence.
result PINN predictions are influenced by a kernel term reflecting architecture choices.
Neural sorting network improves sorting accuracy for complex inputs.
problem Sorting complex inputs like multi-digit images.
method Differentiable swap function and permutation-equivariant Transformer network.
result Our method outperforms baseline methods on various sorting benchmarks.
Unified bounds for neural networks incorporating physical laws.
problem Limitations in existing generalization analyses for PINNs and VPINNs.
method Unified framework using Taylor expansion and Koopman-based analysis.
result High-rank networks can generalize well even with differential operators.
DEUs learn nonlinear activation functions from data, reducing network size.
problem Fixed activation functions in neural networks limit performance.
method Differential equation units (DEUs) learn nonlinear activation functions from data.
result DEUs enable neurons to change their activation functions during training.
Proposes a method to train neural networks that solve differential equations faster.
problem Training neural networks that solve differential equations becomes computationally expensive.
method Introduces a differentiable surrogate for numerical solver time cost using higher-order derivatives.
result Trains models that are faster to solve while maintaining nearly the same accuracy.
Differentiable mask prunes deep networks for vision and text.
problem Efficiently compressing deep networks for edge devices.
method Introduces a differentiable mask for sparsity induction.
result Successfully prunes weights, filters, and nodes of convolutional and recurrent networks.
Paper discovers differential equations from data using neural networks and Bayesian methods.
problem Discovering differential equations from datasets using machine learning.
method Integrates neural network-based surrogates with Sparse Bayesian Learning (SBL).
result Proposes a robust model discovery algorithm and a Physics Informed Normalizing Flow (PINF).
Neural networks model financial data with Lévy processes.
problem Forecasting chaotic financial time series with big jumps.
method Lévy-induced stochastic differential equation network approximated by neural networks.
result The method improves prediction accuracy using non-Gaussian Lévy processes.
Physics-informed neural networks approximate diffusion process pdfs efficiently.
problem Approximating the probability density function of diffusion processes.
method Physics-informed neural networks solving Fokker-Planck or integro-differential equations.
result Neural network solutions approximate target solutions for various types of differential equations.
New method improves training stochastic neural networks with tighter guarantees.
problem Training stochastic neural networks with provable guarantees.
method Developed partially-aggregated estimators and reformulated PAC-Bayesian bounds.
result Derives a differentiable objective leading to tighter generalisation guarantees.
VSDN models sporadic time series with neural SDEs.
problem Modeling irregular and sparse time series data.
method Variational Bayesian method and neural SDEs.
result VSDNs outperform state-of-the-art models in prediction and interpolation.
Differentiable NAS method optimizes network architecture and parameters efficiently.
problem Challenging to simultaneously guarantee effectiveness and efficiency in network architecture search.
method Differentiable architecture search with ensemble Gumbel-Softmax estimator.
result End-to-end mechanism for searching network architectures, discovering high-performance architectures efficiently.
Improves discrete latent representations using differentiable approximation bridges.
problem Improving discrete latent representations in neural networks.
method Training with a differentiable approximation bridge (DAB) neural network.
result Improves state-of-the-art performance in various domains.
Paper presents an ADMM-based approach to efficiently integrate quadratic programming layers into neural networks.
problem Integrating quadratic programs into neural networks for optimization.
method An ADMM-based network layer architecture for solving quadratic programs efficiently.
result The ADMM layer is approximately an order of magnitude faster than existing methods for medium scaled problems.
Efficient echo state network with explicit memory performs well on benchmark tasks.
problem Training differentiable neural computers is difficult and time-consuming.
method Echo state network with an explicit memory.
result Echo state network can recognize all regular languages, including those contractive networks cannot.
New method solves PDEs for any initial condition without retraining.
problem Solving PDEs for different initial conditions requires retraining neural solvers.
method Formulate solution as conditional probability distribution.
result Approximates PDE solution for arbitrary initial conditions.
Graph neural networks are extended to continuous-depth models using differential equations.
problem Improving graph neural networks for static and dynamic graph data.
method Formalizing GNNs as GDEs, blending discrete structures with differential equations.
result GDEs offer computational advantages in static settings and improved performance in dynamic settings.
ICON learns differential equation operators from prompts, reducing retraining and improving few-shot learning.
problem Training neural networks to solve differential equations without retraining for new problems.
method In-Context Operator Networks (ICON) that learns operators from prompted data and applies them to new problems.
result ICON can generalize to new operators beyond the training distribution and requires only a few demos.
Normalization layers improve the accuracy of Differentially Private training of deep neural networks.
problem Reduced accuracy in deep neural networks with Differentially Private training.
method Proposed a novel method for integrating batch normalization with Differentially Private Stochastic Gradient Descent (DPSGD) without additional privacy loss.
result Training deeper networks with better utility-privacy trade-off is possible.
A new method to simplify deep neural networks by removing unnecessary parts.
problem Overly complex deep neural networks require significant resource investment for size reduction.
method A fully differentiable sparsification method that optimizes a regularized objective function with stochastic gradient descent.
result The method can learn both the sparsified structure and weights of a network in an end-to-end manner.
Neural differential equations combine deep learning and differential equations for modeling complex systems.
problem Modeling complex systems with high capacity and efficiency.
method Combining neural networks and differential equations, focusing on neural ordinary, controlled, and stochastic differential equations.
result NDEs offer high-capacity function approximation, strong priors, and handle irregular data efficiently.
Hybrid model combines neural networks and fluid dynamics for efficient, generalized simulations.
problem Inefficient and poor generalization of deep learning approximations of fluid dynamics.
method Combines graph neural networks with a differentiable PDE solver inside a neural network.
result Hybrid model generalizes well to new scenarios and outperforms both neural network and traditional methods.
Differentiable resampling improves particle filter performance.
problem Non-differentiability of traditional resampling in particle filters.
method Introduced a neural network resampler (particle transformer) trained with a likelihood-based loss function.
result Learned resampling outperforms traditional methods on synthetic and real-world tasks.
FDNet learns PDEs from data with fast predictions.
problem Discovering complex systems behavior from data.
method Finite difference neural networks (FDNet) to learn PDEs from trajectory data.
result FDNet predicts future behavior with few trainable parameters.
A new differentiable quantization method for neural networks.
problem Training neural networks for resource-constrained devices.
method Differentiable quantization procedure using categorical distributions over a quantization grid.
result The method can train networks that can be effectively discretized without loss of performance.
A new differential entropy estimator for neural networks training.
problem Lack of effective differential entropy estimators for neural network training.
method KNIFE: a fully parameterized, differentiable kernel-based estimator of differential entropy.
result KNIFE effectively estimates differential entropy and improves neural network training.
DANCE optimizes neural network and accelerator design for faster, more efficient DNN execution.
problem Challenges in optimizing neural network and accelerator design for efficient DNN execution.
method Differentiable approach to co-exploration of accelerator and network architecture design.
result Significantly shorter time to achieve superior accuracy and hardware cost metrics.
New findings connect shaped and unshaped neural networks using differential equations.
problem Understanding the behavior of neural networks with different activation scaling methods.
method Deriving differential equation-based asymptotic characterizations for shaped and unshaped neural networks.
result Two types of unshaped networks converge to the same infinite-depth-and-width limit at initialization.
This paper extends geometric study of neural networks to non-differentiable layers and random walks.
problem Understanding the geometric properties of neural networks, especially those with non-differentiable activation functions.
method Singular Riemannian geometry approach to convolutional, residual, and recursive neural networks.
result Illustrated geometric findings with numerical experiments on image classification and thermodynamic problems.
Neural networks trained with Q-learning converge to optimal control.
problem Optimal control of reinforcement learning with neural networks.
method Analysis of a limit differential equation derived from Q-learning.
result The limit differential equation has a unique stationary solution that is optimal.
Differentiable NAS frameworks grow networks wider and deeper, revealing biases in wiring evolution.
problem Understanding the evolution of neural architecture wiring in differentiable NAS methods.
method Unified view on searching algorithms, local cost minimization, empirical and theoretical analyses.
result Implicit inductive biases cause observed searching patterns in differentiable NAS methods.
We develop a scalable method for Bayesian neural networks with stochastic differential equations.
problem Uncertainty quantification in deep neural networks.
method Gradient-based stochastic variational inference in continuous-depth Bayesian neural networks.
result Gradient estimator with zero variance as the approximation improves.
Paper introduces a differentiable regularizer for condition number to improve neural network stability.
problem Maintaining numerical stability in neural networks to ensure reliable and performant models.
method Introduces a novel differentiable regularizer for the condition number of weight matrices.
result Derives a differentiable formula for the gradient of the regularizer, promoting matrices with low condition numbers.