Paper studies the theoretical equivalence between implicit and explicit neural networks in high dimensions.
problem Lack of theoretical analysis of implicit and explicit neural networks.
method Examined high-dimensional implicit neural networks and established their equivalence to explicit networks.
result Equivalence between implicit and explicit neural networks in high dimensions.
This work improves neural network calibration using explicit regularization.
problem Improving predictive uncertainty in neural networks.
method Introducing a probabilistic calibration measure and exploring explicit regularization techniques.
result Explicit regularization improves log-likelihood and predictive uncertainty.
Paper proposes an efficient method to optimize neural networks without backpropagation.
problem Computational inefficiency and scalability issues in neural network optimization.
method Derives explicit solutions to optimize neural networks, reducing computational costs.
result Explicit solutions achieve near-optimality and can discover better optima than backpropagation.
Efficient echo state network with explicit memory performs well on benchmark tasks.
problem Training differentiable neural computers is difficult and time-consuming.
method Echo state network with an explicit memory.
result Echo state network can recognize all regular languages, including those contractive networks cannot.
DEQs and explicit networks are nearly equivalent for Gaussian mixtures.
problem Understanding the equivalence between DEQs and explicit neural networks.
method Random matrix theory and analysis of kernel matrices.
result A shallow explicit network can mimic the kernel of a DEQ.
Efficient neural network optimization reduces costs and improves model performance.
problem High computational costs in optimizing neural networks, especially at scale.
method Introduces self-attentive feed-forward neural units (SAFFU) for efficient optimization.
result Explicit solutions outperform models optimized by backpropagation alone, and further training with backpropagation leads to better optima from smaller data sets.
We propose two neural network based mixture models in this article. The proposed mixture models are explicit in nature. The explicit models have analytical forms with the advantages of computing likelihood and efficiency of generating samples. Computation of likelihood is an important aspect of our models. Expectation-…
New method finds Lie group representations without explicit groups, enabling new neural network architectures.
problem Building neural networks equivariant to arbitrary Lie groups.
method Algorithm to find Lie group representations from Lie algebra structure constants. Self-contained method for constructing Lie group-equivariant neural networks.
result First object-tracking model equivariant to the Poincaré group.
Formula found for neural network error with fixed weights.
problem Understanding error in neural networks with fixed weights.
method Provided an explicit formula for approximation error.
result Explicit formula for neural network error with fixed weights.
Neural networks can separate non-separable data using feature maps.
problem Non-separable data in neural networks.
method Characterization of feedforward neural networks and use of feature maps.
result ReLU neural networks can separate concentric data.
Deep neural-kernel models combine neural networks and kernel machines for scalable large datasets.
problem Combining neural networks and kernel machines for efficient large-scale learning.
method Hybrid neural-kernel architecture using explicit feature mapping and pooling layers.
result The deep neural-kernel models are effective and scalable on benchmark datasets.
GNIs induce a regulariser that penalizes high-frequency components in neural network activations.
problem Understanding the regularizing effect of Gaussian noise injections on neural network activations.
method Deriving the explicit regularizer by marginalizing out injected noise and analyzing its effect in the Fourier domain.
result GNIs induce a regularizer that produces calibrated classifiers with large margins.
New framework monitors neural network training and reveals regularisation mechanisms.
problem Overfitting in neural networks and the need for explicit regularizers.
method Model Gradient Similarity (MGS) framework to measure and monitor regularisation.
result Explicit regularizers increase Model Gradient Similarity (MGS).
A new explicit scheme calculates XVA adjustments using neural networks and conditional expectations.
problem Calculating cross valuation adjustments (XVA) in realistic financial scenarios.
method Simulation/regression scheme for BSDEs, using neural networks and quantile regressions.
result The scheme outperforms Picard iterations in high-dimensional and hybrid market risks.
Two methods for pricing swing contracts using neural networks or explicit functions.
problem Evaluating optimal energy purchases in swing contracts with firm constraints.
method Two approaches: explicit parametric function and neural network approximation.
result Neural network approach provides better prices in shorter computation time.
In recent years, deep neural networks have yielded state-of-the-art performance on several tasks. Although some recent works have focused on combining deep learning with recommendation, we highlight three issues of existing models. First, these models cannot work on both explicit and implicit feedback, since the networ…
Paper presents a method to summarize HMC samples for neural networks, providing meaningful uncertainty estimates.
problem Lack of interpretable summary statistics for HMC samples in neural networks due to permutation symmetry.
method Introducing a transpositions metric to quantify permutations and using rebasin method to summarize HMC samples.
result Compact representation of HMC samples provides meaningful uncertainty estimates for each weight in a neural network.
Self-referential meta learning avoids explicit optimization by modifying itself.
problem Dependency on human engineering in meta learning algorithms.
method Investigates self-referential meta learning systems that modify themselves without explicit optimization.
result Self-referential neural networks can improve their own modifications without explicit optimization.
Noise injection before gradient steps helps in regularization for neural networks.
problem Improving generalization in overparametrized neural networks.
method Injecting small noise perturbations before computing gradient steps, especially in layer-wise fashion.
result Small noise perturbations can explicitly regularize neural networks without variance explosion.
Nonparametric neural-network estimation of current-status data
problem Estimation of conditional cumulative distribution function with current-status data
method Neural-network sieve maximum likelihood estimator
result Explicit convergence rate for Hölder smoothness
In this paper we show how to augment classical methods for inverse problems with artificial neural networks. The neural network acts as a prior for the coefficient to be estimated from noisy data. Neural networks are global, smooth function approximators and as such they do not require explicit regularization of the er…
Semi-supervised learning algorithms typically construct a weighted graph of data points to represent a manifold. However, an explicit graph representation is problematic for neural networks operating in the online setting. Here, we propose a feed-forward neural network capable of semi-supervised learning on manifolds w…
Bounds neural network output distribution to Gaussian for random initialization.
problem Quantifying the distribution of randomly initialized deep neural networks.
method Quantitative Gaussian approximation using quadratic Wasserstein distance.
result Explicit inequalities show how network sizes affect Gaussian behavior.
Simplifies neural network models by explicitly enforcing constraints in Cartesian coordinates.
problem Learning dynamics of complex systems efficiently and accurately.
method Embedding systems into Cartesian coordinates and using Lagrange multipliers to enforce constraints.
result Explicitly enforcing constraints leads to a 100x improvement in accuracy and data efficiency.
Optimizes multi-period portfolios with tail-risk constraints using neural networks.
problem Maximizing expected return while managing tail-risk constraints over multiple periods.
method Recurrent neural network approach to approximate optimal policy.
result Validated in financial and insurance models, capturing long-term risk dynamics.
EPFGNN models graph connections for better node classification.
problem Graph node classification issues due to feature aggregation.
method EPFGNN models graph as a Markov Random Field with explicit pairwise factors and a GNN backbone.
result EPFGNN improves semi-supervised node classification performance.
Neural dynamical systems are dynamical systems that are described at least in part by neural networks. The class of continuous-time neural dynamical systems must, however, be numerically integrated for simulation and learning. Here, we present a compact neural circuit for two common numerical integrators: the explicit …
We prove that the global minimum of the backpropagation (BP) training problem of neural networks with an arbitrary nonlinear activation is given by the ridgelet transform. A series of computational experiments show that there exists an interesting similarity between the scatter plot of hidden parameters in a shallow ne…
Study on size and depth of neural networks for approximating benign functions, showing barriers and explicit results.
problem Understanding how size and depth of neural networks affect their ability to approximate benign functions.
method Analyzing ReLU networks for benign functions, proving barriers and explicit results.
result Explicit benign functions that cannot be approximated by networks of certain sizes or depths, showing barriers to size and depth separation.
Heavy-tailed distributions emerge in SGD's parameter evolution.
problem Understanding heavy-tailed distributions in SGD parameter evolution.
method Continuous diffusion approximation of SGD (homogenized SGD) analysis.
result Explicit upper and lower bounds on tail-index of homogenized SGD.
The paper studies Lipschitz bounds for integral kernels under differentiability assumptions.
problem Understanding the Lipschitz continuity of feature maps associated with integral kernels.
method Analyzes differentiability assumptions to derive explicit formulas for Lipschitz constants and conditions for non-Lipschitz continuity.
result Explicit formulas and conditions for Lipschitz continuity of feature maps associated with various kernels.
Many efforts have been made to use various forms of domain knowledge in malware detection. Currently there exist two common approaches to malware detection without domain knowledge, namely byte n-grams and strings. In this work we explore the feasibility of applying neural networks to malware detection and feature lear…
Study the expressivity and training complexity of polynomial neural networks.
problem Understanding the expressivity and training complexity of polynomial neural networks.
method Use algebraic geometry to describe neuromanifolds and neurovarieties, analyzing their dimension and learning degree.
result Characterized the dimension and learning degree of neuromanifolds, providing geometric and complexity measures.
Study Gaussian-process limits of neural networks using tensor programs.
problem Understanding the behavior of neural networks as they approach infinite width.
method Quantitative analysis through tensor programs and Wasserstein distance.
result Explicit finite-width error bounds, showing convergence to Gaussian-process limits.
Over-parameterized neural networks generalize well in practice without any explicit regularization. Although it has not been proven yet, empirical evidence suggests that implicit regularization plays a crucial role in deep learning and prevents the network from overfitting. In this work, we introduce the gradient gap d…
Not all neural network architectures are created equal, some perform much better than others for certain tasks. But how important are the weight parameters of a neural network compared to its architecture? In this work, we question to what extent neural network architectures alone, without learning any weight parameter…
This paper uncovers the low-rank structure of neural network Hessians.
problem Understanding the structure of Hessians in neural networks.
method Proposes a decoupling conjecture to decompose layer-wise Hessians into Kronecker products of smaller matrices.
result Proves the structure of top eigenspaces in 2-layer networks and shows high overlap in top eigenvectors across different models.
The paper proves neural networks' consistency and optimal convergence rates for various function classes.
problem Proving neural networks' consistency and optimal convergence rates for diverse function classes.
method Analyzes wide and deep ReLU neural networks trained on logistic loss and Kolmogorov-Donoho optimal function classes.
result Proves universal consistency and minimax optimal convergence rates for neural networks.
Proposes methods to add constraints to neural networks to improve stability and generalization.
problem Improving stability and generalization of neural networks.
method Constraint-based regularization using stochastic gradient Langevin dynamics.
result Constraints help stabilize and improve the robustness of deep neural networks.
It remains a puzzle that why deep neural networks (DNNs), with more parameters than samples, often generalize well. An attempt of understanding this puzzle is to discover implicit biases underlying the training process of DNNs, such as the Frequency Principle (F-Principle), i.e., DNNs often fit target functions from lo…
Neural networks have been shown to be vulnerable against minor adversarial perturbations of their inputs, especially for high dimensional data under ℓ∞ attacks. To combat this problem, techniques like adversarial training have been employed to obtain models which are robust on the training set. However, the …
Study shows neural networks trained with GD converge to Gaussian processes with polynomial decay.
problem Understanding convergence of neural networks to Gaussian processes during training.
method Explicit upper bounds on quadratic Wasserstein distance between trained networks and Gaussian approximations.
result Polynomial decay of approximation error with network width and training time.
In this paper, we show the implementation of deep neural networks applied in process control. In our approach, we based the training of the neural network on model predictive control. Model predictive control is popular for its ability to be tuned by the weighting matrices and by the fact that it respects the constrain…
Geodesics found in deep linear networks.
problem Finding shortest paths in deep neural networks.
method Derived ODEs and explicit solutions for geodesics.
result Horizontal straight lines are geodesics in invariant manifold.
Graph Neural Networks improve El Niño forecasts.
problem Improving seasonal forecasting models for ENSO.
method Application of spatiotemporal Graph Neural Networks.
result Preliminary results outperform state-of-the-art systems for 1 and 3-month projections.
Wide neural networks can learn complex functions like gravitational force law.
problem Learning complex functions like gravitational force law with neural networks.
method Extending theoretical bounds to analytic functions on the sphere using SGD and ReLU networks.
result Wide ReLU networks can learn analytic functions efficiently with proportional number of samples.
Method trains a debiased model from a biased one by focusing on samples that contradict the bias.
problem Training neural networks can lead to biased predictions due to spurious correlations.
method Train a pair of neural networks, intentionally biasing one and debiasing the other by focusing on contradictory samples.
result Our method significantly improves training against various types of biases, sometimes outperforming explicit supervision methods.
Researchers analyze a new neural network training method.
problem Training robust configurations in discrete weight neural networks.
method Replicated simulated annealing combining physics and classical simulated annealing.
result Explicit criteria for algorithm convergence and successful sampling.