Bayesian Neural Nets improve model stability and fit.
problem Improving model stability and fit in time series prediction.
method Assign Bayesian Neural Nets to drift and diffusion terms of SDE, infer posterior using SGLD.
result Significantly improved stability and better model fit on benchmarks.
Bayesian method improves neural net convergence for character recognition.
problem Improving convergence rate of neural network training algorithms.
method Customization of Kalman filter into Bayesian statistics for initialization of weights.
result Improved convergence rate for backpropagation training algorithm.
SDE-Net quantifies uncertainty in deep nets using stochastic dynamics.
problem Uncertainty quantification in deep neural networks.
method Viewing DNN transformations as state evolution of a stochastic dynamical system, introducing a Brownian motion term for epistemic uncertainty.
result SDE-Net outperforms existing methods in uncertainty estimation across various tasks.
Bayesian Layers adds uncertainty to neural networks, enabling faster experimentation and scalability.
problem Enabling neural networks to quantify uncertainty in predictions.
method Drop-in replacements for common layers, capturing uncertainty over weights, activations, etc.
result Bayesian Layers can fit large models like 5-billion parameter Bayesian Transformers.
A novel method for Bayesian predictive distribution modeling with neural nets.
problem Modeling and quantifying prediction uncertainty in neural networks.
method Evidential Deep Learning, Bayesian Neural Net, progressive moment matching, PAC bound.
result Improves model fit and uncertainty quantification on various benchmarks.
Neural surrogate predicts SPN rates from token trajectories.
problem Challenging parameter estimation in SPNs with covariates.
method 1D Convolutional Residual Network trained on Gillespie-simulated SPN realizations.
result Surrogate predicts rate-function coefficients with RMSE = 0.043.
Deep Bayesian neural nets can use simpler weight approximations without sacrificing performance.
problem The need for complex weight posterior approximations in deep Bayesian neural networks.
method Theoretical and empirical analysis of mean-field variational inference in deep networks.
result Mean-field variational weight posteriors in deep networks can induce similar function-space distributions as complex approximations in shallower networks.
Bayesian ReLU nets fix asymptotic overconfidence with infinite features.
problem Bayesian ReLU nets can be asymptotically overconfident far from training data.
method Extend finite ReLU BNNs with infinite ReLU features via a Gaussian process.
result The resulting model is asymptotically maximally uncertain far from the data.
Deterministic neural nets have been shown to learn effective predictors on a wide range of machine learning problems. However, as the standard approach is to train the network to minimize a prediction loss, the resultant model remains ignorant to its prediction confidence. Orthogonally to Bayesian neural nets that indi…
R2D2-Net improves Bayesian neural networks by preventing over-shrinkage of important weights.
problem Bayesian neural networks struggle with choosing appropriate priors, leading to over-shrinkage or poor predictive performance.
method Proposes R2D2-Net with an R^2-induced Dirichlet Decomposition prior and variational Gibbs inference algorithm.
result R2D2-Net effectively shrinks irrelevant coefficients while preventing key features from over-shrinkage.
This paper improves Bayesian neural nets by using local linearization.
problem Underfitting in Bayesian neural networks.
method Local linearization of Bayesian neural networks to create a generalized linear model (GLM) for predictions.
result The GLM predictive resolves common underfitting problems of the Laplace approximation.
MPHD transfers knowledge across different domains for Bayesian optimization.
problem Optimizing functions with unknown or diverse domains.
method MPHD uses neural nets to map domain-specific contexts to GP specifications, enabling transfer learning across heterogeneous search spaces.
result MPHD improves black-box function optimization performance on diverse domains.
We develop a scalable method for Bayesian neural networks with stochastic differential equations.
problem Uncertainty quantification in deep neural networks.
method Gradient-based stochastic variational inference in continuous-depth Bayesian neural networks.
result Gradient estimator with zero variance as the approximation improves.
We propose a new Bayesian Neural Net formulation that affords variational inference for which the evidence lower bound is analytically tractable subject to a tight approximation. We achieve this tractability by (i) decomposing ReLU nonlinearities into the product of an identity and a Heaviside step function, (ii) intro…
Improves model accuracy for neural nets in stochastic dynamics with partial prior knowledge.
problem Stability and accuracy in neural nets modeling stochastic dynamics with many parameters.
method Three steps: probabilistic weights, partial knowledge incorporation, and PAC-Bayesian training.
result Improved model fit with partial and noisy prior knowledge.
Proposes a new generalization bound for Bayesian deep nets without strict assumptions.
problem Lack of generalization bounds for Bayesian deep nets without strict assumptions.
method Exploits contractivity of Log-Sobolev inequalities to add a loss-gradient norm term to the generalization bound.
result Introduces a new generalization bound for Bayesian deep nets that avoids strict assumptions.
New bound relaxes uniform gradient norm assumptions for PAC-Bayesian bounds.
problem Generalization bounds with strict assumptions like uniformly bounded loss.
method Relax uniform bounds assumptions to on-average bounded loss and gradient norm.
result Proposes a new generalization bound with a surrogate of model complexity.
Two models predict net loan losses using Bayesian and frequentist regression.
problem Predicting net loan losses using financial and sociological data.
method Bayesian and frequentist regression analysis.
result Improved understanding of net loan loss relationships.
Rank-1 BNNs improve efficiency and scalability of Bayesian neural nets.
problem Underfitting and lack of scalability in Bayesian neural networks.
method Propose a rank-1 parameterization of BNNs and use mixture approximate posteriors.
result Rank-1 BNNs achieve state-of-the-art performance across various datasets.
PNCs balance tractability and expressiveness in probabilistic modeling.
problem Balancing tractability and expressiveness in probabilistic models.
method Introduce probabilistic neural circuits (PNCs) as a mix of Bayesian networks and neural networks.
result PNCs are powerful function approximators.
This paper improves ensemble learning for vision tasks by encouraging diversity in predictions.
problem Generating effective ensembles of neural networks for multi-modal data.
method Explicitly optimize a diversity inducing adversarial loss for learning stochastic latent variables.
result Significant improvements in classification accuracy and out-of-distribution detection compared to baselines.
Large multilayer neural networks trained with backpropagation have recently achieved state-of-the-art results in a wide range of problems. However, using backprop for neural net learning still has some disadvantages, e.g., having to tune a large number of hyperparameters to the data, lack of calibrated probabilistic pr…
This thesis builds theoretical foundations for deep learning, proving complexity theorems and training algorithms.
problem Rigorous explanation of deep learning's success and mathematical challenges.
method Developed circuit complexity theorems, classification theorems, and training algorithms for neural nets.
result Established the existence of high-complexity neural functions and provided rigorous proofs for training algorithms.
Hierarchical Bayesian networks and neural networks with stochastic hidden units are commonly perceived as two separate types of models. We show that either of these types of models can often be transformed into an instance of the other, by switching between centered and differentiable non-centered parameterizations of …
TyXe enables flexible Bayesian neural networks in Pytorch.
problem Uncertainty estimation in neural networks.
method Separates architecture, prior, inference, and likelihood specification; modular choices for priors, guides, and inference techniques.
result Minimal modifications to existing code for Bayesian neural networks.
Effective training of deep neural networks suffers from two main issues. The first is that the parameter spaces of these models exhibit pathological curvature. Recent methods address this problem by using adaptive preconditioning for Stochastic Gradient Descent (SGD). These methods improve convergence by adapting to th…
Bayesian neural networks predict stress fields and uncertainty in materials.
problem Uncertainty in stress field predictions for complex materials.
method Modified Bayesian U-net architecture with three inference algorithms.
result High accuracy predictions and interpretable uncertainty estimates.
BCD Nets use variational inference to estimate DAGs with uncertainty.
problem Uncertainty in inferring causal graphs from limited data.
method Variational inference framework for Bayesian DAG estimation.
result BCD Nets outperform maximum-likelihood methods in low data regimes.
An important class of distance metrics proposed for training generative adversarial networks (GANs) is the integral probability metric (IPM), in which the neural net distance captures the practical GAN training via two neural networks. This paper investigates the minimax estimation problem of the neural net distance ba…
We present a new algorithm to train a robust neural network against adversarial attacks. Our algorithm is motivated by the following two ideas. First, although recent work has demonstrated that fusing randomness can improve the robustness of neural networks (Liu 2017), we noticed that adding noise blindly to all the la…
New method solves high-dimensional Bayesian inverse problems efficiently.
problem Efficiently solving high-dimensional Bayesian inverse problems with limited data.
method Physics-informed Neural Operators with RealNVP architecture for invertibility and differentiability.
result Accurate approximations of the full posterior without additional forward solves or sampling.
This research explores the dynamics of linearised neural nets, revealing distinct learning phases and layer growth rates.
problem Understanding the fundamental mechanics of neural nets and their learning dynamics.
method Derivation of properties of learning dynamics in general multi-layer linear neural nets, including orthogonal networks.
result Linear multi-layer neural nets exhibit distinct phases of learning with different layer growth rates, and nonlinearity affects these dynamics.
The study forecasts, reconstructs, and selects features of ocean waves using neural networks.
problem Forecasting, reconstructing, and feature selection of ocean waves.
method Recurrent and sequence-to-sequence neural networks, Bayesian hyperparameter optimization, Elastic Net method.
result Proposed methods outperform alternatives in significant wave height reconstruction.
A scalable method for Bayesian inference in large linear models.
problem High computational cost in Bayesian linear models for large networks.
method Sample-based inference and g-prior for hyperparameter selection.
result Linearised neural network inference on large datasets (ResNet-18, ResNet-50, U-Net).
Stochastic neural net weights are used in a variety of contexts, including regularization, Bayesian neural nets, exploration in reinforcement learning, and evolution strategies. Unfortunately, due to the large number of weights, all the examples in a mini-batch typically share the same weight perturbation, thereby limi…
Kernel methods outperform neural nets in operator learning tasks.
problem Learning operators between Banach spaces from partial observations.
method Kernel-based framework with a priori error analysis and numerical comparisons.
result Kernel methods are competitive with neural nets in cost-accuracy trade-off.
In this paper, we introduce transformations of deep rectifier networks, enabling the conversion of deep rectifier networks into shallow rectifier networks. We subsequently prove that any rectifier net of any depth can be represented by a maximum of a number of functions that can be realized by a shallow network with a …
New training method for neural nets using multilevel entropic regularization.
problem Training efficiency and generalization bounds for neural nets.
method Multilevel relative entropy, chaining mutual information, Gibbs posterior distribution.
result Proves the Gibbs posterior achieves the unique minimum of the empirical risk minimization problem.
GIT-Net uses neural networks to approximate PDE operators efficiently.
problem Approximating PDE operators for complex geometries.
method Parametrizes adaptive generalized integral transforms with deep neural networks.
result GIT-Net outperforms existing neural network operators in multiple areas.
Proposes HDBEN for heteroscedastic regression with improved sparsity and variance modeling.
problem Violation of constant error variance in high-dimensional regression.
method HDBEN framework using hierarchical Bayesian priors with ℓ1 and ℓ2 penalties. result Achieves posterior concentration, variable selection consistency, and asymptotic normality.
The vast majority of current machine learning algorithms are designed to predict single responses or a vector of responses, yet many types of response are more naturally organized as matrices or higher-order tensor objects where characteristics are shared across modes. We present a new machine learning algorithm BaTFLE…
DP-Net uses dynamic programming for efficient deep neural network compression.
problem Efficiently compressing deep neural networks while maintaining accuracy.
method Dynamic Programming for optimal weight quantization and clustering-friendly training.
result Achieves up to 77X compression ratio on Wide ResNet with minimal accuracy loss.
Deep nets improve function approximation and learning in high dimensions.
problem Designing neural networks for rotation-invariant function approximation.
method Developed deep neural networks with multiple hidden layers for radial function approximation.
result Deep nets achieve near-optimal function approximation and learning rates not possible by shallow nets.
Deep nets outperform shallow nets in complex feature realization.
problem Realizing complex data features with deep nets.
method Refined covering number estimates and analysis of approximation rates.
result Deep nets can improve performance without additional capacity costs for complex features.
Study measures impact of data and neural net similarity on transferability in restaurant sales data.
problem Identify indicators for successful transferability of neural nets across different data sets.
method Empirical study on sales data from six restaurants, calculating indicators based on data and neural net similarities.
result Negative correlations between transferability and indicators, allowing better model performance and fewer transfers.
RadiX-Net generates diverse sparse neural topologies.
problem Sparse neural networks require more efficient storage and training.
method Deterministically generates RadiX-Nets from sparse topologies.
result RadiX-Nets can train to the same precision as dense DNNs at lower cost.
Learning to Optimize is a recently proposed framework for learning optimization algorithms using reinforcement learning. In this paper, we explore learning an optimization algorithm for training shallow neural nets. Such high-dimensional stochastic optimization problems present interesting challenges for existing reinf…
DNF-Net tackles tabular data challenges with neural architecture.
problem Handling tabular data efficiently using neural networks.
method DNF-Net uses a neural architecture with inductive bias corresponding to logical Boolean formulas in disjunctive normal form over affine soft-threshold decision terms.
result DNF-Net significantly outperforms fully connected networks on tabular data.