Proposes a new generalization bound for Bayesian deep nets without strict assumptions.
problem Lack of generalization bounds for Bayesian deep nets without strict assumptions.
method Exploits contractivity of Log-Sobolev inequalities to add a loss-gradient norm term to the generalization bound.
result Introduces a new generalization bound for Bayesian deep nets that avoids strict assumptions.
SDE-Net quantifies uncertainty in deep nets using stochastic dynamics.
problem Uncertainty quantification in deep neural networks.
method Viewing DNN transformations as state evolution of a stochastic dynamical system, introducing a Brownian motion term for epistemic uncertainty.
result SDE-Net outperforms existing methods in uncertainty estimation across various tasks.
Deep Bayesian neural nets can use simpler weight approximations without sacrificing performance.
problem The need for complex weight posterior approximations in deep Bayesian neural networks.
method Theoretical and empirical analysis of mean-field variational inference in deep networks.
result Mean-field variational weight posteriors in deep networks can induce similar function-space distributions as complex approximations in shallower networks.
New bounds explain deterministic non-smooth deep nets without large Lipschitz constants.
problem Challenges in explaining generalization of deterministic non-smooth deep nets.
method De-randomized PAC-Bayes margin bounds for deterministic non-convex and non-smooth predictors.
result New bounds avoid large Lipschitz constants, providing generalization guarantees.
New bound relaxes uniform gradient norm assumptions for PAC-Bayesian bounds.
problem Generalization bounds with strict assumptions like uniformly bounded loss.
method Relax uniform bounds assumptions to on-average bounded loss and gradient norm.
result Proposes a new generalization bound with a surrogate of model complexity.
This thesis builds theoretical foundations for deep learning, proving complexity theorems and training algorithms.
problem Rigorous explanation of deep learning's success and mathematical challenges.
method Developed circuit complexity theorems, classification theorems, and training algorithms for neural nets.
result Established the existence of high-complexity neural functions and provided rigorous proofs for training algorithms.
We propose a novel method for closed-form predictive distribution modeling with neural nets. In quantifying prediction uncertainty, we build on Evidential Deep Learning, which has been impactful as being both simple to implement and giving closed-form access to predictive uncertainty. We employ it to model aleatoric un…
Rank-1 BNNs improve efficiency and scalability of Bayesian neural nets.
problem Underfitting and lack of scalability in Bayesian neural networks.
method Propose a rank-1 parameterization of BNNs and use mixture approximate posteriors.
result Rank-1 BNNs achieve state-of-the-art performance across various datasets.
Two models predict net loan losses using Bayesian and frequentist regression.
problem Predicting net loan losses using financial and sociological data.
method Bayesian and frequentist regression analysis.
result Improved understanding of net loan loss relationships.
Bayesian Neural Nets improve model stability and fit.
problem Improving model stability and fit in time series prediction.
method Assign Bayesian Neural Nets to drift and diffusion terms of SDE, infer posterior using SGLD.
result Significantly improved stability and better model fit on benchmarks.
In this paper, we introduce transformations of deep rectifier networks, enabling the conversion of deep rectifier networks into shallow rectifier networks. We subsequently prove that any rectifier net of any depth can be represented by a maximum of a number of functions that can be realized by a shallow network with a …
This paper considers the power of deep neural networks (deep nets for short) in realizing data features. Based on refined covering number estimates, we find that, to realize some complex data features, deep nets can improve the performances of shallow neural networks (shallow nets for short) without requiring additiona…
This paper improves Bayesian neural nets by using local linearization.
problem Underfitting in Bayesian neural networks.
method Local linearization of Bayesian neural networks to create a generalized linear model (GLM) for predictions.
result The GLM predictive resolves common underfitting problems of the Laplace approximation.
Deterministic neural nets have been shown to learn effective predictors on a wide range of machine learning problems. However, as the standard approach is to train the network to minimize a prediction loss, the resultant model remains ignorant to its prediction confidence. Orthogonally to Bayesian neural nets that indi…
We develop a scalable method for Bayesian neural networks with stochastic differential equations.
problem Uncertainty quantification in deep neural networks.
method Gradient-based stochastic variational inference in continuous-depth Bayesian neural networks.
result Gradient estimator with zero variance as the approximation improves.
We describe Bayesian Layers, a module designed for fast experimentation with neural network uncertainty. It extends neural network libraries with drop-in replacements for common layers. This enables composition via a unified abstraction over deterministic and stochastic functions and allows for scalability via the unde…
BCD Nets use variational inference to estimate DAGs with uncertainty.
problem Uncertainty in inferring causal graphs from limited data.
method Variational inference framework for Bayesian DAG estimation.
result BCD Nets outperform maximum-likelihood methods in low data regimes.
Bayesian method improves neural net convergence for character recognition.
problem Improving convergence rate of neural network training algorithms.
method Customization of Kalman filter into Bayesian statistics for initialization of weights.
result Improved convergence rate for backpropagation training algorithm.
DSCF-Net learns deep features for clustering with robustness and locality preservation.
problem Unsupervised deep representation learning for clustering.
method Integrates robust deep concept factorization, deep self-expressive representation, and adaptive locality preserving feature learning.
result Delivers state-of-the-art performance on public databases.
DP-Net uses dynamic programming for efficient deep neural network compression.
problem Efficiently compressing deep neural networks while maintaining accuracy.
method Dynamic Programming for optimal weight quantization and clustering-friendly training.
result Achieves up to 77X compression ratio on Wide ResNet with minimal accuracy loss.
Based on the tree architecture, the objective of this paper is to design deep neural networks with two or more hidden layers (called deep nets) for realization of radial functions so as to enable rotational invariance for near-optimal function approximation in an arbitrarily high dimensional Euclidian space. It is show…
Effective training of deep neural networks suffers from two main issues. The first is that the parameter spaces of these models exhibit pathological curvature. Recent methods address this problem by using adaptive preconditioning for Stochastic Gradient Descent (SGD). These methods improve convergence by adapting to th…
RegPred Net forecasts foreign exchange rates with improved accuracy and interpretability.
problem Multi-step forecasting of Foreign Exchange (FX) rates.
method Bayesian optimization for hyperparameter tuning of a multi-layered regression network.
result RegPred Net significantly outperforms other models in terms of RMSE and correlation metrics.
PNCs balance tractability and expressiveness in probabilistic modeling.
problem Balancing tractability and expressiveness in probabilistic models.
method Introduce probabilistic neural circuits (PNCs) as a mix of Bayesian networks and neural networks.
result PNCs are powerful function approximators.
In this paper, we develop an alternating direction method of multipliers (ADMM) for deep neural networks training with sigmoid-type activation functions (called \textit{sigmoid-ADMM pair}), mainly motivated by the gradient-free nature of ADMM in avoiding the saturation of sigmoid-type activations and the advantages of …
This paper improves ensemble learning for vision tasks by encouraging diversity in predictions.
problem Generating effective ensembles of neural networks for multi-modal data.
method Explicitly optimize a diversity inducing adversarial loss for learning stochastic latent variables.
result Significant improvements in classification accuracy and out-of-distribution detection compared to baselines.
Mode connectivity is a surprising phenomenon in the loss landscape of deep nets. Optima -- at least those discovered by gradient-based optimization -- turn out to be connected by simple paths on which the loss function is almost constant. Often, these paths can be chosen to be piece-wise linear, with as few as two segm…
Method learns invariances in deep nets without human validation.
problem Manual selection of data augmentation parameters is cumbersome.
method Differentiable Laplace approximation for Bayesian model selection.
result Method successfully recovers invariances and improves generalization.
Bayesian ReLU nets fix asymptotic overconfidence with infinite features.
problem Bayesian ReLU nets can be asymptotically overconfident far from training data.
method Extend finite ReLU BNNs with infinite ReLU features via a Gaussian process.
result The resulting model is asymptotically maximally uncertain far from the data.
Proposes HDBEN for heteroscedastic regression with improved sparsity and variance modeling.
problem Violation of constant error variance in high-dimensional regression.
method HDBEN framework using hierarchical Bayesian priors with ℓ1 and ℓ2 penalties. result Achieves posterior concentration, variable selection consistency, and asymptotic normality.
Proposes a new deep learning model for uncertainty quantification and propagation.
problem High-dimensional uncertainty quantification and propagation problems.
method Integrates U-net with Gaussian Gated Linear Network (GGLN) to create GLU-net.
result Less complex architecture with 44% fewer parameters than existing models.
This paper investigates the influence of different acoustic features, audio-events based features and automatic speech translation based lexical features in complex emotion recognition such as curiosity. Pretrained networks, namely, AudioSet Net, VoxCeleb Net and Deep Speech Net trained extensively for different speech…
GIT-Net uses neural networks to approximate PDE operators efficiently.
problem Approximating PDE operators for complex geometries.
method Parametrizes adaptive generalized integral transforms with deep neural networks.
result GIT-Net outperforms existing neural network operators in multiple areas.
We are interested in the development of surrogate models for uncertainty quantification and propagation in problems governed by stochastic PDEs using a deep convolutional encoder-decoder network in a similar fashion to approaches considered in deep learning for image-to-image regression tasks. Since normal neural netwo…
A zoo of deep nets is available these days for almost any given task, and it is increasingly unclear which net to start with when addressing a new task, or which net to use as an initialization for fine-tuning a new model. To address this issue, in this paper, we develop knowledge flow which moves 'knowledge' from mult…
Proposes Π-Nets, polynomial neural networks, for improved representation power.
problem Improving representation power in deep learning models.
method Introduces Π-Nets, a new class of deep polynomial neural networks. result Demonstrates Π-Nets outperform standard DCNNs and achieve state-of-the-art results. Dual U-net models improve multi-channel MRI image reconstruction.
problem Improving MRI image reconstruction from multi-channel data.
method Two-element U-nets (W-nets) in k-space and image domains, evaluated for four configurations.
result Dual domain methods are more advantageous for simultaneous reconstruction of all channels.
X-ray computed tomography (CT) using sparse projection views is a recent approach to reduce the radiation dose. However, due to the insufficient projection views, an analytic reconstruction approach using the filtered back projection (FBP) produces severe streaking artifacts. Recently, deep learning approaches using la…
How well does a classic deep net architecture like AlexNet or VGG19 classify on a standard dataset such as CIFAR-10 when its width --- namely, number of channels in convolutional layers, and number of nodes in fully-connected internal layers --- is allowed to increase to infinity? Such questions have come to the forefr…
G-Net uses deep learning for complex counterfactual outcome prediction.
problem Estimating counterfactual outcomes under dynamic treatment strategies.
method G-Net is a sequential deep learning framework for G-computation.
result G-Net can handle complex temporal data and provide accurate treatment effects.
Efficient method for uncertainty estimation in DNNs with improved accuracy.
problem Vital assessment of deep neural networks' reliability in safety-critical applications.
method Multi-loss sub-ensembles for parallel predictions from similar models differing by their loss.
result Improved accuracy on classification tasks and competitive uncertainty measures.
Yes, they do. This paper provides the first empirical demonstration that deep convolutional models really need to be both deep and convolutional, even when trained with methods such as distillation that allow small or shallow models of high accuracy to be trained. Although previous research showed that shallow feed-for…
MPHD transfers knowledge across different domains for Bayesian optimization.
problem Optimizing functions with unknown or diverse domains.
method MPHD uses neural nets to map domain-specific contexts to GP specifications, enabling transfer learning across heterogeneous search spaces.
result MPHD improves black-box function optimization performance on diverse domains.
Deep learning system diagnoses AVNFH from plain radiographs.
problem Challenging AVNFH diagnosis from plain radiographs.
method Deep convolutional neural networks for end-to-end diagnosis.
result AVN-net achieves state-of-the-art AUC of 0.97 in AVNFH detection.
DS2CF-Net learns hierarchical representations with deep coupled factorization and enriched prior.
problem Learning deep hierarchical representations from data.
method Dual-constrained Deep Semi-Supervised Coupled Factorization Network (DS2CF-Net) with enriched prior.
result DS2CF-Net achieves state-of-the-art performance in representation learning and clustering.
Estimates individualized treatment effects using shared RBF-net neurons.
problem Identifying differential treatment effects based on covariates.
method Non-parametric radial basis function (RBF)-nets with shared hidden neurons in a Bayesian framework.
result Demonstrated through simulations and real data, the method identifies interesting treatment effects.
HAD-Net forecasts glucose levels with insights into insulin and carbs diffusion.
problem Inaccurate predictions in glucose level forecasting without context understanding.
method Hybrid model combining deep learning and physiological models, using recurrent attention network.
result Achieves competitive performance in glucose level forecasting with plausible diffusion insights.
Efficiently calibrates volatility models using Chebyshev Tensors.
problem Calibrating pricing models efficiently.
method Used Chebyshev Tensors to speed up calibration of the rough Bergomi volatility model.
result Chebyshev Tensors can calibrate the rough Bergomi volatility model 40,000 times more efficiently than brute-force methods.