Randomly trained neural networks can generalize well if there's a simpler underlying teacher model.
problem Why randomly trained neural networks generalize well despite interpolating training data.
method Examined a random neural network that interpolates training data and showed it generalizes well if there's a simpler underlying teacher model.
result Randomly trained neural networks can generalize well if there's a simpler underlying teacher model.
Bayesian neural networks with functional priors improve surrogate modeling in mechanics.
problem Challenges in integrating prior knowledge and quantifying uncertainties in high-dimensional NN parameter spaces.
method Anchored ensembling to integrate a priori information and learn low-rank correlations between NN parameters.
result Effective transfer of knowledge between function-space and parameter-space priors improves surrogate model accuracy and uncertainty estimation.
Learning probability distributions on the weights of neural networks (NNs) has recently proven beneficial in many applications. Bayesian methods, such as Stein variational gradient descent (SVGD), offer an elegant framework to reason about NN model uncertainty. However, by assuming independent Gaussian priors for the i…
We propose a novel approach for nonlinear regression using a two-layer neural network (NN) model structure with sparsity-favoring hierarchical priors on the network weights. We present an expectation propagation (EP) approach for approximate integration over the posterior distribution of the weights, the hierarchical s…
Improved compressed sensing using a generator that learns from measurements.
problem Signal recovery accuracy in compressed sensing.
method Proposes a framework that uses measurement-conditional generative models to refine signal estimation.
result Uniformly superior performance with up to an order of magnitude reduction in reconstruction error.
PAC-Bayesian bounds improve understanding of K-NN classifier performance.
problem Improving the understanding of K-NN classifier's generalization error.
method PAC-Bayesian analysis applied to K-NN classifier in kernel space.
result PAC-Bayesian bounds provide a function of the number of redundant training examples.
Ensembles of neural networks (NNs) have long been used to estimate predictive uncertainty; a small number of NNs are trained from different initialisations and sometimes on differing versions of the dataset. The variance of the ensemble's predictions is interpreted as its epistemic uncertainty. The appeal of ensembling…
We show that a simple modification of the 1-nearest neighbor classifier yields a strongly Bayes consistent learner. Prior to this work, the only strongly Bayes consistent proximity-based method was the k-nearest neighbor classifier, for k growing appropriately with sample size. We will argue that a margin-regularized 1…
Deterministic neural networks (NNs) are increasingly being deployed in safety critical domains, where calibrated, robust, and efficient measures of uncertainty are crucial. In this paper, we propose a novel method for training non-Bayesian NNs to estimate a continuous target as well as its associated evidence in order …
This paper analyzes deep Stable neural networks, showing convergence rates under different growth settings.
problem Analyzing the behavior of deep Stable neural networks as width increases.
method Large-width asymptotic analysis and convergence rates for fully connected feed-forward deep Stable NNs.
result The rescaled deep Stable NN converges weakly to a Stable SP under joint growth, with sup-norm convergence rates established.
This work establishes the equivalence between neural networks and support vector machines.
problem Establishing the equivalence between neural networks and support vector machines.
method Proposed a method to establish the equivalence between infinitely wide neural networks trained by soft margin loss and standard soft margin SVMs with NTK trained by subgradient descent.
result The equivalence between NN and SVM is established, enabling practical applications such as non-vacuous generalization bounds and robustness certificates.
Bayesian neural networks' performance varies with prior choice, affecting their ability to identify unknowns.
problem The impact of prior choice on Bayesian neural networks' ability to identify unknowns.
method Evaluation of different prior distributions on classification tasks using BNNs and NNs with Monte Carlo dropout.
result Prior choice significantly impacts BNNs' ability to identify unknowns, affecting true and false positive rates.
For sensitive problems, such as medical imaging or fraud detection, Neural Network (NN) adoption has been slow due to concerns about their reliability, leading to a number of algorithms for explaining their decisions. NNs have also been found vulnerable to a class of imperceptible attacks, called adversarial examples, …
Adaptive NN method improves matrix completion for non-smooth data.
problem Matrix completion with non-smooth non-linear functions under high missingness.
method Two-sided nearest neighbors with \Holder function class non-linearity.
result NN error rate matches oracle's for latent factors, non-trivial for wide range of missingness.
Understanding the uncertainty of a neural network's (NN) predictions is essential for many purposes. The Bayesian framework provides a principled approach to this, however applying it to NNs is challenging due to large numbers of parameters and data. Ensembling NNs provides an easily implementable, scalable method for …
A new method for Bayesian neural networks using probabilistic backpropagation.
problem Approximating posterior distributions in Bayesian neural networks.
method Variational Expectation Propagation (VEP) with probabilistic backpropagation.
result Efficient algorithm for approximate integration over posterior distributions.
Can engineering neural networks be approached in a disciplined way similar to how engineers build software for civil aircraft? We present nn-dependability-kit, an open-source toolbox to support safety engineering of neural networks for autonomous driving systems. The rationale behind nn-dependability-kit is to consider…
Improves neural network performance by enriching training dataset.
problem Achieving worst-case performance guarantees in neural networks.
method Adapting training dataset during training to reduce worst-case violations.
result Improved worst-case performance guarantees in neural networks.
This work approximates finite neural networks with Gaussian processes, providing error bounds and applications in prior selection.
problem Approximating finite neural networks with Gaussian processes for error bounds and uncertainty quantification.
method Iterative approximation of neural network layers as mixtures of Gaussian processes, using optimal transport and Gaussian processes.
result The ability to return a mixture of Gaussian processes that is ε-close to the neural network at a finite set of input points.
Tabular data analysis shows GBDTs and NNs perform similarly on most datasets.
problem Determining when NNs outperform GBDTs on tabular data.
method Large-scale comparison of 19 algorithms across 176 datasets.
result For many datasets, GBDTs and NNs perform similarly, with hyperparameter tuning often more important than algorithm choice.
Better uncertainty estimates for neural networks using Gaussian process priors.
problem Poor uncertainty estimates in neural networks, especially on out-of-distribution data.
method Characterize the function-space prior of an ensemble of infinitely-wide neural networks as a Gaussian process and use it to build a probabilistic model.
result The approach improves calibration of neural networks, especially under distributional shift.
Improved nearest neighbors for missing data in latent factor models.
problem Estimation with missing data in latent factor models.
method Doubly robust nearest neighbors (NN) that considers both row and column neighbors.
result Provides a consistent estimate and near-quadratic improvement in error.
In this paper we present an algorithm for rapid Bayesian analysis that combines the benefits of nested sampling and artificial neural networks. The blind accelerated multimodal Bayesian inference (BAMBI) algorithm implements the MultiNest package for nested sampling as well as the training of an artificial neural netwo…
Introduces DF framework for sampling decisions from target distributions.
problem Sampling from target distributions with additional guidance.
method DF framework based on MDP and Path Integral Diffusion.
result DF enhances guided sampling across various applications.
SPIDER uses deep neural networks for streaming tensor factorization.
problem Lack of effective approach for deep tensor factorization of streaming data.
method Bayesian neural networks with spike-and-slab prior, Taylor expansions, moment matching, and EPI framework.
result Effective incremental updates for latent factors and NN weights.
The study analyzes a three-layer neural network's training dynamics using a functional-space mean-field theory.
problem Understanding the training dynamics of partially-trained three-layer neural networks.
method Generalized mean-field theory to functional spaces, proving convergence and feature learning.
result The training loss of the model decays to zero at a linear rate in the L2 regression setting. Unified framework identifies nonlinear systems using characteristic curves and neural networks.
problem Balancing interpretability and flexibility in nonlinear system identification.
method Combines differential equation structure with neural networks, using characteristic curves as modular components.
result NN-CC approach outperforms other methods in complex nonlinear systems.
New method extracts aleatoric and epistemic uncertainties from regression-based neural networks.
problem Need for principled uncertainty reasoning in machine learning systems.
method Learning evidential distributions for aleatoric and epistemic uncertainties.
result Allows for the simultaneous extraction of both uncertainties without sampling or out-of-distribution data.
BaNk-UCB tackles batched nonparametric bandits with k-NN regression and UCB.
problem Sequential decision-making with limited online feedback in domains like medicine and marketing.
method Combines k-NN regression with UCB principle for fully nonparametric, adaptive, and simple implementation.
result Near-optimal regret guarantees under Lipschitz smoothness and margin assumptions, with minimax-optimal rates.
New algorithm learns from partial labels in general scenarios.
problem Learning from partial labels when each instance has a bag of labels.
method Adaptive nearest-neighbors algorithm PL A-kNN. result PL A-kNN outperforms state-of-the-art methods in general PLL scenarios. AWNN improves matrix completion by adaptively weighting nearest neighbors.
problem Matrix completion with optimal nearest neighbor weights and radii selection.
method Adaptively weighted nearest neighbor method for matrix completion.
result Theoretical guarantees and synthetic experiments support the effectiveness of AWNN.
This paper characterizes how randomized neural networks generalize well in multi-dimensional tasks.
problem Understanding the generalization of randomized neural networks in multi-dimensional tasks.
method Characterizes RSNs as an IGAM formalized by an optimization problem with a regularization functional and loss.
result RSNs generalize well in multi-dimensional tasks, akin to spline regression under certain conditions.
Enhances deep kernel learning with stochastic latent variables for better model regularization.
problem Weak model regularization in deep kernel learning, especially on small datasets.
method Introduces DLVKL model with stochastic latent variables, NSDE for expressive posterior, and hybrid prior.
result DLVKL-NSDE outperforms existing deep GPs on large datasets.
We propose a simple approach which, given distributed computing resources, can nearly achieve the accuracy of k-NN prediction, while matching (or improving) the faster prediction time of 1-NN. The approach consists of aggregating denoised 1-NN predictors over a small number of distributed subsamples. We show, bot…
Two new algorithms improve performance in adversarial bandits with unbounded losses.
problem Adversarial Multi-Armed Bandits with unbounded losses.
method Developed UMAB-NN and UMAB-G for non-negative and general unbounded losses respectively.
result UMAB-NN achieves the first adaptive and scale-free regret bound for non-negative unbounded losses.
Study bridges GARCH and NN models for volatility forecasting.
problem Lack of interaction between GARCH and NN approaches for volatility forecasting.
method Established equivalence between GARCH and NN models, introduced GARCH-NN approach.
result GARCH-NN approach enhances volatility forecasting compared to standalone models.
A fast method for LOOCV in k-NN regression reduces computation time.
problem Efficient computation of LOOCV for k-NN regression.
method Identical LOOCV estimate to (k+1)-NN MSE on training data.
result LOOCV computation can be done with (k+1)-NN regression once.
Adds layers to NNs to protect them from reverse engineering.
problem Extracting the underlying model of a Neural Network.
method Introducing parasitic layers that approximate a noisy identity mapping with a Convolutional NN.
result The protected NN's predictions remain mostly unchanged while making reverse-engineering more complex.
Artificial neural networks (NN) are instrumental in realizing highly-automated driving functionality. An overarching challenge is to identify best safety engineering practices for NN and other learning-enabled components. In particular, there is an urgent need for an adequate set of metrics for measuring all-important …
A neural network (NN) is a parameterised function that can be tuned via gradient descent to approximate a labelled collection of data with high precision. A Gaussian process (GP), on the other hand, is a probabilistic model that defines a distribution over possible functions, and is updated in light of data via the rul…
Prototype rules simplify multiclass classification in metric spaces, achieving consistency and reduced complexity.
problem Multiclass classification in metric spaces, focusing on universal consistency and convergence rates.
method Novel Proto-NN and hybrid rules for multiclass classification in metric spaces, analyzing convergence rates.
result Proto-NN is universally consistent and simpler to implement, with similar computational advantages.
Neural Networks (NN) have recently emerged as backbone of several sensitive applications like automobile, medical image, security, etc. NNs inherently offer Partial Fault Tolerance (PFT) in their architecture; however, the biased PFT of NNs can lead to severe consequences in applications like cryptography and security …
An ensemble of neural networks is known to be more robust and accurate than an individual network, however usually with linearly-increased cost in both training and testing. In this work, we propose a two-stage method to learn Sparse Structured Ensembles (SSEs) for neural networks. In the first stage, we run SG-MCMC wi…
Bagging reduces variance in LID estimation by preserving local distribution of NN distances.
problem High estimation variance from limited data in small neighborhoods.
method Subbagging to preserve local distribution of NN distances, combined with ensemble size.
result Bagging significantly reduces variance and MSE in LID estimation.
In the k-nearest neighborhood model (k-NN), we are given a set of points P, and we shall answer queries q by returning the k nearest neighbors of q in P according to some metric. This concept is crucial in many areas of data analysis and data processing, e.g., computer vision, document retrieval and machi…
We focus on estimating \emph{a priori} generalization error of two-layer ReLU neural networks (NNs) trained by mean squared error, which only depends on initial parameters and the target function, through the following research line. We first estimate \emph{a priori} generalization error of finite-width two-layer ReLU …
Machine Learning (ML) is making a strong resurgence in tune with the massive generation of unstructured data which in turn requires massive computational resources. Due to the inherently compute- and power-intensive structure of Neural Networks (NNs), hardware accelerators emerge as a promising solution. However, with …
This paper studies large-width asymptotics for ReLU neural networks with α-Stable initializations.
problem Characterizing the large-width behavior of ReLU neural networks with α-Stable initializations.
method Analysis of the large-width distributions and training dynamics of ReLU neural networks initialized with α-Stable distributions.
result For ReLU neural networks with α-Stable initializations, the large-width training dynamics achieve zero training error at a linear rate, characterized by a random kernel.