This paper analyzes output activation functions for adversarial losses.
problem Understanding which output activation functions form a well-behaved adversarial loss.
method Variational divergence minimization and a comparative framework for adversarial losses.
result There is no single winning combination of output activation functions and regularization approaches across all settings.
We replace the output layer of deep neural nets, typically the softmax function, by a novel interpolating function. And we propose end-to-end training and testing algorithms for this new architecture. Compared to classical neural nets with softmax function as output activation, the surrogate with interpolating function…
Paper proposes active learning for structured output design, improving Gaussian process model predictions.
problem Finding optimal input parameters for achieving desired structured outputs.
method Developed new acquisition functions to minimize prediction error of Gaussian process model, incorporating output correlations.
result Effectiveness demonstrated in synthetic and real data experiments, including materials informatics.
The paper proves neural networks with ReLU and softmax can approximate any function.
problem Approximating functions and class labels in neural networks.
method Extended universal approximator theory to neural networks with ReLU and softmax.
result Neural networks with ReLU and softmax can approximate any function and class labels.
When approximating a black-box function, sampling with active learning focussing on regions with non-linear responses tends to improve accuracy. We present the FLOLA-Voronoi method introduced previously for deterministic responses, and theoretically derive the impact of output uncertainty. The algorithm automatically p…
DeepICMGP surrogate models multiple outputs efficiently.
problem Challenges in modeling dependencies between multiple outputs using traditional multi-output GPs.
method Introduces hierarchical coregionalization structures across layers in DGPs.
result Demonstrates competitive performance and active learning strategies.
Active learning method reduces labeling cost for regression models with aggregated data.
problem Reducing labeling cost for training regression models with aggregated data.
method Sequentially selects sets to be labeled using mutual information quantifying model parameter uncertainty.
result Achieves better predictive performance with fewer labeled sets.
This paper provides an overview of activation functions in neural networks.
problem Confusion in activation function selection and properties in deep learning.
method Analytic review of popular activation functions.
result Clarification of activation function properties and selection.
Softmax is an output activation function for modeling categorical probability distributions in many applications of deep learning. However, a recent study revealed that softmax can be a bottleneck of representational capacity of neural networks in language modeling (the softmax bottleneck). In this paper, we propose an…
We study algebraic varieties of ReLU networks to understand their representable functions.
problem Understanding the functions that ReLU neural networks can represent.
method We introduce algebraic varieties associated with ReLU networks and derive polynomial equations to characterize representable functions.
result Conditions under which ReLU networks attain their expected dimension, providing insight into their structural properties.
New method learns neural network activation functions from data.
problem Learning activation functions for neural networks.
method Model each neuron's activation function as a small neural network.
result Learned activation functions improve network performance.
Randomly initialized wide neural networks with zero-mean activations are nearly independent, potentially solving AI interpretability limits.
problem Measuring the limits of AI interpretability.
method Randomly initialized neural networks with large width and zero-mean activation functions.
result Neural networks with zero-mean activations are nearly independent, solving the computational no-coincidence conjecture.
Hi-fi priors enhance BNNs by learning flexible activations.
problem Challenging to impose function-space priors on BNNs.
method Optimization techniques to learn flexible activations.
result BNNs with flexible activations can achieve desired priors.
A new method for the unsupervised learning of sparse representations using autoencoders is proposed and implemented by ordering the output of the hidden units by their activation value and progressively reconstructing the input in this order. This can be done efficiently in parallel with the use of cumulative sums and …
New activation improves deep learning accuracy and robustness.
problem Improving accuracy and robustness of deep neural nets with limited data.
method Replaces softmax with graph Laplacian-based interpolating function.
result Significantly improves natural and robust accuracy.
Improves active learning efficiency by warping input space based on observed outputs.
problem Insensitivity of Gaussian process uncertainty to actual observations.
method Input warping with learned monotone reparameterization to adjust acquisition function behavior.
result Significantly improved sample efficiency across various benchmarks, especially in non-stationary conditions.
Study shows MSE with sigmoid can match SCE in classification tasks, especially with noisy data.
problem Inconsistent errors in neural network classification tasks.
method Introduced Output Reset algorithm to use MSE with sigmoid activation.
result MSE with sigmoid activation achieves comparable accuracy and convergence rates to Softmax Cross-Entropy, especially in noisy data scenarios.
Minimum width for ReLU networks to approximate L^p functions is max(d_x+1, d_y).
problem Characterizing the minimum width for ReLU networks to approximate L^p functions.
method Analyzing networks with ReLU activation functions and proving the minimum width required.
result The minimum width required for the universal approximation of L^p functions is exactly max(d_x+1, d_y).
New method selects inputs for Bayesian regression with few samples.
problem Optimal experimental design for high-dimensional inputs.
method Output-weighted optimal sampling using Bayesian regression.
result New criterion considers output values of existing samples.
A new mutual information lower bound for multimodal regression active learning.
problem Lack of effective acquisition functions for multimodal regression active learning.
method Introduces a Two-Index framework for separating epistemic and aleatoric sources of uncertainty, deriving MI-LB as a closed-form approximation.
result MI-LB consistently outperforms baselines on multimodal regression tasks.
Safe active learning for multi-output Gaussian processes reduces data acquisition costs and ensures safety.
problem Expensive data acquisition and safety concerns in multi-output regression problems.
method Proposes a safe active learning approach considering data informativeness and safety constraints.
result Improved convergence compared to competitors on simulated and real-world datasets.
Scientists and engineers rely on accurate mathematical models to quantify the objects of their studies, which are often high-dimensional. Unfortunately, high-dimensional models are inherently difficult, i.e. when observations are sparse or expensive to determine. One way to address this problem is to approximate the or…
URNNs are as expressive as general RNNs with ReLU activations.
problem Expressiveness of URNNs compared to general RNNs.
method Input-output equivalence between URNNs and contractive RNNs with ReLU activations.
result URNNs are as expressive as general RNNs with ReLU activations.
This research improves neural network performance with adaptive activation functions in sparse data settings.
problem Limited data availability in scientific and engineering problems.
method Investigation of two types of adaptive activation functions with individual trainable parameters.
result Adaptive activation functions, especially with individual trainable parameters, enhance prediction accuracy and confidence in sparse data settings.
`Biologically inspired' activation functions, such as the logistic sigmoid, have been instrumental in the historical advancement of machine learning. However in the field of deep learning, they have been largely displaced by rectified linear units (ReLU) or similar functions, such as its exponential linear unit (ELU) v…
Paper develops a method to create accurate emulators of expensive computer codes.
problem High cost and complexity of running complex computer codes.
method Active learning with Gaussian processes to construct emulators.
result Accurate and compact emulators created for expensive codes.
New methods for uncertainty in neural networks with leaky ReLU activations.
problem Uncertainty in feed-forward neural networks with random input perturbations.
method Analytical expressions for PDF and moments of neural network output, linearization of leaky ReLU, Gaussian copula surrogate models.
result Accurate statistical results for large input perturbations, excellent agreement with Monte Carlo simulations.
Optimizes sampling for faster convergence in Bayesian experimental design and uncertainty quantification.
problem Efficiently selecting samples for faster convergence in Bayesian experimental design and uncertainty quantification.
method Output-weighted acquisition functions leveraging likelihood ratio to guide sampling towards relevant regions.
result Superiority of the proposed method in uncertainty quantification and rare event identification.
Multi-output Gaussian processes (MOGP) are probability distributions over vector-valued functions, and have been previously used for multi-output regression and for multi-class classification. A less explored facet of the multi-output Gaussian process is that it can be used as a generative model for vector-valued rando…
New method estimates active subspaces for jump-discontinuous functions.
problem Estimating active subspaces for discontinuous functions like ABMs.
method Extending active subspaces to discontinuous functions, using Gaussian process.
result Identifies important parameters in ABM simulations of refugee movement.
Deep narrow networks can approximate any continuous function.
problem Approximating continuous functions with neural networks of bounded width and arbitrary depth.
method Showed neural networks of arbitrary depth, width n+m+2, and activation function ρ is dense in C(K;Rm) for K⊆Rn with K compact. result Neural networks of bounded width and arbitrary depth can approximate any continuous function.
Paper proposes a new activation function to reduce overfitting and large weight update issues.
problem Overfitting and large weight update problems in neural networks.
method Introduces a new activation function called Thresholded Exponential Rectified Linear Units (TERELU).
result TERELU shows better performance in reducing overfitting and large weight update issues compared to other activation functions.
A new method reduces high-dimensional parameter spaces for faster numerical tasks.
problem Efficiently reducing high-dimensional parameter spaces for numerical tasks.
method Local Active Subspaces (LAS) combining active subspaces with clustering techniques.
result Significant speed-up in numerical tasks through efficient dimension reduction.
Improved active output selection reduces calibration time by 10% or more.
problem Efficiently calibrate models with noisy data.
method Improved active output selection strategy considering noise estimate.
result At least 10% fewer measurements needed compared to existing strategies.
Bayesian Layers adds uncertainty to neural networks, enabling faster experimentation and scalability.
problem Enabling neural networks to quantify uncertainty in predictions.
method Drop-in replacements for common layers, capturing uncertainty over weights, activations, etc.
result Bayesian Layers can fit large models like 5-billion parameter Bayesian Transformers.
We improve the robustness of Deep Neural Net (DNN) to adversarial attacks by using an interpolating function as the output activation. This data-dependent activation remarkably improves both the generalization and robustness of DNN. In the CIFAR10 benchmark, we raise the robust accuracy of the adversarially trained Res…
ReLU networks trained with MILPs match deep learning accuracy.
problem Training deep neural networks efficiently.
method Iterative training with Mixed Integer Linear Programs (MILPs).
result ReLU networks can be trained with MILPs achieving similar accuracy to deep learning methods.
Wide deep neural networks with Gaussian weights approximate Gaussian processes closely.
problem Understanding the approximation of deep neural networks with Gaussian weights to Gaussian processes.
method Established novel rates for the Gaussian approximation of random deep neural networks with Gaussian parameters and Lipschitz activation functions in the wide limit.
result The distance between the network output and the Gaussian approximation scales inversely with the width of the network.
Formalizes explanations as blending input and model output.
problem Creating clear and consistent explanations for model predictions.
method Defines properties of explanation functions and links them to model layers.
result Consistency of activations across layers implies consistency of explanations.
Study efficient interactive learning for structured outputs with reliable computation.
problem Interactive learning with noisy labels and structured output spaces.
method Identify and utilize CRISPs (probabilistic models) that guarantee reliable and efficient computation of probabilistic quantities.
result CRISPs enable robust and efficient active and skeptical learning in large structured output spaces.
We develop a method to learn neural network activations with controlled Lipschitz constant.
problem Increase neural network capacity while controlling Lipschitz constant.
method Variational framework to learn activation functions with piecewise-linear constraints.
result Proves existence of solutions with continuous and piecewise-linear activations.
Procedure removes training data dependency from deep networks, improving generalization.
problem Removing dependency on training data in deep networks for better generalization.
method Deterministic and stochastic parts to ensure forgetting, leveraging activation and weight dynamics.
result New bound on information extraction from black-box networks, ensuring forgetting in activations.
An increasing body of evidence suggests that the trial-to-trial variability of spiking activity in the brain is not mere noise, but rather the reflection of a sampling-based encoding scheme for probabilistic computing. Since the precise statistical properties of neural activity are important in this context, many model…
Convex neural networks enforce convex constraints on weights and activations, improving generalization.
problem Improving generalization and reducing overfitting in neural networks.
method Enforce convex constraints on weights and activations, using non-negative weights and non-decreasing convex activation functions.
result Convex neural networks self-regularize, outperforming base architectures and achieving similar performance to convolutional architectures.
Neural networks learn to mimic brain neurons with two-input activation functions, improving performance and robustness.
problem Training neural networks to mimic the complex interactions of brain neurons.
method Developed a network-in-network architecture with two-input activation functions, optimized hyperparameters, and compared to conventional ReLU networks.
result Two-input activation functions can learn soft XOR functions, improving network performance and robustness.
Exact bounds derived for neural network outputs with noisy inputs.
problem Bounding the output distribution of neural networks with random inputs.
method Applying ReLU NNs to derive bounds for general NNs, then using these to find exact error guarantees.
result Exact upper and lower bounds for the output distribution of neural networks with random inputs.
Complex-valued neural networks can approximate any continuous function with bounded widths and depths.
problem Approximating continuous functions with complex-valued neural networks of bounded widths and depths.
method Analyzing activation functions and proving universality for complex-valued networks.
result Deep narrow complex-valued networks are universal if and only if their activation function is neither holomorphic, nor antiholomorphic, nor R-affine. Fragility curves which express the failure probability of a structure, or critical components, as function of a loading intensity measure are nowadays widely used (i) in Seismic Probabilistic Risk Assessment studies, (ii) to evaluate impact of construction details on the structural performance of installations under se…