CCAC calibrates DNN classifiers on OOD datasets by separating mis-classified samples.
problem Calibrating DNN classifiers on out-of-distribution datasets is challenging.
method CCAC introduces an auxiliary class to map DNN output to calibrated confidence, separating mis-classified from correctly classified samples.
result CCAC consistently outperforms prior methods on various DNN models, datasets, and applications.
Improved DNN calibration without sacrificing accuracy.
problem Poor calibration of over-parametrized DNNs in safety-critical applications.
method Decoupling feature extraction and classification layers, and applying Gaussian priors.
result Significant improvement in model calibration with minimal training cost.
Trained DNN models are increasingly adopted as integral parts of software systems, but they often perform deficiently in the field. A particularly damaging problem is that DNN models often give false predictions with high confidence, due to the unavoidable slight divergences between operation data and training data. To…
Proposes an accuracy-preserving calibration method for DNNs.
problem Calibration of deep neural networks (DNNs) to measure prediction reliability.
method Uses Concrete distribution on the probability simplex to calibrate DNNs without accuracy loss.
result The proposed method outperforms previous methods in accuracy-preserving calibration tasks.
New loss improves DNN calibration without sacrificing accuracy.
problem Overfitting and overconfident predictions in DNNs.
method Proposes a new loss function inspired by Bayes decision theory.
result Improves DNN calibration without compromising accuracy.
UTS improves DNN uncertainty calibration without labels, robust to domain shift.
problem Improving uncertainty calibration of DNNs under domain shift.
method UTS uses unlabeled test samples and a novel weighted NLL loss function.
result UTS outperforms other methods in domain shift scenarios.
Deep Neural Networks (DNNs) have achieved state-of-the-art accuracy performance in many tasks. However, recent works have pointed out that the outputs provided by these models are not well-calibrated, seriously limiting their use in critical decision scenarios. In this work, we propose to use a decoupled Bayesian stage…
DBLE improves confidence calibration of DNNs by learning distances in representation space.
problem Poor confidence calibration of deep neural networks (DNNs).
method DBLE trains a confidence model jointly with the classification model, using distances in the representation space.
result DBLE outperforms alternative single-model confidence calibration approaches and ensemble methods.
In this paper we study the probabilistic properties of the posteriors in a speech recognition system that uses a deep neural network (DNN) for acoustic modeling. We do this by reducing Kaldi's DNN shared pdf-id posteriors to phone likelihoods, and using test set forced alignments to evaluate these using a calibration s…
TCR improves DNN robustness to noisy labels with minimal overhead.
problem Training on noisy labeled datasets degrades DNN generalization.
method TCR combines original labels and previous epoch predictions for regularization.
result TCR consistently enhances DNN robustness to label noise.
KCal calibrates deep networks by embedding logits in a metric space.
problem Overconfident predictions from DNNs, especially in high-risk applications.
method KCal learns a metric space on the penultimate-layer latent embedding and generates predictions using kernel density estimates.
result KCal provides a provable full calibration guarantee and consistently outperforms baselines.
We apply supervised deep neural networks (DNNs) for pricing and calibration of both vanilla and exotic options under both diffusion and pure jump processes with and without stochastic volatility. We train our neural network models under different number of layers, neurons per layer, and various different activation fun…
Mixup~\cite{zhang2017mixup} is a recently proposed method for training deep neural networks where additional samples are generated during training by convexly combining random pairs of images and their associated labels. While simple to implement, it has been shown to be a surprisingly effective method of data augmenta…
New framework calibrates computer models using deep learning and quantile regression.
problem Uncertainty in computer model input parameters due to high-dimensional time series data.
method Deep neural network with long-short term memory layers for inverse modeling, quantile regression for interval predictions.
result Accurate point and interval estimates for input parameters in WRF-hydro model.
Optimizing proper loss yields calibrated models under specific conditions.
problem Understanding when optimizing proper loss functions leads to calibrated predictions.
method Local optimality condition and Lipschitz functions.
result Predictors with local optimality are nearly calibrated and nearly locally optimal.
Deep neural network (DNN) regression models are widely used in applications requiring state-of-the-art predictive accuracy. However, until recently there has been little work on accurate uncertainty quantification for predictions from such models. We add to this literature by outlining an approach to constructing predi…
Annealing Double-Head calibrates deep neural networks during training.
problem Overestimation or underestimation of predictive confidence in deep neural networks.
method An additional calibration head and Annealing technique to dynamically scale logits.
result State-of-the-art model calibration performance achieved without post-processing.
Paper proposes a method to predict deep neural network confidences with guarantees.
problem Quantifying uncertainty in deep neural networks for safety-critical applications.
method Uses Clopper-Pearson confidence intervals and histogram binning for calibrated prediction.
result Demonstrates the effectiveness of predicted confidences in improving DNN performance and safety.
Post-processes deep networks with StoNet to quantify uncertainty.
problem Uncertainty quantification in predictions from large-scale deep neural networks.
method Feeds DNN output into StoNet, trains StoNet with sparse penalty, constructs prediction intervals.
result Proposed approach constructs honest confidence intervals with shorter lengths and better calibration.
A novel approach models rating transitions using Lie groups and Deep Learning.
problem Modeling rating transitions with geometric properties and stochastic processes.
method Introducing Itô-SDEs on Lie groups, using TimeGAN for calibration, and examining rating matrix properties.
result The geometric approach using Lie groups and Deep Learning generates a good fit for rating transitions.
New study on neural network calibration, linking it to generalization gap.
problem Neural networks lack strong guarantees on calibration.
method Decomposed calibration error into train set and generalization gap.
result Models with small generalization gap are well-calibrated.
Develops a novel SABR DNN for accurate volatility surface calibration.
problem Inaccurate SABR model approximation for high volatility, long maturities, and out-of-the-money options.
method A specialized Artificial Deep Neural Network (DNN) architecture trained on a large dataset of interest rate volatility surfaces.
result Arbitrage-free calibration of real market volatility surfaces and Cap/Floor prices for any maturity and strike.
A new method quantifies deep neural network uncertainty by mixing OVA and AVA classifiers.
problem Uncertainty quantification in deep neural networks, especially for out-of-distribution data.
method Mixing predictions from OVA and AVA classifiers to improve uncertainty quantification.
result Achieves state-of-the-art performance in quantifying out-of-distribution data.
Deep neural networks (DNNs) have excellent representative power and are state of the art classifiers on many tasks. However, they often do not capture their own uncertainties well making them less robust in the real world as they overconfidently extrapolate and do not notice domain shift. Gaussian processes (GPs) with …
Focal loss improves deep neural networks' accuracy and calibration.
problem Miscalibration in deep neural networks.
method Using focal loss and temperature scaling to improve model calibration.
result Focal loss leads to state-of-the-art calibrated models without sacrificing accuracy.
Recently, Deep Neural Networks (DNNs) have been achieving impressive results on wide range of tasks. However, they suffer from being well-calibrated. In decision-making applications, such as autonomous driving or medical diagnosing, the confidence of deep networks plays an important role to bring the trust and reliabil…
Deep neural networks provide meaningful uncertainty estimates for large-scale simulations.
problem Uncertainty estimates for deep neural network predictions from large-scale simulations.
method General variational inference approach to calibrate Bayesian uncertainties.
result Calibrated Bayesian uncertainties preserved physics-correlations in predicted quantities.
Develops a calibration prediction interval for non-parametric regression and neural networks.
problem Lack of accurate conditional prediction in regression settings.
method Calibration Prediction Interval (cPI) using Deep Neural Networks (DNN) or kernel methods.
result Asymptotically valid coverage rate and high probability of coverage rate with large sample sizes.
A post-hoc framework improves model performance by calibrating different feature spaces.
problem Improving AUC performance on binary classification tasks for overconfident models.
method Identifies heterogeneous partitions of the feature space and applies post-hoc calibration techniques to each partition.
result Theoretical optimality of the framework for any model, demonstrated on deep neural networks.
Hybrid model improves forest growth predictions.
problem Misspecified assumptions in mechanistic models.
method Forest Informed Neural Networks (FINN) combining DVM and DNN.
result DNN learned improved growth process functional form.
This study compares two methods for uncertainty estimation in CNNs, finding Conformal Prediction more reliable.
problem CNNs often overestimate uncertainty, leading to unreliable predictions.
method Bayesian approximation via Monte Carlo Dropout and Conformal Prediction.
result Conformal Prediction produces more reliable uncertainty estimates than Monte Carlo Dropout.
SNGP improves DNNs' uncertainty estimation with minimal changes.
problem Uncertainty estimation in deep learning models for real-time applications.
method Formalizing uncertainty as a minimax problem, SNGP adds weight normalization and replaces the output layer with a Gaussian process.
result SNGP outperforms other single-model approaches in uncertainty estimation across vision and language tasks.
Deep neural networks can accurately approximate option prices in stochastic volatility models.
problem Approximating option prices in complex stochastic volatility models.
method Use deep neural networks to approximate option prices for a general class of stochastic volatility models.
result Deep neural networks can approximate option prices up to small error ε with sub-polynomial network size growth.
TS improves class coverage but reduces CP set size, offering a trade-off.
problem Combining temperature scaling with conformal prediction for deep classifiers.
method Empirical study and mathematical theory of TS's effect on CP.
result TS allows trading prediction set size and conditional coverage.
New methods compress models without real data, reducing accuracy loss.
problem Compression requires real data, which is often unavailable or sensitive.
method Synthetic data generation from trained models for calibration and fine-tuning.
result Best method shows negligible accuracy loss compared to original training set.
To solve deep neural network (DNN)'s huge training dataset and its high computation issue, so-called teacher-student (T-S) DNN which transfers the knowledge of T-DNN to S-DNN has been proposed. However, the existing T-S-DNN has limited range of use, and the knowledge of T-DNN is insufficiently transferred to S-DNN. To …
Deep neural networks (DNNs) have emerged as key enablers of machine learning. Applying larger DNNs to more diverse applications is an important challenge. The computations performed during DNN training and inference are dominated by operations on the weight matrices describing the DNN. As DNNs incorporate more layers a…
Sparse DNNs face scalability issues; MIT/IEEE/Amazon challenge analyzes best solutions.
problem Scalability issues in Sparse Deep Neural Networks (DNNs).
method Mathematically defined DNN inference computation, community submissions from various fields.
result Sparse DNN execution time, TmDNN, is strongly dependent on the number of operations, Nmop. Whereas deep neural network (DNN) is increasingly applied to choice analysis, it is challenging to reconcile domain-specific behavioral knowledge with generic-purpose DNN, to improve DNN's interpretability and predictive power, and to identify effective regularization methods for specific tasks. This study designs a pa…
Deep Neural Network (DNN) acoustic models have yielded many state-of-the-art results in Automatic Speech Recognition (ASR) tasks. More recently, Recurrent Neural Network (RNN) models have been shown to outperform DNNs counterparts. However, state-of-the-art DNN and RNN models tend to be impractical to deploy on embedde…
Abstract Neural Networks (ANNs) improve DNN verification efficiency.
problem Efficiently verify safety-critical DNNs without slowing exponentially.
method Introduces ANNs that use abstract domains and activation functions to overapproximate DNNs.
result ANNs can soundly overapproximate DNNs with fewer nodes, improving verification efficiency.
This paper explains robust overfitting in wide DNNs using adversarial training and NTK theory.
problem Robust overfitting in adversarially trained wide DNNs.
method Theoretical analysis using neural tangent kernel (NTK) theory and adversarial training dynamics.
result Adversarial training can lead to robust overfitting in wide DNNs, which can be mitigated by the proposed Adv-NTK method.
DNN pruning reduces memory footprint and computational work of DNN-based solutions to improve performance and energy-efficiency. An effective pruning scheme should be able to systematically remove connections and/or neurons that are unnecessary or redundant, reducing the DNN size without any loss in accuracy. In this p…
Supervised learning is the workhorse for regression and classification tasks, but the standard approach presumes ground truth for every measurement. In real world applications, limitations due to expense or general in-feasibility due to the specific application are common. In the context of agriculture applications, yi…
We present a novel deep Recurrent Neural Network (RNN) model for acoustic modelling in Automatic Speech Recognition (ASR). We term our contribution as a TC-DNN-BLSTM-DNN model, the model combines a Deep Neural Network (DNN) with Time Convolution (TC), followed by a Bidirectional Long Short-Term Memory (BLSTM), and a fi…
Deep neural networks (DNNs) are now a central component of nearly all state-of-the-art speech recognition systems. Building neural network acoustic models requires several design decisions including network architecture, size, and training loss function. This paper offers an empirical investigation on which aspects of …
DNNs improve accuracy by using more evidence from images.
problem Understanding why DNNs generalize well and improving model selection metrics.
method Minimal sufficient views (MSVs) to identify key evidence regions in images.
result DNNs with more evidence regions in images have higher generalization performance.
Energy savings for DNN inference on resource-constrained devices.
problem Energy efficiency in deep learning inference for constrained devices.
method Efficiently searches through equivalent DNN graphs to find the one with the least execution cost.
result Achieves 24% energy savings with minimal performance impact.