This work evaluates and benchmarks calibration metrics for data-driven regression models.
problem Conflicting results from different calibration metrics make it hard to compare and interpret model performance.
method Systematically extracted and benchmarked 14 regression calibration metrics across various data types and recalibration methods.
result Many metrics disagree on the same recalibration result, highlighting the need for careful metric selection.
This paper establishes the asymptotic consistency of the {\it loss-calibrated variational Bayes} (LCVB) method. LCVB was proposed in~\cite{LaSiGh2011} as a method for approximately computing Bayesian posteriors in a `loss aware' manner. This methodology is also highly relevant in general data-driven decision-making con…
New method calibrates heterogeneous treatment effect models.
problem Difficulty in estimating and calibrating heterogeneous treatment effects.
method Defined and proposed a robust estimator for HTE calibration, based on doubly robust treatment effect estimators.
result Proposed method evaluates calibration of learned HTE models, addressing overfitting and high-dimensionality.
New framework calibrates decision robustness using inverse conformal risk control.
problem Inadequate robustness levels in decision-making due to ad hoc choices.
method Constructs valid estimators to trace miscoverage-regret Pareto frontier.
result Provides distribution-free, finite-sample guarantees on robustness levels.
Data-driven decision-making often overestimates benefits due to the winner's curse.
problem Accurate policy evaluation in data-driven decision-making.
method Model-based policy evaluation using estimated models from data.
result Model-based methods can produce large, spurious reported benefits even when true effects are zero.
Bayesian neural networks improve uncertainty in data-driven VFMs for oil and gas wells.
problem Uncertainty and robustness in data-driven VFMs for oil and gas wells.
method Bayesian neural networks with variational inference for uncertainty quantification.
result Variational inference provides more robust predictions on future data.
RoPE framework calibrates misspecified simulators for reliable inference.
problem Misspecification compromises reliability of simulation-based inference.
method Data-driven calibration using optimal transport and a small calibration set.
result RoPE framework improves inference accuracy and uncertainty calibration.
New method calibrates uncertainty in molecular property predictions.
problem Uncalibrated uncertainty estimates in molecular property predictions.
method Message Passing Neural Networks with calibrated probabilistic predictive distribution.
result Accurate molecular formation energy predictions with well-calibrated uncertainty.
A data-driven approach called CaNN (Calibration Neural Network) is proposed to calibrate financial asset price models using an Artificial Neural Network (ANN). Determining optimal values of the model parameters is formulated as training hidden neurons within a machine learning framework, based on available financial op…
The optimization of composition and processing to obtain materials that exhibit desirable characteristics has historically relied on a combination of scientist intuition, trial and error, and luck. We propose a methodology that can accelerate this process by fitting data-driven models to experimental data as it is coll…
New method calibrates stochastic reduced-order models from data efficiently.
problem Challenges in estimating drift and diffusion coefficients from data for high-dimensional systems.
method Uses a novel relationship between conditional score and transition density to constrain model coefficients directly from finite-lag statistics.
result Validated on various systems, the method reproduces statistical and dynamical properties of the original models.
Improves data-driven reachability estimation for complex systems.
problem Estimating reachable states in complex dynamical systems with unknown parameters.
method Uses Christoffel functions and conformal prediction to improve sample efficiency and robustness.
result Guaranteed convergence to the true reach set with improved sample efficiency and robustness.
Blade uses diffusion priors to accurately and calibratedly infer complex systems.
problem Derivative-free Bayesian inversion for high-dimensional, nonlinear problems with costly forward models.
method Blade employs an ensemble of interacting particles and diffusion models as priors, querying forward models only through evaluations.
result Blade produces well-calibrated posterior samples that existing methods cannot, improving with more iterations and particles.
Machine learning approximates implied volatility and dividend yield for American options.
problem Challenges in extracting implied information from American options due to computational costs.
method Employing a data-driven machine learning approach, specifically a Calibration Neural Network (CaNN), to estimate implied volatility and dividend yield efficiently.
result Machine learning can be used to estimate implied volatility and dividend yield for American options efficiently.
The paper proposes a neural network method to calibrate LSV models without interpolation.
problem Calibrating LSV models with market option prices using neural networks.
method Parametrizing leverage function with neural networks and learning parameters from market prices; using deep hedging for variance reduction.
result The method accurately calibrates LSV models and outperforms interpolation methods.
Interest in agent-based models of financial markets and the wider economy has increased consistently over the last few decades, in no small part due to their ability to reproduce a number of empirically-observed stylised facts that are not easily recovered by more traditional modelling approaches. Nevertheless, the age…
Decision-calibrated prediction sets improve power system operations by reducing unnecessary costs.
problem Balancing operating costs and reliability in power systems with renewable uncertainty.
method Learn conditional prediction sets as sub-level sets of norm-based score functions, calibrate uncertainty sets based on reliability of downstream decisions.
result Decision-calibrated sets lead to more efficient operations with smaller uncertainty sets and lower costs compared to standard coverage-based calibration.
MCP extends conformal prediction to vector-valued score functions without data splitting.
problem Fixed prediction set shapes in scalar score functions limit coverage guarantees.
method MCP uses a single optimization problem for prediction set design and calibration, eliminating data splitting.
result RemMCP and RelMCP achieve target coverage with smaller or comparable prediction set sizes, reducing variance.
Framework improves PV forecasting by accounting for missing data uncertainty.
problem Uncertainty from missing data in PV power data.
method Combines stochastic multiple imputation with Rubin's rule.
result Improves prediction interval calibration without sacrificing point prediction accuracy.
This paper tackles the problem of selecting among several linear estimators in non-parametric regression; this includes model selection for linear regression, the choice of a regularization parameter in kernel ridge regression, spline smoothing or locally weighted regression, and the choice of a kernel in multiple kern…
An innovative method optimizes engine calibration to improve efficiency and reduce emissions.
problem Complex engines with many tunable parameters require efficient calibration methods.
method Combines Principal Component Decomposition with constrained Bayesian Optimization to minimize pressure curve deviation.
result Optimal engine calibration found after 64.4s with a 0.017% efficiency gain.
Paper proposes a new framework for individualized treatment rules that generalize better across different distributions.
problem Existing individualized treatment rules may not generalize well when training and testing distributions differ.
method Distributionally robust individualized treatment rules (DR-ITR) framework that maximizes worst-case value function across close distributions.
result Calibrated DR-ITR outperforms standard ITR in generalizability.
Sig-SDE model integrates signatures with SDEs for financial data.
problem Calibrating models to exotic financial products with non-linear dependencies.
method Integrating signatures from stochastic analysis with neural SDEs.
result Sig-SDE provides theoretical guarantees for convergence.
New methods for volatility modeling using rough paths and signatures.
problem Calibrating implied volatility surfaces in various stochastic models.
method Analytical approximations and signature-based models based on rough path theory.
result Signature-based models achieve comparable accuracy to analytical expansions and can capture more complex dynamics.
A new method for designing accurate emulators using deep learning with interval calibration.
problem Designing accurate emulators for scientific processes with modern machine learning methods.
method Learn-by-Calibrating (LbC) approach based on interval calibration.
result Significant improvements in generalization error over widely-used loss functions.
Unified framework for hierarchical image classification with epistemic uncertainty.
problem Overconfident predictions and lack of logical consistency in deep learning models.
method Neurosymbolic approach with epistemic deep learning, using focal set reasoning and differentiable fuzzy logic.
result Maintains accuracy on par with transformer baselines while providing more calibrated and interpretable predictions.
We study the out-of-sample properties of robust empirical optimization problems with smooth φ-divergence penalties and smooth concave objective functions, and develop a theory for data-driven calibration of the non-negative "robustness parameter" δ that controls the size of the deviations from the nominal model. Bu…
ARHNN method improves electricity price forecasting accuracy.
problem Improving accuracy in electricity price forecasting.
method Combines Autoregressive Hybrid Nearest Neighbors (ARHNN) method with calibration sample selection and forecast combination.
result ARHNN method outperforms benchmarks by up to 10% in German, Spanish, and New England markets.
Regional rainfall-runoff modeling is an old but still mostly out-standing problem in Hydrological Sciences. The problem currently is that traditional hydrological models degrade significantly in performance when calibrated for multiple basins together instead of for a single basin alone. In this paper, we propose a nov…
Study evaluates uncertainty estimation methods in binary classification models.
problem Difficulty in quantifying uncertainty in complex models like deep learning.
method Approximate Bayesian inference with synthetic datasets and empirical tests.
result Deep learning-based algorithms do not consistently reflect lack of evidence for out-of-distribution data.
Framework disentangles deep feature uncertainty for efficient inference.
problem Inference-time uncertainty estimation for reliable decision-making.
method Uncertainty-Guided Inference-Time Selection framework.
result Significantly tighter prediction intervals and 60% compute reduction.
Work addresses long-term accuracy issues in IoT air quality sensors.
problem Limited accuracy of IoT air quality sensors in long-term field deployments.
method Adaptive machine learning strategies for network calibration.
result Prolongs the validity of multisensor calibration models for continuous learning.
Bayesian framework calibrates imperfect models using physics-informed priors and Hamiltonian Monte Carlo.
problem Quantifying uncertainty in imperfect computer models described by differential equations.
method Physics-informed Gaussian process priors, discrepancy function, Hamiltonian Monte Carlo, data approximations.
result Framework accurately recovers true parameters and produces accurate predictions.
The paper introduces a new model selection criterion for various time series models.
problem Designing adaptive model selection criteria for a wide range of time series models.
method The approach involves a penalized contrast akin to Hannan and Quinn's criterion, with a data-driven calibrated term.
result The new criteria select the true model almost surely asymptotically for a wide range of time series models.
ADML combines debiased learning with data-driven model selection for efficient inference.
problem Debiased machine learning estimators can be unstable and biased in nonparametric models.
method Data-driven model selection techniques combined with debiased machine learning.
result ADML estimators yield superefficient inference for pathwise differentiable parameters.
New methods for tuning alpha in Gibbs posteriors improve speed and accuracy.
problem Inconsistency in Bayesian inference and lack of fast tuning methods for alpha.
method Proposed two data-driven methods: sample-splitting and bootstrapping. Formulated alpha-posteriors for three models.
result Sample-splitting outperforms SafeBayes in speed and accuracy, especially in complex models.
Prediction models can harm patients even when accurate, leading to self-fulfilling prophecies.
problem Prediction models can lead to harmful decisions that worsen patient outcomes.
method Formal characterization of harmful prediction models and analysis of their impact.
result Well-calibrated models are ineffective for decision-making as they do not change the data distribution.
We introduce a methodology for nonlinear inverse problems using a variational Bayesian approach where the unknown quantity is a spatial field. A structured Bayesian Gaussian process latent variable model is used both to construct a low-dimensional generative model of the sample-based stochastic prior as well as a surro…
Enhanced regime shifts detection using unstructured text and financial data.
problem Detecting regime shifts in financial markets is challenging due to noisy and multicollinear data.
method Combines LLM reasoning on unstructured text and statistical validation on financial time series.
result Framework achieves F1 score of 0.82, outperforming pure data-driven methods.
We consider the problem of reconstructing the dynamic state matrix of transmission power grids from time-stamped PMU measurements in the regime of ambient fluctuations. Using a maximum likelihood based approach, we construct a family of convex estimators that adapt to the structure of the problem depending on the avail…
It is well known that the out-of-sample performance of Markowitz's mean-variance portfolio criterion can be negatively affected by estimation errors in the mean and covariance. In this paper we address the problem by regularizing the mean-variance objective function with a weighted elastic net penalty. We show that the…
New methods for estimating and inferring nonparametric structural functions and elasticities.
problem Estimating and inferring nonparametric structural functions and their derivatives.
method Data-driven sieve dimension choice and uniform confidence bands construction.
result Optimal estimation and inference procedures with minimax rates of convergence.
OptCS optimizes model selection after conformal inference, controlling FDR and power loss.
problem Challenges in model selection for conformal inference, especially when limited labeled data and many model choices are available.
method OptCS framework that allows valid statistical testing after flexible data-driven model optimization, using novel multiple testing procedures.
result Valid conformal p-values constructed despite substantial data reuse, maintaining FDR control.
Neural clustering learns time series affinity from statistical features.
problem Challenging time series clustering with unknown cluster shapes and structures.
method Amortized neural inference using statistical features.
result Competitive clustering accuracy without manual specification of cluster shapes.
Paper introduces a method to control early classification accuracy gaps.
problem Maintaining accuracy in early classification without full input processing.
method Statistical framework for a calibrated stopping rule.
result Reduces up to 94% of timesteps while controlling accuracy gaps.
The study compares on-chain option prices with a model and finds significant differences.
problem Measuring and comparing on-chain option prices with a model-based benchmark.
method Used a two-regime MS-AR-(GJR)-GARCH model to estimate volatility and GLS to compare prices.
result On-chain option prices are significantly higher than model-based benchmarks, especially for call options.
New PRGP model improves traffic flow estimation.
problem Lack of models combining physics and ML for traffic flow.
method Physics regularized Gaussian process (PRGP) with discrete formulations.
result PRGP model outperforms calibrated physics models and ML methods.
This research improves debt collection strategies using advanced machine learning.
problem Accurate estimation of propensity to pay and cashflow for optimal debt collection.
method Developed a machine learning framework with pre-processing and model selection.
result The proposed model outperforms current industry strategies.