In many real-world applications of machine learning classifiers, it is essential to predict the probability of an example belonging to a particular class. This paper proposes a simple technique for predicting probabilities based on optimizing a ranking loss, followed by isotonic regression. This semi-parametric techniq…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “real-world probability”
This work improves deep neural network probability estimation methods.
problem Estimating probabilities from high-dimensional data with inherent uncertainty.
method Investigates and compares methods for probability estimation using deep neural networks, proposing a new method that promotes consistent probabilities.
result The new method outperforms existing approaches on most metrics on simulated and real-world data.
Unified framework models multiple financial and insurance term structures.
problem Modeling multiple term structures in various markets.
method Extended Heath-Jarrow-Morton (HJM) approach under real-world probability.
result Characterization of local martingale deflators and existence of affine realizations.
Develops European power option pricing under correlated interest rate and asset processes.
problem Pricing European power options under correlated interest rate and asset processes.
method Martingale method and Girsannov transform.
result Derives European power option pricing formulae under two market assumptions.
We investigate the existence of affine realizations for Lévy driven interest rate term structure models under the real-world probability measure, which so far has only been studied under an assumed risk-neutral probability measure. For models driven by Wiener processes, all results obtained under the risk-neutral appro…
This paper proposes two approaches that quantify the exact relationship among the viability, the absence of arbitrage, and/or the existence of the numéraire portfolio under minimal assumptions and for general continuous-time market models. Precisely, our first and principal contribution proves the equivalence among the…
In retailer management, the Newsvendor problem has widely attracted attention as one of basic inventory models. In the traditional approach to solving this problem, it relies on the probability distribution of the demand. In theory, if the probability distribution is known, the problem can be considered as fully solved…
Credit Value Adjustment (CVA) is the difference between the value of the default-free and credit-risky derivative portfolio, which can be regarded as the cost of the credit hedge. Default probabilities are therefore needed, as input parameters to the valuation. When liquid CDS are available, then implied probabilities …
Binary classification is highly used in credit scoring in the estimation of probability of default. The validation of such predictive models is based both on rank ability, and also on calibration (i.e. how accurately the probabilities output by the model map to the observed probabilities). In this study we cover the cu…
Study optimizes tree-based models for better alignment of predicted scores and actual probabilities.
problem Traditional calibration metrics fail to align predicted scores with actual probabilities when score distributions deviate from the underlying data.
method Optimizes tree-based models (Random Forest, XGBoost) using Kullback-Leibler (KL) divergence to minimize the difference between predicted and true probability distributions.
result Optimized tree-based models yield superior alignment between predicted scores and actual probabilities without significant performance loss.
Submodular function maximization finds application in a variety of real-world decision-making problems. However, most existing methods, based on greedy maximization, assume it is computationally feasible to evaluate F, the function being maximized. Unfortunately, in many realistic settings F is too expensive to evaluat…
Improved probabilistic forecasts using behavioral transformations.
problem Improving accuracy and consistency of probabilistic asset price forecasts.
method Behavioral transformation of fundamental expectations to disentangle sentiment-induced biases.
result Substantial forecast gains across various models and risk-preferences.
The paper bounds and identifies joint probabilities in causal inference with monotonicity assumptions.
problem Bounding and identifying joint probabilities of potential outcomes and observed variables under monotonicity assumptions.
method Proposes new families of monotonicity assumptions, formulates bounding problem as linear programming, introduces new monotonicity assumption for identification.
result Validated methods through numerical experiments and applied to real-world datasets.
This paper proposes a paradigm shift in the valuation of long term annuities, away from classical no-arbitrage valuation towards valuation under the real world probability measure. Furthermore, we apply this valuation method to two examples of annuity products, one having annual payments linked to a mortality index and…
This paper develops GPCA for probability distributions using Otto-Wasserstein geometry.
problem Analyzing modes of variation in datasets of probability measures.
method Geodesic Principal Component Analysis (GPCA) on Wasserstein space with neural networks.
result Identification of geodesic curves that capture modes of variation in probability distributions.
Develops a new minimax probability machine for imbalanced classification tasks.
problem Imbalanced classification tasks with non-decomposable performance measures.
method Derives an equivalent form of the MPMF model for solving linear and nonlinear classifiers.
result Demonstrates the effectiveness of the new model on real-world datasets.
New probability path model improves flow matching forecasting performance.
problem Impact of probability path model selection on flow matching forecasting performance.
method Proposed a novel probability path model designed to improve forecasting performance.
result Our model achieves faster convergence during training and improved predictive performance compared to existing models.
Diffusion reach probability between two nodes on a network is defined as the probability of a cascade originating from one node reaching to another node. An infinite number of cascades would enable calculation of true diffusion reach probabilities between any two nodes. However, there exists only a finite number of cas…
While recent developments in autonomous vehicle (AV) technology highlight substantial progress, we lack tools for rigorous and scalable testing. Real-world testing, the evaluation environment, places the public in danger, and, due to the rare nature of accidents, will require billions of miles in or…
A novel post-hoc calibration method reduces neural network calibration errors.
problem Neural networks produce poorly calibrated probabilities, leading to underconfidence and overconfidence.
method Probability bounding (PB) via box-constrained softmax (BCSoftmax) function.
result Consistently reduces calibration errors on four real-world datasets.
This work assesses DNNs for estimating conditional probabilities.
problem Lack of uncertainty characterization in DNNs for probabilistic applications.
method Investigates DNNs' ability to estimate conditional probabilities using synthetic and real-world datasets.
result DNNs' precision in estimating conditional probabilities is influenced by probability density and inter-categorical sparsity.
Bonus-malus Systems vs Delays in Claims Reporting and Settlement: Analysis of Ruin Probabilitiesq-fin.RM
The study analyzes how bonus-malus systems and delayed claims settlement affect insurance companies' financial stability.
problem Analyzing the impact of bonus-malus systems and delayed claims settlement on insurance companies' financial stability.
method Examined a discrete-time risk model with time-varying premiums, evaluating two types of claims and settlement delays.
result Delayed settlement of by-claims leads to lower ruin probabilities under specific assumptions.
Focal loss improves classification but not class-posterior probability estimation.
problem Improving class-posterior probability estimation from focal loss.
method Proved classification-calibration and derived a transformation to recover true class-posterior probabilities.
result A transformation of the confidence score from focal loss minimization allows recovery of true class-posterior probabilities.
This paper investigates the pricing and hedging of variance swaps under a volatility model. Explicit pricing and hedging formulas of variance swaps are obtained under the benchmark approach, which only requires the existence of the numéraire portfolio. The growth optimal portfolio is the numéraire portfolio and u…
Hybrid continuous-discrete models naturally represent many real-world applications in robotics, finance, and environmental engineering. Inference with large-scale models is challenging because relational structures deteriorate rapidly during inference with observations. The main contribution of this paper is an efficie…
Many classification applications require accurate probability estimates in addition to good class separation but often classifiers are designed focusing only on the latter. Calibration is the process of improving probability estimates by post-processing but commonly used calibration algorithms work poorly on small data…
New method detects anomalies in time series data, especially useful for monitoring services.
problem Detecting anomalies in time series data, especially for monitoring services and cloud resources.
method Models time series of probability distributions over real values, scales to millions of time series.
result Outperforms state-of-the-art methods in detecting anomalies on various data sets.
The paper calculates ruin probabilities for insurers with phase-type distributed claims.
problem Calculating ruin probabilities for insurers with specific claim distributions.
method Change-of-measure technique applied to phase-type distributed claim amounts.
result The mixture of Erlangs best fits real-world loss data, improving risk assessment.
The paper introduces metrics to rank potential outcomes for better decision-making.
problem Optimal action selection in uncertain situations using causal reasoning.
method Introducing two new metrics: probabilities of potential outcome ranking (PoR) and probability of achieving the best potential outcome (PoB). Establishing identification theorems and deriving bounds for these metrics, and presenting estimation methods.
result The estimators' finite-sample properties and their application to a real-world dataset are demonstrated.
GOCPD detects change points by maximizing the probability of two independent models.
problem Large false discovery rates in online change point detection methods.
method GOCPD uses ternary search to find change points by maximizing the probability of two independent models.
result GOCPD accelerates CPD with logarithmic complexity for single change point detection.
ESRLCM clusters similar responses, more broadly than traditional models.
problem Clustering multivariate categorical data with common response patterns.
method Bayesian Equivalence Set Restricted Latent Class Model (ESRLCM).
result ESRLCM identifies clusters with similar item response probabilities.
Deep neural networks enjoy a powerful representation and have proven effective in a number of applications. However, recent advances show that deep neural networks are vulnerable to adversarial attacks incurred by the so-called adversarial examples. Although the adversarial example is only slightly different from the i…
We construct the term structure of the (forward-looking, US market) equity risk premium from SPX option chains. The method is "model-light". Risk-neutral probability densities are estimated by fitting -component Gaussian mixture models to option quotes, where is a small integer (here 4 or 5). These densities are…
The paper critiques ε-fairness, showing it can lead to unfair outcomes and proposes a utility-based approach.
problem The limitations of probabilistic fairness metrics in real-world contexts.
method Utility-based approach to measure fairness, addressing the issue of unavailable data on false negatives.
result A utility-based approach uncovers necessary actions to achieve true fairness, contrasting with traditional probability-based evaluations.
In this paper, we propose to tackle the problem of reducing discrepancies between multiple domains referred to as multi-source domain adaptation and consider it under the target shift assumption: in all domains we aim to solve a classification problem with the same output classes, but with labels' proportions differing…
Study proves consistency of spectral clustering on hierarchical networks.
problem Consistency of spectral clustering on hierarchical stochastic block models.
method Recursive bi-partitioning algorithm based on Fiedler vector of graph Laplacian.
result Strong consistency of the method under various model parameters.
Bayesian neural networks are compressed using feature and weight pruning based on posterior inclusion probabilities.
problem Efficiently compressing Bayesian neural networks to reduce computation cost and improve generalizability.
method Bayesian model selection principles are applied to obtain posterior inclusion probabilities for pruning and feature selection.
result Pruned models show better generalizability on simulated and real-world data.
Although many successful ensemble clustering approaches have been developed in recent years, there are still two limitations to most of the existing approaches. First, they mostly overlook the issue of uncertain links, which may mislead the overall consensus process. Second, they generally lack the ability to incorpora…
What do binary (or probabilistic) forecasting abilities have to do with overall performance? We map the difference between (univariate) binary predictions, bets and "beliefs" (expressed as a specific "event" will happen/will not happen) and real-world continuous payoffs (numerical benefits or harm from an event) and sh…
This paper presents a kernel-based discriminative learning framework on probability measures. Rather than relying on large collections of vectorial training examples, our framework learns using a collection of probability distributions that have been constructed to meaningfully represent training data. By representing …
This work broadens optimal transport map estimation theory to stochastic settings.
problem Existing theory for optimal transport map estimation is restricted to deterministic maps under specific conditions.
method Introduces a novel metric for evaluating stochastic maps, develops computationally efficient estimators with robust guarantees.
result First general-purpose theory for map estimation compatible with real-world stochastic applications.
Estimates joint probability distribution from 1-way marginals using low-rank tensors and random projections.
problem Nonparametric estimation of joint probability mass function (PMF) from limited data.
method Low-rank tensor decomposition and random projections to link data to PMF estimation.
result Estimates joint density from 1-way marginals using transformed space and novel algorithm.
Exact Bayesian inference for discrete models using probability generating functions.
problem Discrete statistical models with infinite support and continuous priors.
method Probabilistic programming language with automatic differentiation and probability generating functions.
result Genfer tool provides exact solutions for a wide range of inference problems.
New technique for multiple-source adaptation without density estimation.
problem Multiple-source adaptation problem.
method Discriminative technique that uses conditional probabilities from unlabeled data.
result Our technique outperforms previous generative solutions and other domain adaptation baselines.
This paper introduces Probability Engineering to improve deep learning models.
problem Challenges in traditional probabilistic modeling for AI applications.
method Treats learned probability distributions as engineering artifacts and actively modifies them.
result Improves robustness, efficiency, adaptability, and trustworthiness of deep learning models.
Evidence Networks simplify Bayesian model comparison for complex models.
problem Bayesian model comparison challenges with intractable likelihoods or priors.
method Loss functions and neural networks for fast, amortized estimation of Bayes factors.
result Evidence Networks provide accurate and scalable Bayes factor estimation.
A novel notion of barycenter for probability distributions based on optimal weak mass transportstat.ML
A new method for averaging probability distributions based on optimal weak mass transport.
problem Averaging probability distributions in a geometric way.
method Weak barycenters based on optimal weak mass transport.
result Extracts common geometric information shared by all input distributions.
Framework improves classifier calibration under differential privacy for domain shift.
problem Improving classifier calibration under domain shift with privacy constraints.
method Differential privacy framework for adapting recalibration algorithms.
result Novel accuracy temperature scaling algorithm outperforms existing methods on private datasets.