Universally valid ground truth is almost impossible to obtain or would come at a very high cost. For supervised learning without universally valid ground truth, a recommended approach is applying crowdsourcing: Gathering a large data set annotated by multiple individuals of varying possibly expertise levels and inferri…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “ground truth inference”
RealCause provides a realistic benchmark for causal inference.
problem Lack of a reliable benchmark for comparing causal effect estimators.
method Flexible generative models to create a benchmark that is both ground-truth and realistic.
result Evaluation of over 1500 causal estimators provides evidence for choosing hyperparameters using predictive metrics.
Ranking a set of objects involves establishing an order allowing for comparisons between any pair of objects in the set. Oftentimes, due to the unavailability of a ground truth of ranked orders, researchers resort to obtaining judgments from multiple annotators followed by inferring the ground truth based on the collec…
Algebraic Ground Truth Inference: Non-Parametric Estimation of Sample Errors by AI Algorithmsstat.ML
Estimates classifier errors without ground truth using algebraic geometry.
problem Lack of ground truth in real-world production systems.
method Non-parametric estimation using algebraic geometry to solve the self-assessment problem.
result Accuracy estimators are better than one part in a hundred.
Autonomy and adaptation of machines requires that they be able to measure their own errors. We consider the advantages and limitations of such an approach when a machine has to measure the error in a regression task. How can a machine measure the error of regression sub-components when it does not have the ground truth…
Polymarket-v1 Database tracks 1.2B trades across 1.3M markets with 100% ground-truth direction.
problem Lack of ground-truth data in prediction markets archives.
method Ground-truth archive of 1.2B trades from Polymarket's CTF Exchange.
result Ground-truth data reveals systematic errors in microstructure metrics.
New method falsifies causal discovery results without ground truth.
problem Evaluation of causal discovery algorithms without ground truth data.
method Detects incompatibilities between causal graphs learned on different subsets of variables.
result Detection of incompatibilities can falsify wrongly inferred causal relations.
Neural Machine Translation (NMT) generates target words sequentially in the way of predicting the next word conditioned on the context words. At training time, it predicts with the ground truth words as context while at inference it has to generate the entire sequence from scratch. This discrepancy of the fed context l…
Self-supervised methods learn from noisy data alone, useful for imaging problems.
problem Inferring signals from noisy and incomplete observations.
method Learning a solver from measurement data alone, without ground-truth references.
result Self-supervised methods can learn meaningful estimates from noisy data.
A method for inferring ground-truth signals from degraded sensor data.
problem Inferring ground-truth signals from multiple degraded sensor signals.
method Iterative correction of degraded signals using a Bayesian multi-sensor data fusion method.
result The method effectively infers ground-truth signals from noisy and degraded sensor data.
New method accounts for uncertainty in medical AI evaluations.
problem Uncertainty in ground truth affects AI model performance estimates.
method Statistical aggregation approach to infer probabilities of medical conditions.
result Performance estimates are significantly lower when uncertainty is accounted for.
New metric improves latent dynamics inference from neural data.
problem Limitations of co-smoothing in predicting latent dynamics.
method Few-shot co-smoothing to assess latent dynamics.
result High co-smoothing models often have extraneous dynamics, which few-shot co-smoothing detects.
PPI uses proxy data to improve inference from limited labels across related tasks.
problem Statistical inference with limited labels across multiple related tasks.
method Prediction-powered inference framework that uses cross-task recalibration to improve power and accuracy.
result Cross-task recalibration can substantially reduce confidence interval widths when labels are scarce.
Misspecification-Aware Simulation-Based Inference via Side-Channel Guidance
problem Simulation-based inference (SBI) of latent parameters is hindered by simulator misspecification.
method Misspecification-Aware Simulation-Based Inference (MA-SBI) turns side-channel text into a posterior correction.
result MA-SBI matches the oracle posterior across 10 seeds and two backbones.
New method selects best HTE estimator without ground-truth treatment effects.
problem Selecting best HTE estimator from multiple candidates.
method Cross-fitted, exponentially weighted test statistic with two-way sample splitting.
result Empirically, reliable error control and reduced false selections.
Paper proposes a method to recover accurate labels from partially valid data in multi-label learning.
problem Tackles noisy supervision in multi-label learning with partially valid labels.
method Develops a two-stage method that estimates label enrichment and ground-truth confidences.
result Demonstrates improved performance over state-of-the-art PML methods.
Tests for classifier independence without ground truth labels.
problem Validation of classifier independence without ground truth labels.
method Exact solution for independent binary classifiers using algebraic geometry.
result Self-consistent test for classifier independence without ground truth labels.
RoPE framework calibrates misspecified simulators for reliable inference.
problem Misspecification compromises reliability of simulation-based inference.
method Data-driven calibration using optimal transport and a small calibration set.
result RoPE framework improves inference accuracy and uncertainty calibration.
A new score measures data reliability without ground truth.
problem Assessing reliability of datasets without access to ground truth.
method Define ground-truth-based orderings and propose Gram determinant score.
result Gram determinant score effectively captures data quality across diverse observation processes.
Paper recovers top-two answers and confusion probability in multi-choice crowdsourcing.
problem Recovering top-two answers and confusion probability in multi-choice crowdsourcing tasks.
method Proposes a two-stage inference algorithm based on a model quantifying task difficulty and worker reliability.
result Achieves minimax optimal convergence rate and outperforms other algorithms in synthetic and real data experiments.
Recently developed deep-learning-based denoisers often outperform state-of-the-art conventional denoisers such as the BM3D. They are typically trained to minimize the mean squared error (MSE) between the output image of a deep neural network (DNN) and a ground truth image. Thus, it is important for deep-learning-based …
Paper introduces metrics to evaluate missing data imputation without ground truth.
problem Handling missing data in time series without ground truth.
method Introduces Wasserstein distance (WD) and Jensen-Shannon divergence (JSD) as metrics to evaluate imputation quality.
result WD and JSD are effective metrics for assessing missing data imputation quality.
QUACKIE creates a new benchmark for NLP interpretability.
problem Evaluating NLP interpretability methods is challenging due to biased ground truths.
method Formulated a custom classification task from question-answering datasets, generating unbiased ground truths.
result Demonstrated the effectiveness of current interpretability methods on the new benchmark.
SAMPLR optimizes for ground truth in aleatoric parameters to avoid curriculum-induced covariate shift.
problem Curriculum learning shifts training distribution, leading to suboptimal policies in aleatoric settings.
method SAMPLR optimizes ground-truth utility function, avoiding curriculum-induced covariate shift.
result SAMPLR preserves optimality under ground-truth distribution, promoting robustness across various environments.
Unsupervised clustering can reproduce categorization systems if features and metrics are correctly selected.
problem Reproducing expert-provided categorization systems using unsupervised clustering.
method Investigated using toy datasets and real-world fund categorization. Used appropriate feature selection and a supervised Random Forest-based distance metric.
result Unsupervised clustering can reproduce ground truth classes if features and metrics are correctly selected.
Paper presents a machine learning method to improve significance tests for misspecified linear models.
problem Misspecification of linear assumptions in social science models leads to inaccurate significance levels.
method Apply machine learning to fit ground truth function, calculate linear approximation, and adjust the estimator.
result The method significantly outperforms linear regression for non-linear ground truth functions.
This paper proposes a set of criteria to evaluate the objectiveness of explanation methods of neural networks, which is crucial for the development of explainable AI, but it also presents significant challenges. The core challenge is that people usually cannot obtain ground-truth explanations of the neural network. To …
Generative model combines multi-dimensional annotations for more accurate ground truth estimation.
problem Inaccurate ground truth estimation from naive annotators' multi-dimensional annotations.
method Proposes a joint multi-dimensional model for global and time-series annotation fusion using Expectation-Maximization algorithm.
result More accurate ground truth estimates through joint modeling of multiple dimensions.
The problem of estimating event truths from conflicting agent opinions in a social network is investigated. An autoencoder learns the complex relationships between event truths, agent reliabilities and agent observations. A Bayesian network model is proposed to guide the learning process by modeling the relationship of…
Latent truth discovery, LTD for short, refers to the problem of aggregating ltiple claims from various sources in order to estimate the plausibility of atements about entities. In the absence of a ground truth, this problem is highly challenging, when some sources provide conflicting claims and others no claims at all.…
Deep neural networks (DNNs) are poorly calibrated when trained in conventional ways. To improve confidence calibration of DNNs, we propose a novel training method, distance-based learning from errors (DBLE). DBLE bases its confidence estimation on distances in the representation space. In DBLE, we first adapt prototypi…
Improves spatio-temporal forecasting by reducing errors between training and inference.
problem Accumulation of small errors in Seq2Seq models during inference due to different distributions of training and inference phases.
method Curriculum learning based on Temporal Progressive Growing Sampling to replace some ground-truth context with generated predictions.
result Better models long-term dependencies and outperforms baseline approaches on two datasets.
A popular approach for large scale data annotation tasks is crowdsourcing, wherein each data point is labeled by multiple noisy annotators. We consider the problem of inferring ground truth from noisy ordinal labels obtained from multiple annotators of varying and unknown expertise levels. Annotation models for ordinal…
New causal distances improve evaluation of causal discovery algorithms.
problem Evaluating causal discovery algorithms using graphical distances is limited.
method Defined causal distances based on causal distributions rather than graphical structure.
result Improved evaluation of causal discovery algorithms on synthetic and real-world datasets.
Preconditioned SGD accelerates convergence for ill-conditioned huge-scale matrix completion.
problem Recovering a low-rank matrix from incomplete data with high condition number.
method Preconditioned Stochastic Gradient Descent (SGD) for huge-scale online optimization.
result Preconditioned SGD converges to ε-accuracy in O(log(1/ε)) iterations, compared to O(κlog(1/ε)) for unpreconditioned SGD.
Obtaining enough labeled data to robustly train complex discriminative models is a major bottleneck in the machine learning pipeline. A popular solution is combining multiple sources of weak supervision using generative models. The structure of these models affects training label quality, but is difficult to learn with…
Study reveals issues with neural autoregressive models and proposes mode recovery cost.
problem Unreasonable affinity of neural autoregressive models to short and long sequences.
method Investigates modes of ground-truth, empirical, and decoding-induced distributions via mode recovery cost.
result Mode recovery cost varies depending on ground-truth distribution and impacts decoding-induced distribution.
It is widely believed that sharing gradients will not leak private training data in distributed learning systems such as Collaborative Learning and Federated Learning, etc. Recently, Zhu et al. presented an approach which shows the possibility to obtain private training data from the publicly shared gradients. In their…
New method estimates model performance bounds without ground truth labels.
problem Evaluation of weakly supervised models without direct access to ground truth labels.
method Formulates model evaluation as a partial identification problem and uses Fréchet bounds for performance estimation.
result Derives accurate and computationally efficient bounds for key metrics like accuracy, precision, recall, and F1-score.
In many security and healthcare systems, the detection and diagnosis systems use a sequence of sensors/tests. Each test outputs a prediction of the latent state and carries an inherent cost. However, the correctness of the predictions cannot be evaluated since the ground truth annotations may not be available. Our obje…
Noise affects the effectiveness of interpolating models, especially those with strong inductive biases.
problem The impact of noise on interpolating models with strong inductive biases.
method Analyzing linear and classification models with sparse ground truths, proving fast rates for interpolators.
result Strong inductive biases can lead to faster but noisier interpolators, contrary to intuition.
In machine learning the best performance on a certain task is achieved by fully supervised methods when perfect ground truth labels are available. However, labels are often noisy, especially in remote sensing where manually curated public datasets are rare. We study the multi-modal cadaster map alignment problem for wh…
When Stability Fails: Hidden Failure Modes Of LLMS in Data-Constrained Scientific Decision-Makingcs.LG
LLMs fail to match statistical ground truth despite stable run-to-run performance.
problem LLMs lack validation against statistical ground truth in automated scientific workflows.
method Introduced a behavioral evaluation framework for LLMs, separating four decision-making dimensions.
result LLMs can exhibit near-perfect stability but diverge from statistical ground truth.
FUSE improves verification quality without ground truth labels.
problem Verification of model outputs using imperfect judges and reward models.
method Ensembling verifiers without ground truth labels using spectral algorithms.
result FUSE matches or improves upon semi-supervised alternatives in test-time scaling experiments.
Improved generalization with iterative self-distillation using weighted ground-truth targets.
problem Improving generalization accuracy in neural networks.
method Iterative kernel regression with weighted ground-truth targets and regularization.
result Closed-form solution for optimal weighting parameter and efficient estimation.
Crowdsourcing has been proven to be an effective and efficient tool to annotate large datasets. User annotations are often noisy, so methods to combine the annotations to produce reliable estimates of the ground truth are necessary. We claim that considering the existence of clusters of users in this combination step c…
New research shows the maximum ℓ1-margin classifier doesn't adapt to sparse ground truths.
problem Understanding the limitations of the maximum ℓ1-margin classifier in high-dimensional settings.
method Analyzing convergence and prediction error rates of the maximum ℓ1-margin classifier.
result Proves tight upper and lower bounds for prediction error, showing benign overfitting.
EMHMM Simulation Studystat.ML
Eye Movement analysis with Hidden Markov Models (EMHMM) is a method for modeling eye fixation sequences using hidden Markov models (HMMs). In this report, we run a simulation study to investigate the estimation error for learning HMMs with variational Bayesian inference, with respect to the number of sequences and the …