arXiv research

New truthful calibration errors improve model ranking in multiclass prediction.

New calibration measure SSCE ensures truthful prediction, unlike existing measures.

A new perfectly truthful calibration measure improves prediction reliability.

Universally valid ground truth is almost impossible to obtain or would come at a very high cost. For supervised learning without universally valid ground truth, a recommended approach is applying crowdsourcing: Gathering a large data set annotated by multiple individuals of varying possibly expertise levels and inferri…

Paper proposes a method to recover accurate labels from partially valid data in multi-label learning.

Tests for classifier independence without ground truth labels.

A new score measures data reliability without ground truth.

RealCause provides a realistic benchmark for causal inference.

Recently developed deep-learning-based denoisers often outperform state-of-the-art conventional denoisers such as the BM3D. They are typically trained to minimize the mean squared error (MSE) between the output image of a deep neural network (DNN) and a ground truth image. Thus, it is important for deep-learning-based …

Estimates classifier errors without ground truth using algebraic geometry.

Ranking a set of objects involves establishing an order allowing for comparisons between any pair of objects in the set. Oftentimes, due to the unavailability of a ground truth of ranked orders, researchers resort to obtaining judgments from multiple annotators followed by inferring the ground truth based on the collec…

Paper introduces metrics to evaluate missing data imputation without ground truth.

QUACKIE creates a new benchmark for NLP interpretability.

Unsupervised clustering can reproduce categorization systems if features and metrics are correctly selected.

This paper proposes a set of criteria to evaluate the objectiveness of explanation methods of neural networks, which is crucial for the development of explainable AI, but it also presents significant challenges. The core challenge is that people usually cannot obtain ground-truth explanations of the neural network. To …

The problem of estimating event truths from conflicting agent opinions in a social network is investigated. An autoencoder learns the complex relationships between event truths, agent reliabilities and agent observations. A Bayesian network model is proposed to guide the learning process by modeling the relationship of…

New method accounts for uncertainty in medical AI evaluations.

Generative model combines multi-dimensional annotations for more accurate ground truth estimation.

New method falsifies causal discovery results without ground truth.

SAMPLR optimizes for ground truth in aleatoric parameters to avoid curriculum-induced covariate shift.

Autonomy and adaptation of machines requires that they be able to measure their own errors. We consider the advantages and limitations of such an approach when a machine has to measure the error in a regression task. How can a machine measure the error of regression sub-components when it does not have the ground truth…

We investigate the problem of truth discovery based on opinions from multiple agents who may be unreliable or biased. We consider the case where agents' reliabilities or biases are correlated if they belong to the same community, which defines a group of agents with similar opinions regarding a particular event. An age…

Self-supervised methods learn from noisy data alone, useful for imaging problems.

Preconditioned SGD accelerates convergence for ill-conditioned huge-scale matrix completion.

Study reveals issues with neural autoregressive models and proposes mode recovery cost.

It is widely believed that sharing gradients will not leak private training data in distributed learning systems such as Collaborative Learning and Federated Learning, etc. Recently, Zhu et al. presented an approach which shows the possibility to obtain private training data from the publicly shared gradients. In their…

New method estimates model performance bounds without ground truth labels.

Polymarket-v1 Database tracks 1.2B trades across 1.3M markets with 100% ground-truth direction.

Noise affects the effectiveness of interpolating models, especially those with strong inductive biases.

A new mechanism reduces expert belief regret in online forecasting.

In machine learning the best performance on a certain task is achieved by fully supervised methods when perfect ground truth labels are available. However, labels are often noisy, especially in remote sensing where manually curated public datasets are rare. We study the multi-modal cadaster map alignment problem for wh…

LLMs fail to match statistical ground truth despite stable run-to-run performance.

We consider a fundamental dynamic allocation problem motivated by the problem of $\textit{securities lending}$ in financial markets, the mechanism underlying the short selling of stocks. A lender would like to distribute a finite number of identical copies of some scarce resource to $n$ clients, each of whom has a priv…

FUSE improves verification quality without ground truth labels.

Improved generalization with iterative self-distillation using weighted ground-truth targets.

The paper analyzes how noise affects distances in high-dimensional data and when they remain useful.

New research shows the maximum ℓ1-margin classifier doesn't adapt to sparse ground truths.

In the modern era, abundant information is easily accessible from various sources, however only a few of these sources are reliable as they mostly contain unverified contents. We develop a system to validate the truthfulness of a given statement together with underlying evidence. The proposed system provides supporting…

A new method finds DAG models without ground truth.

A method for inferring ground-truth signals from degraded sensor data.

Regression analysis is a standard supervised machine learning method used to model an outcome variable in terms of a set of predictor variables. In most real-world applications we do not know the true value of the outcome variable being predicted outside the training data, i.e., the ground truth is unknown. It is hence…

Pairwise Label Smoothing improves deep model generalization by reducing overconfidence.

Separating an audio scene such as a cocktail party into constituent, meaningful components is a core task in computer audition. Deep networks are the state-of-the-art approach. They are trained on synthetic mixtures of audio made from isolated sound source recordings so that ground truth for the separation is known. Ho…

Framework improves self-play for cooperative multi-agent learning.

PoGDiff improves diffusion models on imbalanced datasets.

Unsupervised recalibration (URC) is a general way to improve the accuracy of an already trained probabilistic classification or regression model upon encountering new data while deployed in the field. URC does not require any ground truth associated with the new field data. URC merely observes the model's predictions a…

Non-negative matrix factorization is a basic tool for decomposing data into the feature and weight matrices under non-negativity constraints, and in practice is often solved in the alternating minimization framework. However, it is unclear whether such algorithms can recover the ground-truth feature matrix when the wei…

A network of agents attempt to learn some unknown state of the world drawn by nature from a finite set. Agents observe private signals conditioned on the true state, and form beliefs about the unknown state accordingly. Each agent may face an identification problem in the sense that she cannot distinguish the truth in …