A new method for adapting to label shifts using class probability matching.
problem Adapting to label shifts where class probabilities differ between source and target domains.
method Class Probability Matching using Kernel Methods (CPMKM) framework.
result CPMKM outperforms existing methods on real datasets.
The paper tackles multi-label ranking with uncertain probabilities.
problem Making skeptical inferences for multi-label ranking with sets of probabilities.
method Assumes a convex set of probabilities (credal set) over labels and seeks set-valued predictions.
result Developed methods for making skeptical inferences in multi-label ranking with uncertain probabilities.
Method curates cost-effective, high-quality datasets using AI models.
problem Costly manual labeling of datasets.
method Probably Approximately Correct Labels (PACL) method.
result Curates high-quality datasets with low overall labeling error.
PPI uses predictions and weighting to infer from partially labeled data.
problem Valid inference with partially labeled data.
method Combines model-based predictions with bias correction from labeled data, using Horvitz-Thompson and Hájek corrections.
result IPW-adjusted PPI with estimated propensities performs similarly to known-probability case.
This paper introduces PM and PMLP to enhance SSL by considering probability density and cluster assumptions.
problem Insufficient utilization of unlabeled data in SSL.
method Introduces PM to discern similarity and PMLP to consider cluster assumption in label propagation.
result PMLP outperforms other methods in SSL tasks.
Logistic regression can handle noisy labels effectively when labels are imperfectly assigned by multiple experts.
problem Label noise in supervised classification due to manual labelling by multiple experts.
method Using approximate posterior probabilities of class membership from multiple experts to train logistic regression models.
result Logistic regression can be robust to label noise when classification difficulty is the only source of errors.
Paper introduces methods for more reliable probabilistic predictions with confidence intervals.
problem Inaccurate labeling of datasets due to unreliable probabilistic predictions from weak labeling functions.
method Proposes a methodology to provide confidence intervals for label probabilities using uncertainty sets of distributions.
result Improves reliability of probabilistic predictions and provides confidence intervals for label probabilities.
Mitigates spurious correlations without bias labels.
problem Spurious correlations bias model performance.
method Introduces a novel training objective and debiasing method DPR.
result DPR achieves state-of-the-art performance.
SJS model predicts label shifts in multinomial datasets.
problem Predicting label shifts in multinomial datasets.
method Sparse joint shift model for dataset shift.
result Valid predictions and class prior probabilities estimates.
Multi-label classification aims to classify instances with discrete non-exclusive labels. Most approaches on multi-label classification focus on effective adaptation or transformation of existing binary and multi-class learning approaches but fail in modelling the joint probability of labels or do not preserve generali…
Two strategies extend multi-label chaining for imprecise probability estimates.
problem Handling imprecise probability estimates in multi-label classification.
method Adapting multi-label chaining to use convex sets of distributions (credal sets).
result Adapted approaches produce relevant cautiousness on hard-to-predict instances.
This research tackles multiclass classification by introducing a method for label ranking.
problem The challenge is to predict a sorted list of labels based on input features.
method Inspired by ranking median regression, the One-Versus-One (OVO) approach is used to achieve optimal ranking.
result The OVO approach yields an optimal ranking of labels with overwhelming probability under noise conditions.
We propose to formulate multi-label learning as a estimation of class distribution in a non-linear embedding space, where for each label, its positive data embeddings and negative data embeddings distribute compactly to form a positive component and negative component respectively, while the positive component and nega…
Proposes PA-DSL for correcting noisy human labels in automated data labeling.
problem Noisy human labels in automated data labeling.
method Uses adjudicated cases to correct noisy human labels and debias analyses.
result Maintains nominal coverage and reduces RMSE by 10-17% relative to using only adjudicated labels.
In this paper, we study the classification problem in which we have access to easily obtainable surrogate for true labels, namely complementary labels, which specify classes that observations do \textbf{not} belong to. Let Y and Yˉ be the true and complementary labels, respectively. We first model the annotati…
We consider the problem of estimating the conditional probability of a label in time O(log n), where n is the number of possible labels. We analyze a natural reduction of this problem to a set of binary regression problems organized in a tree structure, proving a regret bound that scales with the depth of the tree. Mot…
Improved self-supervised learning using credal sets.
problem Lack of precise knowledge in pseudo-labels.
method Using credal sets (sets of probability distributions) for labeling unlabeled data.
result Competitive to superior performance in low-label scenarios.
Improves reliability of medical diagnosis uncertainty estimates.
problem Label uncertainty in medical diagnosis.
method Post-hoc alpha-calibration method for neural network classifiers. result Significantly enhances reliability of uncertainty estimates.
New framework for learning with class-conditional multi-label noise.
problem Class labels corrupted with conditional probabilities for multiple labels.
method Formalized as CCMN framework, established unbiased estimators, proved consistency with multi-label loss functions, implemented partial multi-label learning method.
result Effectiveness validated on multiple datasets and metrics.
NoiseRank reduces label noise without supervision, improving classification accuracy.
problem Label noise in datasets from noisy channels.
method NoiseRank uses Markov Random Fields to estimate and rank instances based on their noise probability.
result NoiseRank improves classification accuracy on noisy datasets.
Unsupervised Domain Adaptation (DA) is used to automatize the task of labeling data: an unlabeled dataset (target) is annotated using a labeled dataset (source) from a related domain. We cast domain adaptation as the problem of finding stable labels for target examples. A new definition of label stability is proposed, …
New method uses small perturbations to improve representation learning from few labels.
problem Stability issues and label scarcity in representation learning.
method Introduces small-perturbation ideology on representation probability distribution models.
result Proposed models show better performance in clustering compared to baseline methods.
In Domain Adaptation (DA), the category-relevant losses usually occupy a dominant position, while they are usually built with hard or soft labels in existing models. We observed that hard labels are overconfident due to hard samples existed, and soft labels are ambiguous as too many small noisy probabilities involved, …
A new method for online multi-label stream classification.
problem Challenges in classifying continuous data streams with concept drift and delayed labels.
method Online unsupervised incremental method based on self-organizing maps.
result The method is highly competitive in both stationary and concept drift scenarios.
In this paper, we study a classification problem in which sample labels are randomly corrupted. In this scenario, there is an unobservable sample with noise-free labels. However, before being observed, the true labels are independently flipped with a probability ρ∈[0,0.5), and the random label noise can be class-co…
In most classification tasks there are observations that are ambiguous and therefore difficult to correctly label. Set-valued classifiers output sets of plausible labels rather than a single label, thereby giving a more appropriate and informative treatment to the labeling of ambiguous instances. We introduce a framewo…
We study the effect of imperfect training data labels on the performance of classification methods. In a general setting, where the probability that an observation in the training dataset is mislabelled may depend on both the feature vector and the true label, we bound the excess risk of an arbitrary classifier trained…
Calibration without labels in multiple testing
problem Interpretable error probabilities in large-scale hypothesis testing
method Constructing pseudo-labels from spacings of ordered p-values result Finding that q-value can be severely miscalibrated Label smoothing improves model robustness against misspecification.
problem Improving model robustness against model misspecification.
method Introducing modified label smoothing (MLSLR) that maintains consistent probability estimation while modifying the loss function.
result MLSLR exhibits higher robustness against model misspecification than conventional label smoothing.
Improves confidence calibration in neural networks by smoothing labels based on class similarity.
problem Improving confidence calibration in deep neural networks for safety-critical applications.
method Proposes a novel label smoothing technique where label values are based on similarities with the reference class, using different similarity measurements.
result Consistently outperforms state-of-the-art calibration techniques on various datasets and network architectures.
New CPS model tackles conditional probability shift in machine learning.
problem Discrepancy between source and target distributions in machine learning.
method Conditional Probability Shift Model (CPSM) using multinomial regression and EM algorithm.
result Superior balanced classification accuracy on target data compared to existing methods.
Paper develops proper, lower-bounded losses for weakly supervised classification.
problem Weakly supervised classification with corrupted labels.
method Representation theorem for proper losses, derived condition for lower-boundedness, generalized logit squeezing.
result Proper and lower-bounded losses for weak-label learning.
The paper tackles skeptical binary inferences in multi-label problems with sets of probabilities.
problem Making distributionally robust, skeptical inferences for multi-label problems.
method Study of distributionally robust, skeptical inferences for multi-label problems using Hamming loss.
result Skeptical inferences provide partial predictions for a sufficiently big set of probability distributions.
Algorithm constructs prediction sets with PAC guarantees in label shift settings.
problem Reliable uncertainty quantification in the face of distribution shift.
method Estimates predicted probabilities and confusion matrix, then propagates uncertainty through Gaussian elimination to compute confidence intervals and construct prediction sets.
result Satisfies PAC guarantees and produces smaller, more informative prediction sets.
Label shift refers to the phenomenon where the prior class probability p(y) changes between the training and test distributions, while the conditional probability p(x|y) stays fixed. Label shift arises in settings like medical diagnosis, where a classifier trained to predict disease given symptoms must be adapted to sc…
Friedman's method performs well for estimating class distributions.
problem Estimating prior class probabilities without label observations.
method Friedman's method and DeBias method for designing linear equation systems.
result Friedman's method performs well for binary and multi-class quantification.
In this paper we propose a measure of clustering quality or accuracy that is appropriate in situations where it is desirable to evaluate a clustering algorithm by somehow comparing the clusters it produces with ``ground truth' consisting of classes assigned to the patterns by manual means or some other means in whose v…
In many signal detection and classification problems, we have knowledge of the distribution under each hypothesis, but not the prior probabilities. This paper is aimed at providing theory to quantify the performance of detection via estimating prior probabilities from either labeled or unlabeled training data. The erro…
Classifier chains are popular and effective method to tackle a multi-label classification problem. The aim of this paper is to study the asymptotic properties of the chain model in which the conditional probabilities are of the logistic form. In particular we find conditions on the number of labels and the distribution…
Distillation improves simple models by approximating complex labels.
problem Why does distillation improve simple models?
method Statistical perspective on distillation, connecting to extreme multiclass retrieval.
result Distillation helps by approximating underlying class-probabilities, reducing bias and variance.
Paper recovers top-two answers and confusion probability in multi-choice crowdsourcing.
problem Recovering top-two answers and confusion probability in multi-choice crowdsourcing tasks.
method Proposes a two-stage inference algorithm based on a model quantifying task difficulty and worker reliability.
result Achieves minimax optimal convergence rate and outperforms other algorithms in synthetic and real data experiments.
New approach predicts event probabilities for better event detection.
problem Class imbalance and inaccurate event detection in time series analysis.
method Regression-based approach to predict probability densities at event locations.
result Regression-based approaches outperform segmentation-based methods.
In label-noise learning, \textit{noise transition matrix}, denoting the probabilities that clean labels flip into noisy labels, plays a central role in building \textit{statistically consistent classifiers}. Existing theories have shown that the transition matrix can be learned by exploiting \textit{anchor points} (i.e…
New framework learns from partial feedback in multi-label tasks.
problem Learning from one-sided feedback in multi-label tasks.
method Probably Approximately Correct (PAC) framework for set functions.
result Achieves optimal sample complexity in realizable case, multiplicative approximation guarantees in agnostic case.
A group of transition probability functions form a Shannon's channel whereas a group of truth functions form a semantic channel. Label learning is to let semantic channels match Shannon's channels and label selection is to let Shannon's channels match semantic channels. The Channel Matching (CM) algorithm is provided f…
New method corrects skewed confidence for PbN classification.
problem Weakly supervised binary classification with biased negative data.
method Corrects skewed confidence in negative data to improve classifier.
result Reduces distortion in posterior probability for PbN classification.
New method for PU learning with instance-dependent propensity scores.
problem Learning from positive and unlabeled data with instance-dependent labeling.
method Empirical risk minimization of joint risk function, alternating optimization of posterior probability and propensity score.
result The method achieves comparable or better performance than state-of-the-art methods.
Fairness constraints improve exact recovery in structured prediction models.
problem Exact recovery of fair binary node labels from noisy observations.
method Analyzed Globerson et al. (2015) model with fairness constraints and improved exact recovery for graphs with poor expansion properties.
result Fairness constraints improve the probability of exact recovery from noisy observations.