Paper tackles noisy label distributions in machine learning.
problem Learning from noisy label distributions where true labels are distorted by unknown noise.
method Proposes a probabilistic model with hidden noise parameters and learns it using variational Bayesian methods.
result The proposed model outperforms existing methods in true label estimation.
New method improves false-/true-positive-rate estimation in fraud detection with noisy labels.
problem Estimating FPR/TPR in fraud detection with class-conditional label noise.
method Directly cleaning model's validation data to de-correlate cleaning error with model scores.
result Improves accuracy of FPR/TPR estimates, especially in asymmetric label noise scenarios.
New approach models noisy labels without assuming a unique true label.
problem Aggregating noisy labels in ambiguous scenarios.
method Introduces a novel latent distribution assumption.
result The new assumption is more suitable for tasks with ambiguous labels.
BCCP uses bandit feedback to provide reliable predictions with limited labeled data.
problem Limited labeled data and bandit feedback challenge online set-valued classification.
method BCCP uses stochastic gradient descent to train model and make set-valued inferences with unbiased estimation of true label.
result BCCP offers coverage guarantees on a class-specific granularity.
Method teaches students without teachers, estimating true labels from crowdsourcing.
problem Teaching without access to true labels.
method Apply crowdsourcing techniques to estimate true labels and student models for iterative teaching.
result Teaching performance is particularly effective for low-level students.
Paper proposes a new PLL framework with a progressive identification algorithm.
problem Weakly supervised learning with partial labels.
method Flexible model and optimization algorithm for PLL, progressive identification algorithm.
result Established an estimation error bound and set new state of the art.
In many real-world classification problems, the labels of training examples are randomly corrupted. Most previous theoretical work on classification with label noise assumes that the two classes are separable, that the label noise is independent of the true class label, or that the noise proportions for each class are …
Method learns true labels from noisy annotators using regularization.
problem Learning from noisy labels in supervised learning.
method Regularized estimation of annotator confusion matrices.
result Method outperforms state-of-the-art methods in image classification.
The paper studies how to use AI-generated labels in econometrics to avoid bias.
problem Small misclassification errors in AI-generated labels can lead to large biases in econometric estimators.
method The paper proposes a coupled-label bootstrap method to correct bias and deliver valid inference.
result The coupled-label bootstrap method is valid without the strong independence condition between true and imputed labels.
We tackle biased complementary labels in classification problems.
problem Learning with biased complementary labels.
method Estimating transition probabilities without bias, modifying loss functions, and extending deep neural networks.
result The proposed method converges to optimal classification with true labels.
This paper introduces a new learning method for classification without true labels.
problem Learning with only complementary labels, not true class labels.
method Derives a novel framework for arbitrary losses and models, using unbiased risk estimation.
result Demonstrates improved risk estimator through non-negative correction and gradient ascent.
In most classification tasks there are observations that are ambiguous and therefore difficult to correctly label. Set-valued classifiers output sets of plausible labels rather than a single label, thereby giving a more appropriate and informative treatment to the labeling of ambiguous instances. We introduce a framewo…
New framework for learning from various weak supervision types.
problem Scarcity of labeled data in real-world problems.
method Probabilistic framework based on maximum likelihood principle for deep neural networks.
result General method for learning from noisy labels, complementary labels, and coarse-grained labels.
UREs lead to overfitting in complex models, especially in complementary label learning.
problem Overfitting in weakly supervised learning with complementary labels.
method Proposed a surrogate complementary loss (SCL) framework to reduce gradient variance.
result SCL mitigates overfitting and improves URE-based methods.
Paper tackles online adaptation to changing label distributions.
problem Adapting machine learning models to changing label distributions in real-world settings.
method Leverages novel analysis to show estimation of expected test loss is possible without true labels. Proposes adaptation algorithms inspired by classical online learning techniques.
result Empirically verified that OGD is particularly effective and robust to various label shift scenarios.
Generative Augmented Inference improves AI-generated data for causal inference.
problem Challenges in using AI-generated annotations for reliable causal inference.
method Generative Augmented Inference (GAI) treats AI outputs as informative features for learning true labels, flexibly modeling the relationship using nonparametric methods.
result GAI significantly reduces estimation error and improves confidence interval quality compared to human-only and PPI-based methods.
Paper proposes a method to adapt classifiers using complementary labels instead of true labels.
problem Training classifiers with true labels from the source domain is costly and sometimes impossible.
method Proposes a novel setting with complementary labels and a complementary label adversarial network (CLARINET).
result CLARINET significantly outperforms baselines on handwritten digits and object recognition tasks.
Crowdsourcing has become a primary means for label collection in many real-world machine learning applications. A classical method for inferring the true labels from the noisy labels provided by crowdsourcing workers is Dawid-Skene estimator. In this paper, we prove convergence rates of a projected EM algorithm for the…
Two algorithms reduce label complexity in machine learning using stratified sampling.
problem Reducing the number of true labels needed for machine learning evaluation.
method Proposes two algorithms that estimate strata properties and optimize evaluation accuracy.
result Demonstrates algorithms are rate optimal and reduce label complexity.
Detects harmful shifts without labels for model performance.
problem Detecting distribution shifts without access to labels.
method Uses a proxy derived from predictions of an error estimator.
result High power and false alarm control under various shifts.
New loss functions make deep nets robust to noisy labels.
problem Label noise in training data affects deep neural networks.
method Developed conditions for loss functions to be robust to label noise.
result Mean absolute value loss is inherently robust to label noise.
Develops methods to measure and reduce fairness in datasets with limited protected attribute labels.
problem Measuring and reducing fairness in datasets with limited protected attribute labels.
method Proposes methods to estimate fairness metrics and train models to limit fairness violations using probabilistic protected attribute labels.
result Our methods provide tighter bounds on true disparity and effectively reduce fairness violations with lesser fairness-accuracy trade-offs.
Crowdsourcing is a popular paradigm for effectively collecting labels at low cost. The Dawid-Skene estimator has been widely used for inferring the true labels from the noisy labels provided by non-expert crowdsourcing workers. However, since the estimator maximizes a non-convex log-likelihood function, it is hard to t…
Framework corrects noisy labels to improve DNN performance.
problem Performance degradation due to noisy labels in large-scale datasets.
method Joint optimization of DNN parameters and true labels estimation.
result Significantly outperforms state-of-the-art methods in experiments.
OASIS optimizes ER evaluation by reducing labelling needs with optimal sampling.
problem Extreme class imbalance in ER leads to high labelling costs.
method OASIS uses a biased instrumental distribution and Bayesian updates to focus on unlabelled items.
result OASIS estimates F-measure, precision, recall converge to true values with significant labelling reductions.
The paper resolves the paradox of using unlabeled data for treatment effect estimation.
problem Using unlabeled data to estimate propensity scores for treatment effect estimation.
method Proposes a simple procedure to reconcile the use of estimated propensity scores with the advice to use true propensity scores.
result Direct regression may be preferable to inverse-propensity weighting in many circumstances.
Neural networks can memorize random labels just as well as true labels.
problem Understanding how neural networks memorize random labels.
method Empirical experiments and similarity measurement of learned patterns.
result DNNs have a 'One way to Learn' but 'N ways to Memorize'.
ALIS uses probabilistic sampling to minimize true loss in active learning.
problem Efficiently choosing data points to label in active learning.
method Proposes ALIS algorithm with optimal sampling distribution.
result Derives upper bounds on true loss for any sampling procedure.
A cautious deep learning approach reduces overconfidence by predicting null sets.
problem Overconfidence in classifier predictions over the entire space.
method Constructing conformal prediction sets based on p(x∣y), not p(y∣x). result The approach reduces overconfidence and predicts null sets for uncertain cases.
Majority Vote is optimal for reliable data labeling under certain conditions.
problem Reliable data labeling requires aggregating multiple annotators' labels, but the optimality of Majority Vote is not well understood.
method Characterized conditions under which Majority Vote achieves the optimal label estimation error.
result Majority Vote optimally recovers labels for a given class distribution under tolerable annotation noise limits.
Classifier chains are popular and effective method to tackle a multi-label classification problem. The aim of this paper is to study the asymptotic properties of the chain model in which the conditional probabilities are of the logistic form. In particular we find conditions on the number of labels and the distribution…
Clarinet uses complementary labels to train classifiers with less source data.
problem Training classifiers with true-label data from source domain is costly.
method Proposes CLARINET to train classifiers with complementary-label source data and unlabeled target data.
result CLARINET significantly outperforms baselines in unsupervised domain adaptation.
Paper introduces a novel method for estimating model confidence in deep neural classifiers.
problem Reliable confidence estimation for deep neural classifiers in safety-critical applications.
method Proposes a novel target criterion (true class probability) and learns it from data with an auxiliary model.
result The proposed method outperforms strong baselines in various tasks and network architectures.
Response time improves alignment with diverse human preferences.
problem Standard aggregation of feedback ignores heterogeneity and anonymity.
method Augmenting feedback with response time data and modeling decisions with DDM.
result Estimator of heterogeneous preferences converges to true average preference.
Paper proposes an unbiased risk estimator for PLLAC, handling unseen classes.
problem Handling unseen classes in PLLAC where some classes are not present in the training set.
method Proposes an unbiased risk estimator that estimates the distribution of augmented classes by differentiating known classes from unlabeled data.
result The estimator provides theoretical guarantees and converges to true risk minimizer as data increases.
Method learns generative model structure from unlabeled data.
problem Curating labeled training data is a bottleneck in machine learning.
method Maximizes the ℓ1-regularized marginal pseudolikelihood of observed data. result The method identifies the true structure from unlabeled data with sublinear scaling.
Algorithm constructs prediction sets with PAC guarantees in label shift settings.
problem Reliable uncertainty quantification in the face of distribution shift.
method Estimates predicted probabilities and confusion matrix, then propagates uncertainty through Gaussian elimination to compute confidence intervals and construct prediction sets.
result Satisfies PAC guarantees and produces smaller, more informative prediction sets.
Graph neural networks often assume vertex labels are independent, but we show this is rarely true and propose a method to improve predictions.
problem Graph neural networks often assume vertex labels are conditionally independent given their neighborhood features, which is rarely true.
method We model the joint distribution of residuals on vertices with a parameterized multivariate Gaussian and estimate parameters by maximizing the marginal likelihood of the observed labels.
result Our method achieves substantially higher accuracy than competing baselines and can be interpreted as the strength of correlation among connected vertices.
Noise-Aware Conformal Prediction (NACP) calibrates CP for noisy labels.
problem Calibrating Conformal Prediction with noisy labels.
method Estimate conformal threshold from noisy labels using uniform noise coverage guarantee.
result Finite sample coverage guarantee for uniform noise remains effective in high-class tasks.
A new method for weakly supervised learning that improves model accuracy.
problem Training machine learning models with precise labels is expensive; weak supervision provides a low-cost alternative.
method Data consistent weak supervision algorithm that searches over classifiers to find plausible labelings, considering features of the training data and estimating labels for low/no coverage data.
result Empirically, the method significantly outperforms state-of-the-art weak supervision methods on text and image classification tasks.
Paper monitors DNN accuracy to enhance trustworthiness.
problem Varying DNN accuracy in practice and lack of ground truth labels.
method Post-hoc accuracy monitor model using Monte-Carlo dropout ensemble.
result Accuracy monitor provides close-to-true accuracy estimation.
Label noise in adversarial training leads to robust overfitting, explained and mitigated.
problem Label noise in adversarial training causes robust overfitting.
method Proposed a method to automatically calibrate labels.
result Consistent performance improvements across various models and datasets.
A multiple instance dictionary learning method using functions of multiple instances (DL-FUMI) is proposed to address target detection and two-class classification problems with inaccurate training labels. Given inaccurate training labels, DL-FUMI learns a set of target dictionary atoms that describe the most distincti…
Evaluating prediction models under covariate shift and selective labels
problem Model performance evaluation under distribution shift and selection bias
method Double machine learning
result Accurate estimation of target risk
A new learning method uses data to learn from large model sets.
problem Learning with large sets of candidate models where uniform convergence is hard.
method Data-dependent learning that incorporates empirical data less reliant on prior assumptions.
result Demonstrates improved generalization in various learning assumptions.
In many signal detection and classification problems, we have knowledge of the distribution under each hypothesis, but not the prior probabilities. This paper is aimed at providing theory to quantify the performance of detection via estimating prior probabilities from either labeled or unlabeled training data. The erro…
Generative-discriminative method improves label prediction and instance generation.
problem Difficulty in obtaining high-quality labeled instances.
method Generative-discriminative complementary learning method using CC-GAN.
result Improves accuracy in predicting ordinary labels and generating high-quality instances.
Gaussian Processes outperform other models in estimating uncertainty for radiology report observation detection.
problem Uncertainty quantification in automatic data labelling for semi-supervised learning in clinical NLP.
method Investigation of uncertainty estimates from various predictive models using NLPP and MMPCL metrics.
result Gaussian Processes provide superior performance in quantifying uncertainty for radiology report observation detection.