PTBCC improves accuracy in multi-class annotation aggregation by learning from prototype confusion matrices.
problem Inaccurate and insufficient confusion matrices for annotators in multi-class classification tasks.
method PTBCC (ProtoType learning-driven Bayesian Classifier Combination) uses prototype confusion matrices to capture annotator expertise.
result PTBCC achieves up to 15% accuracy improvement and 3% higher average accuracy compared to existing methods.
Method learns true labels from noisy annotators using regularization.
problem Learning from noisy labels in supervised learning.
method Regularized estimation of annotator confusion matrices.
result Method outperforms state-of-the-art methods in image classification.
Data collection and annotation are time-consuming in machine learning, expecially for large scale problem. A common approach for this problem is to transfer knowledge from a related labeled domain to a target one. There are two popular ways to achieve this goal: adversarial learning and self training. In this article, …
This work improves deep learning from noisy crowdsourced labels.
problem Learning label correction and neural classifier from noisy crowdsourced data.
method Coupled Cross-Entropy Minimization (CCEM) with identifiability and regularization.
result The CCEM criterion correctly identifies annotators' confusion and neural classifier under realistic conditions.
Ideally, what confuses neural network should be confusing to humans. However, recent experiments have shown that small, imperceptible perturbations can change the network prediction. To address this gap in perception, we propose a novel approach for learning robust classifier. Our main idea is: adversarial examples for…
Solves TOD systems' query annotation problem without explicit annotations.
problem Training TOD systems without explicit KB query annotation.
method Reinforcement learning (RL) and pipelined approach for query prediction and system training.
result Improved RL agent with modifications for TOD tasks.
Paper presents a method to train NER models without labelled data using weak supervision.
problem Dealing with NER performance drop in new domains without labelled data.
method Weak supervision through automatic annotation and hidden Markov model integration.
result Improvement of about 7 percentage points in entity-level F1 scores. Develops rough set classifiers using confusion matrices.
problem Evaluating classifier quality in machine learning.
method Combines rough set theory with confusion matrices.
result Defines indices and classifiers based on rough confusion matrices.
The paper investigates how neural network width and depth impact training speed through gradient confusion.
problem Gradient confusion in neural networks during training.
method Formal analysis using gradient confusion, theoretical and experimental results.
result Increasing network width reduces gradient confusion, leading to faster training.
The article explains how to estimate confusion matrices for classifiers using unlabeled data.
problem Estimating sensitivity and specificity of binary medical diagnostic tests without gold standard tests.
method Modifying diagnostic test solutions to estimate confusion matrices for classifiers on unlabeled data.
result The approach can be used to estimate accuracy statistics for supervised or unsupervised binary classifiers on unlabeled data.
Deep active learning aims to reduce annotation costs while leveraging DL's powerful learning.
problem Traditional AL struggles with few labeled samples, while DL requires large datasets.
method Formal classification and systematic overview of existing DAL work.
result DAL can reduce annotation costs while maintaining DL's learning capabilities.
Framework detects fake news using weak social signals from multiple sources.
problem Lack of annotated data for early fake news detection.
method Jointly uses weak social signals and clean data to train deep neural networks in a meta-learning framework.
result Framework outperforms state-of-the-art baselines for early fake news detection.
Paper recovers top-two answers and confusion probability in multi-choice crowdsourcing.
problem Recovering top-two answers and confusion probability in multi-choice crowdsourcing tasks.
method Proposes a two-stage inference algorithm based on a model quantifying task difficulty and worker reliability.
result Achieves minimax optimal convergence rate and outperforms other algorithms in synthetic and real data experiments.
Paper proposes MCC to reduce class confusion for versatile DA.
problem Class confusion in DA methods limits their performance across different scenarios.
method Introduces Minimum Class Confusion (MCC) loss function to handle various DA scenarios.
result MCC significantly improves performance on diverse DA scenarios, including Multi-Source and Multi-Target DA.
In this work, we propose a PAC-Bayes bound for the generalization risk of the Gibbs classifier in the multi-class classification framework. The novelty of our work is the critical use of the confusion matrix of a classifier as an error measure; this puts our contribution in the line of work aiming at dealing with perfo…
Paper proposes a universal probabilistic model for handling instance-dependent label noise.
problem Instance-dependent label noise in data quality challenges DNN training robustness.
method Categorizes instances into confusing and unconfusing, proposes a probabilistic model.
result Significant improvements in robustness over state-of-the-art methods on various datasets.
ConfusionFlow visualizes classifier confusion over time for model comparison.
problem Insufficient performance analysis of classifiers.
method Interactive, model-agnostic visualization tool combining confusion matrices and temporal analysis.
result ConfusionFlow facilitates detailed, comparative analysis of classifier performance over time.
Proposes a new method for deep ensembles that improves accuracy and calibration.
problem Improving accuracy and calibration of deep ensembles.
method Estimates confusion matrices of ensemble members and weighs them according to their inferred performance.
result Empirically shows superiority of soft Dawid Skene over ensemble averaging.
Developed a neural topic model for classifying COVID-19 disinformation.
problem Tackles the challenge of disinformation during the COVID-19 pandemic.
method Classification-aware neural topic model (CANTM) for COVID-19 disinformation.
result Demonstrated the effectiveness of CANTM in classifying COVID-19 disinformation.
New method quantifies classifier uncertainty, revealing large variability in performance metrics.
problem Uncertainty in classifier performance metrics due to small data sets.
method Probability model of the confusion matrix to quantify uncertainty.
result Large uncertainties in classification performance metrics can lead to misleading conclusions.
Clarifies confusion on feature relevance quantification in explainable AI.
problem Confusion between observational and interventional conditional probabilities in feature relevance quantification.
method Uses Shapley values and clarifies the distinction between observational and interventional conditional probabilities based on Pearl's causality theory.
result Unconditional expectations are the right notion for dropping features, contradicting theoretical justification of SHAP.
This paper investigates how machine learning APIs change over time and proposes an efficient method to monitor these changes.
problem Understanding and assessing changes in machine learning APIs over time.
method Systematic investigation of ML API shifts, proposing a principled adaptive sampling algorithm (MASA) for efficient estimation of confusion matrix shifts.
result MASA can accurately estimate confusion matrix shifts using up to 90% fewer samples compared to random sampling.
Machine learning experiments often contain errors, especially in confusion matrices and statistical tests.
problem Errors in machine learning experiments, particularly in confusion matrices and statistical tests.
method Analyzed 49 papers describing 2456 experiments, checking for errors in confusion matrices and statistical significance.
result 22 out of 49 papers contain demonstrable errors, with 7 statistical and 16 related to confusion matrix inconsistency.
EAST aligns neural network classifiers with user-defined evaluation metrics.
problem Mismatch between neural network training and evaluation metrics leads to suboptimal performance.
method EAST uses dynamic thresholding, soft-set confusion matrix, and annealing to align neural network predictions with target evaluation metrics.
result EAST improves alignment between training objectives and evaluation metrics, outperforming existing methods.
Improved satellite image captions enhance descriptiveness without large models.
problem Extracting meaningful text from satellite imagery.
method Evaluated seven models on a large benchmark, extended vocabulary, and introduced a novel confusion matrix.
result Reduced model size by 100x without sacrificing accuracy, offering new deployment opportunities.
This paper revisits the problem of analyzing multiple ratings given by different judges. Different from previous work that focuses on distilling the true labels from noisy crowdsourcing ratings, we emphasize gaining diagnostic insights into our in-house well-trained judges. We generalize the well-known DawidSkene model…
Consistent algorithms for multiclass learning with complex metrics and constraints.
problem Learning with complex performance metrics and constraints.
method General framework for designing consistent algorithms by viewing the problem as an optimization over feasible confusion matrices.
result Rates of convergence to the optimal (feasible) classifier, showing asymptotic consistency.
Study shows annotation instrument design affects model performance in hate speech detection.
problem Impact of annotation instrument design on model performance in hate speech detection.
method Collected annotations from five experimental conditions of an annotation instrument, fine-tuned BERT models on each dataset, evaluated performance on holdout portion.
result Significant differences in model performance and annotations across conditions.
IRT metrics improve model evaluation by assessing latent characteristics.
problem Limitations of classic metrics like precision and F1.
method Introducing psychometric metrics like Item Response Theory (IRT).
result IRT complements classical metrics, offering new insights.
Proposes FACT, a diagnostic for understanding group fairness trade-offs.
problem Group fairness notions often conflict with each other, requiring a cost in model performance.
method Characterizes trade-offs via the fairness-confusion tensor and optimizes accuracy and fairness objectives.
result Demonstrates the use of FACT on synthetic and real datasets to understand accuracy-fairness trade-offs.
WAPPO optimizes feature distributions for better visual transfer in RL.
problem Improving visual transfer in reinforcement learning.
method WAPPO uses Wasserstein Confusion to minimize feature distribution distance.
result WAPPO outperforms previous methods in visual transfer across different environments.
Exploits class similarity for better machine learning models with confidence labels and projective loss functions.
problem Poor model performance due to confusing similar classes.
method Exploits class similarity with confidence labels and projective loss functions.
result Improved model performance on noisy labels.
InstanceFlow visualizes classifier confusion over training epochs.
problem Limited model interpretability through aggregate performance measures.
method Dual-view visualization tool showing instance-level learning behavior.
result Allows temporal analysis of training process and instance-level performance.
Active learning selects both observations and annotation precision for Gaussian Processes.
problem Costly annotation in supervised learning.
method Proposes an active learning algorithm that selects observations and annotation precision, using a modified BALD objective.
result Empirically shows the benefits of adjusting annotation precision in active learning.
Unified approach optimizes neural network training for various metrics.
problem Training and evaluation of neural network binary classifiers often use different metrics.
method Combines differentiable approximation and probabilistic soft sets.
result Effective in optimizing for metrics like F1-Score across various domains.
In this work we addressed the issue of applying a stochastic classifier and a local, fuzzy confusion matrix under the framework of multi-label classification. We proposed a novel solution to the problem of correcting label pairwise ensembles. The main step of the correction procedure is to compute classifier-specific c…
One of the problems on the way to successful implementation of neural networks is the quality of annotation. For instance, different annotators can annotate images in a different way and very often their decisions do not match exactly and in extreme cases are even mutually exclusive which results in noisy annotations a…
Generative model combines multi-dimensional annotations for more accurate ground truth estimation.
problem Inaccurate ground truth estimation from naive annotators' multi-dimensional annotations.
method Proposes a joint multi-dimensional model for global and time-series annotation fusion using Expectation-Maximization algorithm.
result More accurate ground truth estimates through joint modeling of multiple dimensions.
Survey on AL strategies for cost-effective annotation in classification.
problem Real-world AL challenges due to human annotators' limitations.
method Categorizes 60 real-world AL strategies considering multiple annotators, query types, and cost schemes.
result General real-world AL strategy introduced for categorization of 60 strategies.
RAD improves robustness to domain annotation noise without explicit domain annotations.
problem Robustness to domain annotation noise in training data.
method Regularized Annotation of Domains (RAD) for last layer retraining.
result RAD outperforms state-of-the-art methods even with 5% noise in training data.
Accurate annotation of medical image is the crucial step for image AI clinical application. However, annotating medical image will incur a great deal of annotation effort and expense due to its high complexity and needing experienced doctors. To alleviate annotation cost, some active learning methods are proposed. But …
The study challenges the notion that partial data annotation is inferior, suggesting it can sometimes outperform complete annotation.
problem The inefficiency and high cost of completely annotating structured data.
method Information theoretic formulation applied to three diverse structured learning tasks.
result Learning from partial structures can sometimes outperform learning from complete ones.
A new method uses triplet embeddings to improve human annotation for hidden constructs.
problem Improving human annotation for hidden constructs in machine learning.
method Proposes a novel annotation approach using triplet embeddings to lift absolute annotations to relative comparisons.
result Successfully represents synthetic hidden constructs in time under noisy sampling conditions.
Optimizes classification algorithms with bounds on error rates.
problem Bounding uncertainties in classifier outputs for diagnostic testing.
method Set-theoretic and probabilistic arguments to derive uniform error bounds.
result Optimal partition minimizes the largest Gershgorin radius of the confusion matrix.
Unified framework for comparing classification metrics across different imbalance rates.
problem Differences in scale and sensitivity to class imbalance rates in classification metrics.
method Introduces outperformance standardization (OPS) function to map metrics to a common scale.
result Unified o-value metric provides clear comparison across different imbalance rates.
ActiveLab improves classifier accuracy with fewer annotations by re-labeling.
problem Imperfect labels from multiple annotators in real-world data.
method ActiveLab automatically decides when to re-label examples for better classifier training.
result ActiveLab trains more accurate classifiers with fewer annotations.
Bayesian methods improve text annotation quality.
problem Inconsistent and unreliable human annotations in natural language processing.
method Two semi-supervised Bayesian methods: a deep learning model and an ensemble method.
result Bayesian methods enhance the reliability and performance of BERT models.
Paper proposes an efficient method for bounding box annotation in object detection.
problem Manual annotation of bounding boxes is tedious and resource-intensive.
method Iterative training of object detector on small batches of labeled images, with human annotator correcting errors.
result Significant reduction in human annotation effort, up to 75%.