Proposes ALRL for single shot active learning with pseudo annotators.
problem Real-world applications where human experts are not always available.
method Substitutes human annotators with pseudo annotators providing random labels.
result ALRL outperforms state-of-the-art approaches in real-world datasets.
Proposes a new contrastive loss for semi-supervised medical image segmentation.
problem Lack of labeled data for medical image segmentation.
method Uses pseudo-labels and a local contrastive loss to learn good local representations.
result Achieved high segmentation performance on public cardiac and prostate datasets.
Paper proposes a new framework for semi-supervised learning.
problem Limited annotated data in supervised learning.
method Fuzzy domain constraint-based framework for semi-supervised learning.
result Enhances model quality for semi-supervised learning.
CPATTA uses conformal prediction for efficient test-time adaptation.
problem Low data selection efficiency in existing ATTA methods.
method Conformal Prediction, online weight-update algorithm, domain-shift detector, staged update scheme.
result CPATTA consistently outperforms state-of-the-art methods by 5% in accuracy.
Paper tackles cross-granularity few-shot learning with meta-embedder.
problem Few-shot learning with coarse labels and fine-grained testing.
method Meta-embedder that optimizes visual and semantic discrimination across coarse and fine classes.
result Meta-embedder achieves effective cross-granularity few-shot classification.
Proposes provenance and pseudo-provenance for automated test generation.
problem Invalidation of provenance in generated tests.
method Annotation of generated tests with provenance trails and pseudo-provenance.
result Validates the reliability of generated tests and their relation to seeds.
SoQal uses selective oracle questioning to improve active learning of cardiac signals.
problem Active learning of cardiac signals is challenging due to limited labelled data and high annotation costs.
method Proposes a framework combining selective oracle questioning and Bayesian active learning by consistency.
result SoQal outperforms baseline methods in active learning of cardiac signals, even with noisy oracles.
Unified framework for semi-supervised learning reduces annotation needs.
problem Sparse annotations and large amounts of unlabeled data in computational pathology.
method S5CL integrates fully-supervised, self-supervised, and semi-supervised learning through hierarchical contrastive losses.
result S5CL improves accuracy and F1-score in histopathological datasets with sparse labels.
Proposes ConstraintMatch for semi-supervised clustering with unconstrained data.
problem Leveraging unconstrained data alongside constraints for clustering models.
method Semi-supervised context with pseudo-constraining and pseudo-labeling mechanisms.
result Demonstrates effectiveness of ConstraintMatch over baselines.
Paper proposes an end-to-end learning method for state estimation in robotics.
problem Lack of annotated data for optimising dynamic and measurement models in particle filters.
method End-to-end learning objective based on maximising a pseudo-likelihood function.
result Improves state estimation when large portions of true states are unknown.
New method estimates covariance in deep heteroscedastic regression without labels.
problem Estimating covariance in deep heteroscedastic models is challenging due to sample-dependent covariance and lack of ground truth.
method Proposes a self-supervised approach using KL Divergence and 2-Wasserstein distance for covariance estimation and a neighborhood-based heuristic for pseudo labels.
result Demonstrates effective pseudo labels and a computationally cheaper yet accurate deep heteroscedastic regression.
Study shows annotation instrument design affects model performance in hate speech detection.
problem Impact of annotation instrument design on model performance in hate speech detection.
method Collected annotations from five experimental conditions of an annotation instrument, fine-tuned BERT models on each dataset, evaluated performance on holdout portion.
result Significant differences in model performance and annotations across conditions.
New method improves adversarial learning with unlabeled data.
problem Poor quality of pseudo labels on unlabeled data.
method Robust Co-training (RCT) using deep co-training.
result RCT significantly outperforms baselines in adversarial robustness.
Clearing algorithm improves CT image segmentation accuracy by merging confident annotations.
problem Inaccurate predictions due to noisy annotations from different annotators.
method 3-stage algorithm: scoring annotators, scoring nodules, merging annotations.
result Improves prediction accuracy in CT image segmentation tasks.
New method to estimate doctors' effort in annotating medical images.
problem High effort and expense in annotating medical images.
method Proposes a new criterion to evaluate effort, uses active learning and U-shape network for annotation strategy, and fine annotation platform to reduce effort.
result State-of-the-art segmentation performance achieved with only 60% annotation candidates, reducing effort by 44-47%.
Active learning selects both observations and annotation precision for Gaussian Processes.
problem Costly annotation in supervised learning.
method Proposes an active learning algorithm that selects observations and annotation precision, using a modified BALD objective.
result Empirically shows the benefits of adjusting annotation precision in active learning.
Generative model combines multi-dimensional annotations for more accurate ground truth estimation.
problem Inaccurate ground truth estimation from naive annotators' multi-dimensional annotations.
method Proposes a joint multi-dimensional model for global and time-series annotation fusion using Expectation-Maximization algorithm.
result More accurate ground truth estimates through joint modeling of multiple dimensions.
Survey on AL strategies for cost-effective annotation in classification.
problem Real-world AL challenges due to human annotators' limitations.
method Categorizes 60 real-world AL strategies considering multiple annotators, query types, and cost schemes.
result General real-world AL strategy introduced for categorization of 60 strategies.
RAD improves robustness to domain annotation noise without explicit domain annotations.
problem Robustness to domain annotation noise in training data.
method Regularized Annotation of Domains (RAD) for last layer retraining.
result RAD outperforms state-of-the-art methods even with 5% noise in training data.
The study challenges the notion that partial data annotation is inferior, suggesting it can sometimes outperform complete annotation.
problem The inefficiency and high cost of completely annotating structured data.
method Information theoretic formulation applied to three diverse structured learning tasks.
result Learning from partial structures can sometimes outperform learning from complete ones.
A new method uses triplet embeddings to improve human annotation for hidden constructs.
problem Improving human annotation for hidden constructs in machine learning.
method Proposes a novel annotation approach using triplet embeddings to lift absolute annotations to relative comparisons.
result Successfully represents synthetic hidden constructs in time under noisy sampling conditions.
PTBCC improves accuracy in multi-class annotation aggregation by learning from prototype confusion matrices.
problem Inaccurate and insufficient confusion matrices for annotators in multi-class classification tasks.
method PTBCC (ProtoType learning-driven Bayesian Classifier Combination) uses prototype confusion matrices to capture annotator expertise.
result PTBCC achieves up to 15% accuracy improvement and 3% higher average accuracy compared to existing methods.
ActiveLab improves classifier accuracy with fewer annotations by re-labeling.
problem Imperfect labels from multiple annotators in real-world data.
method ActiveLab automatically decides when to re-label examples for better classifier training.
result ActiveLab trains more accurate classifiers with fewer annotations.
Bayesian methods improve text annotation quality.
problem Inconsistent and unreliable human annotations in natural language processing.
method Two semi-supervised Bayesian methods: a deep learning model and an ensemble method.
result Bayesian methods enhance the reliability and performance of BERT models.
Study improves app feature extraction models with new annotation guidelines and data.
problem Improving the quality and usefulness of app feature extraction models.
method Exploring the effects of annotation guidelines and annotated data on app feature extraction models.
result New annotation guidelines lead to less noisy and more informative app features.
Paper proposes an efficient method for bounding box annotation in object detection.
problem Manual annotation of bounding boxes is tedious and resource-intensive.
method Iterative training of object detector on small batches of labeled images, with human annotator correcting errors.
result Significant reduction in human annotation effort, up to 75%.
Paper proposes a weak supervision technique for CNN semantic segmentation of lung diseases using partially annotated data.
problem Creating annotated datasets for semantic segmentation of lung diseases is laborious and time-consuming.
method Proposes a weak supervision technique that utilizes partially annotated datasets to improve CNN semantic segmentation accuracy.
result Significantly improved segmentation accuracy using partially annotated datasets.
Paper proposes a method for faster object detection annotation in indoor scenes.
problem Efficiently annotate object detection datasets in indoor scenes.
method Two-stage annotation process: manual annotation of a part of the dataset, followed by automatic annotation of the rest.
result The two-stage method significantly reduces the total workload compared to manual annotation of the entire dataset.
Reduces annotation costs in medical imaging by 50%.
problem Challenges in creating large annotated datasets for medical imaging.
method Integrates active learning and transfer learning into a single framework.
result Reduces annotation efforts by at least half.
A new method for faster data annotation using click-supervision and hierarchical object detection.
problem Data annotation bottleneck in modern data collection.
method Semi-automatic approach combining human and neural network for hierarchical object detection.
result Improved annotation speed and accuracy compared to current methods.
Improves off-policy evaluation with imperfect annotations.
problem Limited dataset coverage for evaluating new policies.
method Doubly robust estimators combining IS and DM, incorporating counterfactual annotations.
result Using annotations within the DM component yields the most desirable theoretical results.
A popular approach for large scale data annotation tasks is crowdsourcing, wherein each data point is labeled by multiple noisy annotators. We consider the problem of inferring ground truth from noisy ordinal labels obtained from multiple annotators of varying and unknown expertise levels. Annotation models for ordinal…
Crowdlab uses classifiers to estimate consensus labels and annotator quality.
problem Leveraging multiple annotators for data classification.
method Weighted ensemble approach using any trained classifier.
result Superior estimates for consensus labels and annotator quality.
This study uses partially annotated data to improve TempRel extraction.
problem Lack of fully annotated data for TempRel extraction makes the task labor-intensive and limited in coverage.
method Utilizes partially annotated data (P) for TempRel extraction, even when annotations are missing.
result Partially annotated data (P) can still be a useful supervision signal for TempRel extraction within a constrained learning framework.
Method learns true labels from noisy annotators using regularization.
problem Learning from noisy labels in supervised learning.
method Regularized estimation of annotator confusion matrices.
result Method outperforms state-of-the-art methods in image classification.
Cluster annotation reduces active learning human interactions.
problem Reducing human interactions in active learning.
method Cluster annotation to label multiple samples at once.
result 82% and 87% less human interactions for CIFAR-10 and EuroSAT datasets.
Study uses active learning to automate EEG event annotation.
problem Lack of annotated clinical EEG data for machine learning models.
method Active learning algorithm for automated annotation of six types of EEG events.
result Recognition performance improved 2% absolute, capable of auto-annotating.
The paper proposes incentivizing human annotators with 'golden questions' to improve data quality.
problem Ensuring high-quality human annotations for training large language models.
method A principal-agent model is used to incentivize annotators with bonuses based on the maximum likelihood estimators (MLE) of their annotations. Hypothesis testing is applied to monitor the annotators' performance.
result The hypothesis testing rate for the principal-agent model is of Θ ( 1 / n log n ) Θ(1/\sqrt{n \log n}) Θ ( 1/ n log n ) , highlighting the importance of 'golden questions' for monitoring annotators. The paper proposes EM algorithms for inferring object rankings from noisy judgments.
problem Inferring ground truth rankings from noisy pairwise comparisons.
method Expectation-Maximization (EM) algorithms that learn object attributes and annotator probabilities.
result The algorithms improve accuracy by considering object attributes and annotator quality.
Paper proposes active learning for sound event detection with reduced annotation effort.
problem Reducing annotation effort for sound event detection.
method Change point detection for candidate selection, mismatch-first farthest-traversal for selection, training with context recordings.
result The proposed system achieves similar performance to full annotation with only 2% of data, reducing annotation effort.
Paper proposes a method to reduce annotation time for 3D object detection.
problem Effort and time required for generating 3D object annotations.
method Combines human supervision with pretrained neural networks for 3D point cloud segmentation and bounding box generation.
result Reduces human annotation time by 30x.
G-SimCLR improves unsupervised learning by clustering images into pseudo labels.
problem Improving unsupervised learning for image recognition.
method Proposes a method to cluster images into pseudo labels to batch images of the same category.
result Comparable performance enhancements on CIFAR10 and ImageNet datasets.
Interactive IL beats BC by state-wise annotation cost.
problem Behavior Cloning struggles with annotation cost in sequential decision making.
method Proved Stagger and Warm Stagger algorithms to outperform BC.
result Interactive and hybrid IL methods outperform BC with state-wise annotation.
End-to-end framework learns from imperfect annotations directly.
problem Training machine learning models on imperfect human annotations.
method End-to-end framework merging aggregation with model training and modeling annotator competencies.
result Accuracy gains of up to 25% over state-of-the-art annotation aggregation methods.
Unified framework for deep learning with crowdsourced data.
problem Learning true labels from noisy, sparse, and uncontrolled crowdsourced annotations.
method Bayesian deep learning framework that learns annotator expertise and optimizes model training.
result Framework reduces annotation and training time for deep learning models.
Learning algorithms normally assume that there is at most one annotation or label per data point. However, in some scenarios, such as medical diagnosis and on-line collaboration,multiple annotations may be available. In either case, obtaining labels for data points can be expensive and time-consuming (in some circumsta…
Bayesian model improves truth inference from highly redundant crowd annotations.
problem Inferring true annotations from highly redundant crowd annotations.
method Bayesian graphical model with conjugate priors and iterative expectation-maximisation inference.
result Our technique significantly outperforms majority vote heuristic at one-sided level 0.025.
Meta-learning method for accurate classifier from noisy annotators' data.
problem Accurate learning from noisy labels provided by multiple annotators.
method Meta-learning neural network to embed examples in latent space and estimate annotators' abilities, then adapt classifiers using EM algorithm.
result Meta-learning method improves classifier performance with minimal labeled data.