Generative model combines multi-dimensional annotations for more accurate ground truth estimation.
problem Inaccurate ground truth estimation from naive annotators' multi-dimensional annotations.
method Proposes a joint multi-dimensional model for global and time-series annotation fusion using Expectation-Maximization algorithm.
result More accurate ground truth estimates through joint modeling of multiple dimensions.
Paper proposes a weak supervision technique for CNN semantic segmentation of lung diseases using partially annotated data.
problem Creating annotated datasets for semantic segmentation of lung diseases is laborious and time-consuming.
method Proposes a weak supervision technique that utilizes partially annotated datasets to improve CNN semantic segmentation accuracy.
result Significantly improved segmentation accuracy using partially annotated datasets.
Paper proposes a method to reduce annotation time for 3D object detection.
problem Effort and time required for generating 3D object annotations.
method Combines human supervision with pretrained neural networks for 3D point cloud segmentation and bounding box generation.
result Reduces human annotation time by 30x.
A new method uses triplet embeddings to improve human annotation for hidden constructs.
problem Improving human annotation for hidden constructs in machine learning.
method Proposes a novel annotation approach using triplet embeddings to lift absolute annotations to relative comparisons.
result Successfully represents synthetic hidden constructs in time under noisy sampling conditions.
A new method for faster data annotation using click-supervision and hierarchical object detection.
problem Data annotation bottleneck in modern data collection.
method Semi-automatic approach combining human and neural network for hierarchical object detection.
result Improved annotation speed and accuracy compared to current methods.
Paper proposes an efficient method for bounding box annotation in object detection.
problem Manual annotation of bounding boxes is tedious and resource-intensive.
method Iterative training of object detector on small batches of labeled images, with human annotator correcting errors.
result Significant reduction in human annotation effort, up to 75%.
New method to estimate doctors' effort in annotating medical images.
problem High effort and expense in annotating medical images.
method Proposes a new criterion to evaluate effort, uses active learning and U-shape network for annotation strategy, and fine annotation platform to reduce effort.
result State-of-the-art segmentation performance achieved with only 60% annotation candidates, reducing effort by 44-47%.
Survey on AL strategies for cost-effective annotation in classification.
problem Real-world AL challenges due to human annotators' limitations.
method Categorizes 60 real-world AL strategies considering multiple annotators, query types, and cost schemes.
result General real-world AL strategy introduced for categorization of 60 strategies.
CPATTA uses conformal prediction for efficient test-time adaptation.
problem Low data selection efficiency in existing ATTA methods.
method Conformal Prediction, online weight-update algorithm, domain-shift detector, staged update scheme.
result CPATTA consistently outperforms state-of-the-art methods by 5% in accuracy.
The splendid success of convolutional neural networks (CNNs) in computer vision is largely attributable to the availability of massive annotated datasets, such as ImageNet and Places. However, in medical imaging, it is challenging to create such large annotated datasets, as annotating medical images is not only tedious…
Study investigates impact of labeler socio-cultural background on affect detection model performance.
problem Impact of labeler socio-cultural background on affect detection model performance.
method Investigates the impact of labeler socio-cultural background on affect detection model performance.
result Differences in labeler background impact the performance of affect detection models.
Learning algorithms normally assume that there is at most one annotation or label per data point. However, in some scenarios, such as medical diagnosis and on-line collaboration,multiple annotations may be available. In either case, obtaining labels for data points can be expensive and time-consuming (in some circumsta…
A new model cleans vocal note event annotations in music.
problem Erroneous labels in music datasets.
method Contrastive learning to automatically create local deformations of likely correct labels.
result Transcription model accuracy improves with the proposed strategy.
New approach uses synthetic labels to train models on scarce annotated data for surgical phase recognition.
problem Learning surgical phase recognition from limited annotated data.
method Teacher/Student approach with a CNN-biLSTM-CRF teacher generating synthetic labels for a CNN-LSTM student.
result Improved surgical phase recognition performance with fewer annotated videos.
Evaluates change point detection algorithms on real-world data.
problem Insufficient evaluation of change point detection algorithms on real-world time series.
method Developed a data set of 37 time series from various domains, annotated by human experts, and evaluated 14 algorithms using consistency metrics.
result Demonstrates the need for better evaluation methods in change point detection.
Standard myopic active learning assumes that human annotations are always obtainable whenever new samples are selected. This, however, is unrealistic in many real-world applications where human experts are not readily available at all times. In this paper, we consider the single shot setting: all the required samples s…
This work tackles missing annotations in large sensor datasets.
problem Missing annotations in large sensor datasets negatively affect supervised learning system performance.
method Proposes and evaluates three paradigms to handle gaps: dropping, single label, and unique label, along with a hybrid combination.
result Evaluation of proposed paradigms and hybrid combination shows significant performance improvement.
Automated radar data labeling using GNSS improves efficiency without sacrificing accuracy.
problem Manual annotation of automotive radar data is time-consuming and error-prone.
method The article introduces an automated system using GNSS to acquire data labels.
result The proposed method provides clear advantages in time savings with minimal data distortion.
With the rapid growth of crowdsourcing platforms it has become easy and relatively inexpensive to collect a dataset labeled by multiple annotators in a short time. However due to the lack of control over the quality of the annotators, some abnormal annotators may be affected by position bias which can potentially degra…
Proposes a self-training method for automatic data annotation.
problem Lack of large annotated datasets for supervised deep learning.
method Deep Bayesian Self-Training using variational inference and clustering.
result Superior performance over standard Self-Training baselines.
Paper tackles noisy annotations by considering workers' attention levels.
problem Noisy annotations from workers with varying expertise.
method Proposes a probabilistic model that incorporates workers' attention for accurate label quality estimation.
result Improves aggregated labels by quantifying the relationship between workers' attention and label quality.
A novel method detects multiple mitosis events and mitigates annotation gaps in phase-contrast microscopy.
problem Detecting multiple mitosis events and handling annotation gaps in closely placed cells.
method Estimating a spatiotemporal likelihood map via 3DCNN to detect multiple mitosis events and mitigate annotation gaps.
result Our method outperformed compared methods in terms of F1-score using a challenging dataset.
This work embeds annotations into a multidimensional space to measure classification difficulty.
problem Uncertainty in machine learning models during annotation phase.
method Develops a Bayesian Dirichlet-Multinomial framework to embed annotations and uses stochastic Expectation Maximization with MCMC.
result Embeddings reflect semantic similarities of original classes, aiding in measuring classification difficulty.
Study shows annotation instrument design affects model performance in hate speech detection.
problem Impact of annotation instrument design on model performance in hate speech detection.
method Collected annotations from five experimental conditions of an annotation instrument, fine-tuned BERT models on each dataset, evaluated performance on holdout portion.
result Significant differences in model performance and annotations across conditions.
DALES offers a large annotated aerial LiDAR dataset for 3D deep learning.
problem Lack of large-scale annotated aerial LiDAR datasets for deep learning.
method Collection and annotation of over half a billion hand-labeled points from an ALS scanner.
result DALES is the most extensive publicly available ALS data set with improved resolution and coverage.
Framework uses human annotations to make models robust to spurious correlations.
problem Machine learning models fail when unmeasured variables change test distributions.
method Human annotations to augment training examples, UV-DRO objective for robustness.
result Improvements of 5-10% on digit recognition task and 1.5-5% on NYPD Police Stops analysis.
Study compares methods for detecting novelty in textual data streams.
problem Lack of annotated datasets for novelty detection.
method Simulation framework to create controlled datasets and benchmark methods.
result Evaluation of state-of-the-art methods on different types of novelty.
Clearing algorithm improves CT image segmentation accuracy by merging confident annotations.
problem Inaccurate predictions due to noisy annotations from different annotators.
method 3-stage algorithm: scoring annotators, scoring nodules, merging annotations.
result Improves prediction accuracy in CT image segmentation tasks.
Active learning selects both observations and annotation precision for Gaussian Processes.
problem Costly annotation in supervised learning.
method Proposes an active learning algorithm that selects observations and annotation precision, using a modified BALD objective.
result Empirically shows the benefits of adjusting annotation precision in active learning.
RAD improves robustness to domain annotation noise without explicit domain annotations.
problem Robustness to domain annotation noise in training data.
method Regularized Annotation of Domains (RAD) for last layer retraining.
result RAD outperforms state-of-the-art methods even with 5% noise in training data.
The study challenges the notion that partial data annotation is inferior, suggesting it can sometimes outperform complete annotation.
problem The inefficiency and high cost of completely annotating structured data.
method Information theoretic formulation applied to three diverse structured learning tasks.
result Learning from partial structures can sometimes outperform learning from complete ones.
This study evaluates zero-shot LLMs in finance, finding ChatGPT performs well but fine-tuned models are better.
problem Evaluating zero-shot LLMs in financial tasks.
method Comparison of ChatGPT and fine-tuned models on annotated data.
result Fine-tuned models generally outperform zero-shot LLMs.
Deep learning model uses mixed supervision for brain tumor segmentation.
problem Costly manual tumor segmentation data.
method Extends segmentation networks with an image-level classification branch.
result Significant improvement in segmentation performance.
Solves TOD systems' query annotation problem without explicit annotations.
problem Training TOD systems without explicit KB query annotation.
method Reinforcement learning (RL) and pipelined approach for query prediction and system training.
result Improved RL agent with modifications for TOD tasks.
PTBCC improves accuracy in multi-class annotation aggregation by learning from prototype confusion matrices.
problem Inaccurate and insufficient confusion matrices for annotators in multi-class classification tasks.
method PTBCC (ProtoType learning-driven Bayesian Classifier Combination) uses prototype confusion matrices to capture annotator expertise.
result PTBCC achieves up to 15% accuracy improvement and 3% higher average accuracy compared to existing methods.
Learning-based methods for visual segmentation have made progress on particular types of segmentation tasks, but are limited by the necessary supervision, the narrow definitions of fixed tasks, and the lack of control during inference for correcting errors. To remedy the rigidity and annotation burden of standard appro…
ActiveLab improves classifier accuracy with fewer annotations by re-labeling.
problem Imperfect labels from multiple annotators in real-world data.
method ActiveLab automatically decides when to re-label examples for better classifier training.
result ActiveLab trains more accurate classifiers with fewer annotations.
Bayesian methods improve text annotation quality.
problem Inconsistent and unreliable human annotations in natural language processing.
method Two semi-supervised Bayesian methods: a deep learning model and an ensemble method.
result Bayesian methods enhance the reliability and performance of BERT models.
Majority Vote is optimal for reliable data labeling under certain conditions.
problem Reliable data labeling requires aggregating multiple annotators' labels, but the optimality of Majority Vote is not well understood.
method Characterized conditions under which Majority Vote achieves the optimal label estimation error.
result Majority Vote optimally recovers labels for a given class distribution under tolerable annotation noise limits.
Study improves app feature extraction models with new annotation guidelines and data.
problem Improving the quality and usefulness of app feature extraction models.
method Exploring the effects of annotation guidelines and annotated data on app feature extraction models.
result New annotation guidelines lead to less noisy and more informative app features.
Crowdsourcing and active learning reduce manual annotation in social media event classification.
problem Manual annotation is time-consuming and resource-intensive in social media event classification.
method Crowdsourcing pipeline combined with active learning strategies.
result Active learning strategies help reduce the number of tweets needed for classification.
Extend deep neural networks for multiple anatomical structures in real-time.
problem Lack of comprehensive patient-specific anatomical models for complex surgical planning.
method Class-incremental segmentation framework that extends deep networks with minimal incremental annotation.
result Framework retains previous structure segmentation performance and learns new structures without catastrophic forgetting.
Improves off-policy evaluation with imperfect annotations.
problem Limited dataset coverage for evaluating new policies.
method Doubly robust estimators combining IS and DM, incorporating counterfactual annotations.
result Using annotations within the DM component yields the most desirable theoretical results.
A popular approach for large scale data annotation tasks is crowdsourcing, wherein each data point is labeled by multiple noisy annotators. We consider the problem of inferring ground truth from noisy ordinal labels obtained from multiple annotators of varying and unknown expertise levels. Annotation models for ordinal…
Crowdlab uses classifiers to estimate consensus labels and annotator quality.
problem Leveraging multiple annotators for data classification.
method Weighted ensemble approach using any trained classifier.
result Superior estimates for consensus labels and annotator quality.
New method uses predictions to infer causal effects without labeled data.
problem Data labeling costs limit causal inference experiments.
method Prediction-Powered Causal Inferences (PPCI) using conditional calibration and transfer constraints.
result Valid causal inference achieved on experiments with no human annotations.
Method learns true labels from noisy annotators using regularization.
problem Learning from noisy labels in supervised learning.
method Regularized estimation of annotator confusion matrices.
result Method outperforms state-of-the-art methods in image classification.
LISBET automates social behavior analysis using machine learning.
problem Manual annotation of social behaviors is time-consuming, biased, and misses subtle interactions.
method Self-supervised learning on body tracking data.
result Automated detection and segmentation of social interactions.