arXiv research

Study shows annotation instrument design affects model performance in hate speech detection.

Clearing algorithm improves CT image segmentation accuracy by merging confident annotations.

New method to estimate doctors' effort in annotating medical images.

Active learning selects both observations and annotation precision for Gaussian Processes.

Generative model combines multi-dimensional annotations for more accurate ground truth estimation.

Survey on AL strategies for cost-effective annotation in classification.

RAD improves robustness to domain annotation noise without explicit domain annotations.

The study challenges the notion that partial data annotation is inferior, suggesting it can sometimes outperform complete annotation.

A new method uses triplet embeddings to improve human annotation for hidden constructs.

PTBCC improves accuracy in multi-class annotation aggregation by learning from prototype confusion matrices.

ActiveLab improves classifier accuracy with fewer annotations by re-labeling.

Bayesian methods improve text annotation quality.

Study improves app feature extraction models with new annotation guidelines and data.

Paper proposes an efficient method for bounding box annotation in object detection.

Paper proposes a weak supervision technique for CNN semantic segmentation of lung diseases using partially annotated data.

Paper proposes a method for faster object detection annotation in indoor scenes.

A new method for faster data annotation using click-supervision and hierarchical object detection.

Improves off-policy evaluation with imperfect annotations.

A popular approach for large scale data annotation tasks is crowdsourcing, wherein each data point is labeled by multiple noisy annotators. We consider the problem of inferring ground truth from noisy ordinal labels obtained from multiple annotators of varying and unknown expertise levels. Annotation models for ordinal…

Crowdlab uses classifiers to estimate consensus labels and annotator quality.

Method learns true labels from noisy annotators using regularization.

Cluster annotation reduces active learning human interactions.

This paper presents a generic Bayesian framework that enables any deep learning model to actively learn from targeted crowds. Our framework inherits from recent advances in Bayesian deep learning, and extends existing work by considering the targeted crowdsourcing approach, where multiple annotators with unknown expert…

The paper proposes incentivizing human annotators with 'golden questions' to improve data quality.

Paper proposes active learning for sound event detection with reduced annotation effort.

Paper proposes a method to reduce annotation time for 3D object detection.

The splendid success of convolutional neural networks (CNNs) in computer vision is largely attributable to the availability of massive annotated datasets, such as ImageNet and Places. However, in medical imaging, it is challenging to create such large annotated datasets, as annotating medical images is not only tedious…

Ranking a set of objects involves establishing an order allowing for comparisons between any pair of objects in the set. Oftentimes, due to the unavailability of a ground truth of ranked orders, researchers resort to obtaining judgments from multiple annotators followed by inferring the ground truth based on the collec…

Interactive IL beats BC by state-wise annotation cost.

End-to-end framework learns from imperfect annotations directly.

Learning algorithms normally assume that there is at most one annotation or label per data point. However, in some scenarios, such as medical diagnosis and on-line collaboration,multiple annotations may be available. In either case, obtaining labels for data points can be expensive and time-consuming (in some circumsta…

Bayesian model improves truth inference from highly redundant crowd annotations.

Meta-learning method for accurate classifier from noisy annotators' data.

New method uses image-level and pixel-level annotations for brain tumor segmentation.

Annotating temporal relations (TempRel) between events described in natural language is known to be labor intensive, partly because the total number of TempRels is quadratic in the number of events. As a result, only a small number of documents are typically annotated, limiting the coverage of various lexical/semantic …

To be effective, state of the art machine learning technology needs large amounts of annotated data. There are numerous compelling applications in healthcare that can benefit from high performance automated decision support systems provided by deep learning technology, but they lack the comprehensive data resources req…

ECN framework improves training on noisy structured labels.

Automated image annotation improves model training accuracy.

Semi-automatic data annotation helps experts label unlabeled samples based on feature space projection.

This work improves LePU by modeling the annotation process more realistically.

We consider the problem of Probably Approximate Correct (PAC) learning of a binary classifier from noisy labeled examples acquired from multiple annotators (each characterized by a respective classification noise rate). First, we consider the complete information scenario, where the learner knows the noise rates of all…

System classifies metaphorical violence on cable news.

Paper tackles active learning under human label variation, proposing a new framework.

OLALA automates document layout annotation by selecting ambiguous regions for labeling.

Automated generation of medical reports from chest x-rays using expert annotations.

A new method fuses labels from multiple, differently-annotated schemas.

With the rapid growth of crowdsourcing platforms it has become easy and relatively inexpensive to collect a dataset labeled by multiple annotators in a short time. However due to the lack of control over the quality of the annotators, some abnormal annotators may be affected by position bias which can potentially degra…

Standard myopic active learning assumes that human annotations are always obtainable whenever new samples are selected. This, however, is unrealistic in many real-world applications where human experts are not readily available at all times. In this paper, we consider the single shot setting: all the required samples s…