arXiv research

Paper proposes a weak supervision technique for CNN semantic segmentation of lung diseases using partially annotated data.

For many structured learning tasks, the data annotation process is complex and costly. Existing annotation schemes usually aim at acquiring completely annotated structures, under the common perception that partial structures are of low quality and could hurt the learning process. This paper questions this common percep…

Annotating temporal relations (TempRel) between events described in natural language is known to be labor intensive, partly because the total number of TempRels is quadratic in the number of events. As a result, only a small number of documents are typically annotated, limiting the coverage of various lexical/semantic …

Study shows partially-typed NER datasets can match fully-typed ones in model performance.

Structured learning is appropriate when predicting structured outputs such as trees, graphs, or sequences. Most prior work requires the training set to consist of complete trees, graphs or sequences. Specifying such detailed ground truth can be tedious or infeasible for large outputs. Our main contribution is a large m…

Unified framework for structured prediction with partial labelling.

Paper explores VRM for PSMLC with partially labeled medical images.

Paper proposes an algorithm to recover full supervision from weakly labeled data.

Integrates multiple datasets to solve open set crowdsourcing problems.

Active learning improves inspection systems by using weakly labeled data.

Recent work establishes dataset difficulty and removes annotation artifacts via partial-input baselines (e.g., hypothesis-only models for SNLI or question-only models for VQA). When a partial-input baseline gets high accuracy, a dataset is cheatable. However, the converse is not necessarily true: the failure of a parti…

New methods for graph completion improve image annotation and neuroimaging.

LLMs struggle with zero-shot annotation tasks due to model-internalized priors.

A reject option improves partial-label learning's accuracy.

Partial multi-label learning (PML), which tackles the problem of learning multi-label prediction models from instances with overcomplete noisy annotations, has recently started gaining attention from the research community. In this paper, we propose a novel adversarial learning model, PML-GAN, under a generalized encod…

Study shows annotation instrument design affects model performance in hate speech detection.

A new Q&A labeling method for assigning labels in machine learning.

New method for partial matching of shapes with Varifolds.

Active learning selects both observations and annotation precision for Gaussian Processes.

One of the problems on the way to successful implementation of neural networks is the quality of annotation. For instance, different annotators can annotate images in a different way and very often their decisions do not match exactly and in extreme cases are even mutually exclusive which results in noisy annotations a…

Generative model combines multi-dimensional annotations for more accurate ground truth estimation.

Survey on AL strategies for cost-effective annotation in classification.

RAD improves robustness to domain annotation noise without explicit domain annotations.

Accurate annotation of medical image is the crucial step for image AI clinical application. However, annotating medical image will incur a great deal of annotation effort and expense due to its high complexity and needing experienced doctors. To alleviate annotation cost, some active learning methods are proposed. But …

This paper analyzes and improves active learning techniques for real-world projects.

PTBCC improves accuracy in multi-class annotation aggregation by learning from prototype confusion matrices.

ActiveLab improves classifier accuracy with fewer annotations by re-labeling.

Bayesian methods improve text annotation quality.

Paper proposes an efficient method for bounding box annotation in object detection.

Human annotations serve an important role in computational models where the target constructs under study are hidden, such as dimensions of affect. This is especially relevant in machine learning, where subjective labels derived from related observable signals (e.g., audio, video, text) are needed to support model trai…

Proposes a method to learn invariant representations for interpretability and fairness.

Efficiently learns from partial labels using variational inference.

Annotation guidelines used to guide the annotation of training and evaluation datasets can have a considerable impact on the quality of machine learning models. In this study, we explore the effects of annotation guidelines on the quality of app feature extraction models. As a main result, we propose several changes to…

Improves off-policy evaluation with imperfect annotations.

A popular approach for large scale data annotation tasks is crowdsourcing, wherein each data point is labeled by multiple noisy annotators. We consider the problem of inferring ground truth from noisy ordinal labels obtained from multiple annotators of varying and unknown expertise levels. Annotation models for ordinal…

Crowdlab uses classifiers to estimate consensus labels and annotator quality.

The increase in data collection has made data annotation an interesting and valuable task in the contemporary world. This paper presents a new methodology for quickly annotating data using click-supervision and hierarchical object detection. The proposed work is semi-automatic in nature where the task of annotations is…

This paper presents a generic Bayesian framework that enables any deep learning model to actively learn from targeted crowds. Our framework inherits from recent advances in Bayesian deep learning, and extends existing work by considering the targeted crowdsourcing approach, where multiple annotators with unknown expert…

The paper proposes incentivizing human annotators with 'golden questions' to improve data quality.

Paper proposes active learning for sound event detection with reduced annotation effort.

The splendid success of convolutional neural networks (CNNs) in computer vision is largely attributable to the availability of massive annotated datasets, such as ImageNet and Places. However, in medical imaging, it is challenging to create such large annotated datasets, as annotating medical images is not only tedious…

Training 3D object detectors for autonomous driving has been limited to small datasets due to the effort required to generate annotations. Reducing both task complexity and the amount of task switching done by annotators is key to reducing the effort and time required to generate 3D bounding box annotations. This paper…

Ranking a set of objects involves establishing an order allowing for comparisons between any pair of objects in the set. Oftentimes, due to the unavailability of a ground truth of ranked orders, researchers resort to obtaining judgments from multiple annotators followed by inferring the ground truth based on the collec…

Interactive IL beats BC by state-wise annotation cost.

End-to-end framework learns from imperfect annotations directly.

Learning algorithms normally assume that there is at most one annotation or label per data point. However, in some scenarios, such as medical diagnosis and on-line collaboration,multiple annotations may be available. In either case, obtaining labels for data points can be expensive and time-consuming (in some circumsta…

A new PLL method uses class activation values to improve robustness.

Typilus predicts types for Python programs using neural networks.