arXiv research

The paper proposes incentivizing human annotators with 'golden questions' to improve data quality.

Paper proposes an efficient method for bounding box annotation in object detection.

In this work, we introduce a novel framework that employs cluster annotation to boost active learning by reducing the number of human interactions required to train deep neural networks. Instead of annotating single samples individually, humans can also label clusters, producing a higher number of annotated samples wit…

Survey on AL strategies for cost-effective annotation in classification.

Standard myopic active learning assumes that human annotations are always obtainable whenever new samples are selected. This, however, is unrealistic in many real-world applications where human experts are not readily available at all times. In this paper, we consider the single shot setting: all the required samples s…

CPATTA uses conformal prediction for efficient test-time adaptation.

Human annotations serve an important role in computational models where the target constructs under study are hidden, such as dimensions of affect. This is especially relevant in machine learning, where subjective labels derived from related observable signals (e.g., audio, video, text) are needed to support model trai…

Paper tackles active learning under human label variation, proposing a new framework.

The development of real-time affect detection models often depends upon obtaining annotated data for supervised learning by employing human experts to label the student data. One open question in annotating affective data for affect detection is whether the labelers (i.e., human experts) need to be socio-culturally sim…

End-to-end framework learns from imperfect annotations directly.

MCAL reduces labeling costs by 6x for auto-labeling data sets.

Majority Vote is optimal for reliable data labeling under certain conditions.

We consider the problem of explaining the decisions of deep neural networks for image recognition in terms of human-recognizable visual concepts. In particular, given a test set of images, we aim to explain each classification in terms of a small number of image regions, or activation maps, which have been associated w…

Training 3D object detectors for autonomous driving has been limited to small datasets due to the effort required to generate annotations. Reducing both task complexity and the amount of task switching done by annotators is key to reducing the effort and time required to generate 3D bounding box annotations. This paper…

Study compares DSPy teleprompter algorithms for aligning LLM evaluations with human annotations.

New method pools labels from similar data items to improve learning from small samples.

The increase in data collection has made data annotation an interesting and valuable task in the contemporary world. This paper presents a new methodology for quickly annotating data using click-supervision and hierarchical object detection. The proposed work is semi-automatic in nature where the task of annotations is…

Framework uses human annotations to make models robust to spurious correlations.

Semi-automatic data annotation helps experts label unlabeled samples based on feature space projection.

Bayesian framework evaluates predictors of subjective visual tasks.

SPIN converts weak LLMs to strong ones using self-play.

XRM discovers environments without human annotations for OOD methods.

Pathology reports contain useful information such as the main involved organ, diagnosis, etc. These information can be identified from the free text reports and used for large-scale statistical analysis or serve as annotation for other modalities such as pathology slides images. However, manual classification for a hug…

Paper introduces robust learning from feature feedback, even with annotator errors.

This paper refines human labeling as a measurement process, revealing four sources of variation.

Bayesian methods improve text annotation quality.

Generative model combines multi-dimensional annotations for more accurate ground truth estimation.

Study shows annotation instrument design affects model performance in hate speech detection.

Evaluates change point detection algorithms on real-world data.

Scientific disciplines, such as Behavioural Psychology, Anthropology and recently Social Signal Processing are concerned with the systematic exploration of human behaviour. A typical work-flow includes the manual annotation (also called coding) of social signals in multi-modal corpora of considerable size. For the invo…

LISBET automates social behavior analysis using machine learning.

Neural networks promise to bring robust, quantitative analysis to medical fields, but adoption is limited by the technicalities of training these networks. To address this translation gap between medical researchers and neural networks in the field of pathology, we have created an intuitive interface which utilizes the…

This paper surveys scalable automated alignment methods for LLMs.

Unlike images or videos data which can be easily labeled by human being, sensor data annotation is a time-consuming process. However, traditional methods of human activity recognition require a large amount of such strictly labeled data for training classifiers. In this paper, we present an attention-based convolutiona…

IAL uses interactive learning to improve model performance with minimal human feedback.

Study real-world noisy labels from human annotations for better understanding.

LFD method improves text classification by making features clearer and less label-leaking.

Ranking a set of objects involves establishing an order allowing for comparisons between any pair of objects in the set. Oftentimes, due to the unavailability of a ground truth of ranked orders, researchers resort to obtaining judgments from multiple annotators followed by inferring the ground truth based on the collec…

This paper presents a generic Bayesian framework that enables any deep learning model to actively learn from targeted crowds. Our framework inherits from recent advances in Bayesian deep learning, and extends existing work by considering the targeted crowdsourcing approach, where multiple annotators with unknown expert…

This work creates a system for understanding human movement in spaces.

We conduct large-scale studies on `human attention' in Visual Question Answering (VQA) to understand where humans choose to look to answer questions about images. We design and test multiple game-inspired novel attention-annotation interfaces that require the subject to sharpen regions of a blurred image to answer a qu…

A human-in-the-loop ML framework for precision dosing reduces expert workload and removes bias.

A novel method detects multiple mitosis events and mitigates annotation gaps in phase-contrast microscopy.

Bayesian algorithms improve crowdsourcing with label and instance constraints.

We describe a method for selecting relevant new training data for the LSTM-based domain selection component of our personal assistant system. Adding more annotated training data for any ML system typically improves accuracy, but only if it provides examples not already adequately covered in the existing data. However, …

Grammatical error correction, like other machine learning tasks, greatly benefits from large quantities of high quality training data, which is typically expensive to produce. While writing a program to automatically generate realistic grammatical errors would be difficult, one could learn the distribution of naturally…

New method improves human mesh recovery for obese people.

Bayesian method improves segmentation accuracy with noisy labels.