Driving datasets accelerate the development of intelligent driving and related computer vision technologies, while substantial and detailed annotations serve as fuels and powers to boost the efficacy of such datasets to improve learning-based models. We propose D-City, a large-scale comprehensive collection of dash…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,932 papers · 148 categories
Trend · papers per month
48 results for “annotator diversity”
Develops a method to learn personalized attribute preferences from diverse annotators.
problem Learning attribute preferences from a wide spectrum of annotators with varying interests.
method Multi-task approach with AUC-based loss function and closed-form solution.
result Empirically validated method outperforms traditional consensus-based methods.
The study challenges the notion that partial data annotation is inferior, suggesting it can sometimes outperform complete annotation.
problem The inefficiency and high cost of completely annotating structured data.
method Information theoretic formulation applied to three diverse structured learning tasks.
result Learning from partial structures can sometimes outperform learning from complete ones.
ADS filters data points for efficient batch active learning.
problem Efficiently selecting data points for annotation in parallel settings.
method Active Data Shapley (ADS) using the Shapley value of data.
result Significantly increases efficiency of active learning by 6x.
A new method models uncertainty in medical image segmentation.
problem Inherent ambiguity in anatomical structure and pathology segmentation.
method Hierarchical probabilistic model with variational autoencoder.
result Generates more realistic and diverse segmentation samples.
Semi-automatic data annotation helps experts label unlabeled samples based on feature space projection.
problem Laborious manual data annotation for machine learning.
method Interactive semi-automatic approach using feature space projection and semi-supervised learning.
result Reduces user annotation effort and improves classification accuracy.
SharedRep-RLHF learns shared traits for diverse groups, improving fairness and performance.
problem Uniform-reward RLHF fails to capture diverse preferences, leading to unfairness.
method SharedRep-RLHF learns shared traits among various groups, improving fairness and performance.
result SharedRep-RLHF outperforms MaxMin-RLHF by up to 20% in win rate.
Lazy labels enable deep learning for microscopy segmentation without full annotation.
problem Lack of pixel-wise annotations limits fully supervised learning for bioimage segmentation.
method Lazy labels combined with coarse labels for training a deep neural network.
result Model achieves accurate segmentation with minimal pixel-wise annotations.
Many latent (factorized) models have been proposed for recommendation tasks like collaborative filtering and for ranking tasks like document or image retrieval and annotation. Common to all those methods is that during inference the items are scored independently by their similarity to the query in the latent embedding…
Model for detecting rare labels in imbalanced crowdsourcing data.
problem Detecting rare labels in imbalanced crowdsourcing data.
method Generative aggregation model combining item difficulty and class-dependent annotator competence.
result Our model achieves the highest minority recall while maintaining competitive balanced accuracy.
Adding uninformative labels improves tumor segmentation in low-data mammography.
problem Improving tumor segmentation in mammography with limited data.
method Used seemingly uninformative labels from non-expert annotators to turn a multi-label task into a multi-class problem.
result Performance gains in tumor segmentation are achieved in low-data settings with additional uninformative labels.
Many structured prediction problems (particularly in vision and language domains) are ambiguous, with multiple outputs being correct for an input - e.g. there are many ways of describing an image, multiple ways of translating a sentence; however, exhaustively annotating the applicability of all possible outputs is intr…
Unified framework for medical image segmentation using active and semi-supervised learning.
problem Training medical image segmentation models with limited annotated data.
method RegAL, a unified active semi-supervised framework that optimizes sample acquisition and unlabeled data utilization.
result RegAL consistently outperforms state-of-the-art methods across various datasets and metrics under extreme annotation scarcity.
New method predicts target genres from source genres, unifying music tag systems.
problem Automatic genre inference fails to handle music genre diversity and subjectivity.
method Knowledge-based, statistical, and hybrid translation models.
result Hybrid translation model is most effective for multilabel classification.
Active learning (AL) aims to enable training high performance classifiers with low annotation cost by predicting which subset of unlabelled instances would be most beneficial to label. The importance of AL has motivated extensive research, proposing a wide variety of manually designed AL algorithms with diverse theoret…
Self-supervised learning helps train deep features without needing lots of labeled data.
problem Annotation bottleneck in deep learning.
method Four main families of self-supervised approaches applied to various data modalities.
result Self-supervised methods can now rival fully supervised pre-training across multiple data types.
Dataset for object detection at Oktoberfest beer tent.
problem Realistic object detection in a busy, diverse setting.
method Hand-annotated 1,110 images, provided trained models.
result Challenging dataset for object detection.
The instance segmentation problem intends to precisely detect and delineate objects in images. Most of the current solutions rely on deep convolutional neural networks but despite this fact proposed solutions are very diverse. Some solutions approach the problem as a network problem, where they use several networks or …
LLMs struggle with zero-shot annotation tasks due to model-internalized priors.
problem Impact of model-internalized priors on LLM performance in zero-shot annotation tasks.
method Investigated three dimensions: familiarity, decision stickiness, and susceptibility to misaligned task definitions.
result Nearly two-thirds of zero-shot errors are resistant to correction, with a rescue rate of 34.8%. Definition-Specific Familiarity (DSF) shows a positive association with model performance.
Few-shot cell segmentation from diverse sources to target domain.
problem Efficient cell segmentation from limited annotated images.
method Meta-learning combining cross-domain tasks and invariant representation.
result Promising results from 1-10-shot learning on public databases.
Words That Unite The World: A Unified Framework for Deciphering Central Bank Communications Globallycs.CL
Unified framework deciphers global central bank communications.
problem Misinterpretations of central bank communications can disproportionately impact vulnerable populations.
method Developed the World Central Banks (WCB) dataset, annotated and reviewed sentences, defined tasks, and benchmarked models.
result A model trained on aggregated data across banks outperforms models trained on individual bank data.
NLP tasks are often limited by scarcity of manually annotated data. In social media sentiment analysis and related tasks, researchers have therefore used binarized emoticons and specific hashtags as forms of distant supervision. Our paper shows that by extending the distant supervision to a more diverse set of noisy la…
Improves uncertainty estimates in lung node segmentation.
problem Inaccurate uncertainty estimates in medical segmentation.
method Supervised learning using multi-grader annotation variability.
result Improves predictive uncertainty estimates and sample diversity.
Waymo Open Dataset provides a large, diverse, and synchronized LiDAR and camera dataset for autonomous driving research.
problem Limited diversity and scale in existing self-driving datasets hinder real-world problem alignment.
method Developed a new large-scale, high-quality, diverse dataset with synchronized LiDAR and camera data.
result The dataset is 15x more diverse than the largest existing dataset based on a proposed diversity metric.
Paper learns a versatile model from diverse networks without annotations.
problem Combining knowledge from different specialized networks without access to their training data.
method Transforms features of diverse networks into a common space and forces a student model to mimic them.
result The student model outperforms individual teacher models on various benchmarks.
Active learning selects high-quality examples for text-to-SQL systems.
problem Efficiently annotate large language models for text-to-SQL systems.
method Formalizes example selection as a constrained experimental design problem over semantic query embeddings, proposing a stratified greedy algorithm that maximizes heteroscedastic mutual information.
result Proposed method significantly reduces labeling effort while maintaining high text-to-SQL retrieval accuracy.
We demonstrate a conditional autoregressive pipeline for efficient music recomposition, based on methods presented in van den Oord et al.(2017). Recomposition (Casal & Casey, 2010) focuses on reworking existing musical pieces, adhering to structure at a high level while also re-imagining other aspects of the work. This…
Paper presents an active learning method to reduce skin lesion analysis annotation cost.
problem High cost of manual annotation in skin lesion analysis.
method Decoupled sample selection criteria for informativeness and representativeness, combined with pixel space aggregation for data augmentation.
result 50% annotated data suffices for state-of-the-art performance, outperforming other methods.
NeuCrowd creates high-quality samples from crowdsourced labels to improve representation learning.
problem Limited and inconsistent crowdsourced labels for representation learning.
method Unified framework that generates high-quality n-tuplet samples and learns a neural sampling network.
result NeuCrowd outperforms state-of-the-art baselines in prediction accuracy and AUC.
Improves cross-lingual NER by projecting entities from one language to another.
problem Limited annotated corpora for named entity recognition in many languages.
method Uses machine translation twice: first for sentences, then for entities; matches based on ortho- and phonetic similarity; identifies matches using distributional statistics.
result Improves cross-lingual NER by an average of 4.1 points on 5 diverse languages.
FairGround offers a diverse dataset corpus for fair ML research.
problem Lack of diverse, well-annotated datasets in fair ML research.
method Unified framework and Python package for reproducible fair ML research.
result Advances reproducibility and generalizability of fair ML research.
ORIL learns a reward function from unlabeled data to improve robot learning.
problem Leveraging unlabeled data for robot learning.
method ORIL learns a reward function from demonstrator and unlabeled trajectories, annotates data, and trains an agent via offline reinforcement learning.
result ORIL consistently outperforms BC agents on various robotic tasks.
Study shows annotation instrument design affects model performance in hate speech detection.
problem Impact of annotation instrument design on model performance in hate speech detection.
method Collected annotations from five experimental conditions of an annotation instrument, fine-tuned BERT models on each dataset, evaluated performance on holdout portion.
result Significant differences in model performance and annotations across conditions.
Semi-Supervised Multi-aspect Detection of Misinformation using Hierarchical Joint Decompositioncs.SI
HiJoD detects misinformation using multiple aspects and outperforms state-of-the-art methods.
problem Detecting misinformation with limited labeled data.
method Hierarchical Joint Decomposition framework combining content, context, and domain features.
result HiJoD achieves F1-scores of 74% and 81% on Twitter and Politifact datasets respectively, and is faster than similar approaches.
Active learning selects both observations and annotation precision for Gaussian Processes.
problem Costly annotation in supervised learning.
method Proposes an active learning algorithm that selects observations and annotation precision, using a modified BALD objective.
result Empirically shows the benefits of adjusting annotation precision in active learning.
Novel method decorrelates batches of triplets for active metric learning.
problem Correlation among triplets degrades active learning performance.
method Proposes a novel method to decorrelate batches of triplets, balancing informativeness and diversity.
result Method outperforms state-of-the-art in active metric learning.
One of the problems on the way to successful implementation of neural networks is the quality of annotation. For instance, different annotators can annotate images in a different way and very often their decisions do not match exactly and in extreme cases are even mutually exclusive which results in noisy annotations a…
Generative model combines multi-dimensional annotations for more accurate ground truth estimation.
problem Inaccurate ground truth estimation from naive annotators' multi-dimensional annotations.
method Proposes a joint multi-dimensional model for global and time-series annotation fusion using Expectation-Maximization algorithm.
result More accurate ground truth estimates through joint modeling of multiple dimensions.
Survey on AL strategies for cost-effective annotation in classification.
problem Real-world AL challenges due to human annotators' limitations.
method Categorizes 60 real-world AL strategies considering multiple annotators, query types, and cost schemes.
result General real-world AL strategy introduced for categorization of 60 strategies.
Robustness to Subpopulation Shift with Domain Label Noise via Regularized Annotation of Domainscs.LG
RAD improves robustness to domain annotation noise without explicit domain annotations.
problem Robustness to domain annotation noise in training data.
method Regularized Annotation of Domains (RAD) for last layer retraining.
result RAD outperforms state-of-the-art methods even with 5% noise in training data.
Accurate annotation of medical image is the crucial step for image AI clinical application. However, annotating medical image will incur a great deal of annotation effort and expense due to its high complexity and needing experienced doctors. To alleviate annotation cost, some active learning methods are proposed. But …
We introduce a new benchmark for coreference resolution and NLI, Knowref, that targets common-sense understanding and world knowledge. Previous coreference resolution tasks can largely be solved by exploiting the number and gender of the antecedents, or have been handcrafted and do not reflect the diversity of naturall…
PTBCC improves accuracy in multi-class annotation aggregation by learning from prototype confusion matrices.
problem Inaccurate and insufficient confusion matrices for annotators in multi-class classification tasks.
method PTBCC (ProtoType learning-driven Bayesian Classifier Combination) uses prototype confusion matrices to capture annotator expertise.
result PTBCC achieves up to 15% accuracy improvement and 3% higher average accuracy compared to existing methods.
ActiveLab improves classifier accuracy with fewer annotations by re-labeling.
problem Imperfect labels from multiple annotators in real-world data.
method ActiveLab automatically decides when to re-label examples for better classifier training.
result ActiveLab trains more accurate classifiers with fewer annotations.
Bayesian methods improve text annotation quality.
problem Inconsistent and unreliable human annotations in natural language processing.
method Two semi-supervised Bayesian methods: a deep learning model and an ensemble method.
result Bayesian methods enhance the reliability and performance of BERT models.
TBAL reduces manual annotation but requires validated data.
problem Creating large, high-quality labeled datasets.
method Threshold-based auto-labeling using human validation data.
result Sample complexity bounds on validation data needed.
Paper proposes an efficient method for bounding box annotation in object detection.
problem Manual annotation of bounding boxes is tedious and resource-intensive.
method Iterative training of object detector on small batches of labeled images, with human annotator correcting errors.
result Significant reduction in human annotation effort, up to 75%.
Human annotations serve an important role in computational models where the target constructs under study are hidden, such as dimensions of affect. This is especially relevant in machine learning, where subjective labels derived from related observable signals (e.g., audio, video, text) are needed to support model trai…