ECN framework improves training on noisy structured labels.
problem Structured errors in fine-grained annotations lead to biased models.
method Error-Correcting Networks (ECN) framework.
result ECN improves fine-grained annotation prediction.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “annotation errors”
Generative AI reduces IR evaluation costs but introduces errors; this work provides reliable CIs.
problem Generating relevance annotations using AI introduces errors that affect IR evaluation metrics.
method Proposes two methods: prediction-powered inference and conformal risk control to place reliable CIs around IR metrics.
result Proposed methods accurately capture both variance and bias in evaluation based on AI-generated annotations.
Bayesian method improves segmentation accuracy with noisy labels.
problem Annotation errors in semantic segmentation due to mislabeling and spatial correlations.
method Approximate Bayesian estimation with spatially correlated discrete distributions and variational inference.
result The method achieves performance comparable to clean labels under moderate noise levels.
Paper proposes an efficient method for bounding box annotation in object detection.
problem Manual annotation of bounding boxes is tedious and resource-intensive.
method Iterative training of object detector on small batches of labeled images, with human annotator correcting errors.
result Significant reduction in human annotation effort, up to 75%.
In this work, we introduce a novel framework that employs cluster annotation to boost active learning by reducing the number of human interactions required to train deep neural networks. Instead of annotating single samples individually, humans can also label clusters, producing a higher number of annotated samples wit…
Paper introduces robust learning from feature feedback, even with annotator errors.
problem Learning from human feedback on discriminative features, especially when annotators make mistakes.
method Develops a robust framework for learning with imperfect feedback, deriving regret bounds in adversarial and stochastic settings.
result Regret bounds independent of feature number, showing robust learning is not reducible to non-robust settings.
LLMs struggle with zero-shot annotation tasks due to model-internalized priors.
problem Impact of model-internalized priors on LLM performance in zero-shot annotation tasks.
method Investigated three dimensions: familiarity, decision stickiness, and susceptibility to misaligned task definitions.
result Nearly two-thirds of zero-shot errors are resistant to correction, with a rescue rate of 34.8%. Definition-Specific Familiarity (DSF) shows a positive association with model performance.
A new model cleans vocal note event annotations in music.
problem Erroneous labels in music datasets.
method Contrastive learning to automatically create local deformations of likely correct labels.
result Transcription model accuracy improves with the proposed strategy.
Study proposes a stopping criterion for active learning based on error stability.
problem Improving predictive performance in active learning by adaptively annotating samples.
method Proposes a stopping criterion based on error stability for Bayesian active learning.
result Demonstrates the proposed criterion stops active learning at the appropriate timing for various models and datasets.
Majority Vote is optimal for reliable data labeling under certain conditions.
problem Reliable data labeling requires aggregating multiple annotators' labels, but the optimality of Majority Vote is not well understood.
method Characterized conditions under which Majority Vote achieves the optimal label estimation error.
result Majority Vote optimally recovers labels for a given class distribution under tolerable annotation noise limits.
Build accurate DNN models requires training on large labeled, context specific datasets, especially those matching the target scenario. We believe advances in wireless localization, working in unison with cameras, can produce automated annotation of targets on images and videos captured in the wild. Using pedestrian an…
Annotation errors can significantly hurt classifier performance, yet datasets are only growing noisier with the increased use of Amazon Mechanical Turk and techniques like distant supervision that automatically generate labels. In this paper, we present a robust extension of logistic regression that incorporates the po…
Interactive IL beats BC by state-wise annotation cost.
problem Behavior Cloning struggles with annotation cost in sequential decision making.
method Proved Stagger and Warm Stagger algorithms to outperform BC.
result Interactive and hybrid IL methods outperform BC with state-wise annotation.
Grammatical error correction, like other machine learning tasks, greatly benefits from large quantities of high quality training data, which is typically expensive to produce. While writing a program to automatically generate realistic grammatical errors would be difficult, one could learn the distribution of naturally…
Paper tackles active learning under human label variation, proposing a new framework.
problem Active learning assumes a single ground truth, ignoring human label variation.
method Survey and propose a conceptual framework for HLV-aware active learning.
result Lay a conceptual foundation for HLV-aware active learning.
OLALA automates document layout annotation by selecting ambiguous regions for labeling.
problem Efficiently annotating complex document layouts with limited resources.
method Object-Level Active Learning framework that selects ambiguous regions for labeling and uses semi-automatic correction.
result OLALA significantly boosts model performance and improves annotation efficiency.
We study the problem of training an accurate linear regression model by procuring labels from multiple noisy crowd annotators, under a budget constraint. We propose a Bayesian model for linear regression in crowdsourcing and use variational inference for parameter estimation. To minimize the number of labels crowdsourc…
Decision support tools that rely on supervised learning require large amounts of expert annotations. Using past radiological reports obtained from hospital archiving systems has many advantages as training data above manual single-class labels: they are expert annotations available in large quantities, covering a popul…
This paper refines human labeling as a measurement process, revealing four sources of variation.
problem Systematic variation in human labeling obscures model learning.
method Introduces a statistical framework to decompose labeling outcomes.
result Empirical evidence for four components of labeling variation.
Enhances privacy in data annotation and inspection with synthetic data.
problem Improving privacy in machine learning tasks like data annotation and inspection.
method Employing Bayesian differential privacy to generate higher-fidelity synthetic data.
result Produces higher-fidelity samples, detecting more subtle data errors and biases.
Selective prediction framework reduces errors in molecular structure identification from MS/MS.
problem High-stakes applications require reliable molecular structure identification from MS/MS data.
method Selective prediction framework using risk-coverage tradeoff and uncertainty quantification.
result First-order confidence measures and retrieval-level aleatoric uncertainty achieve strong risk-coverage tradeoffs.
XRM discovers environments without human annotations for OOD methods.
problem Costly and biased manual annotations limit OOD methods.
method XRM trains twin networks to mimic mistakes, eliminating hyper-parameters.
result XRM achieves oracle worst-group-accuracy for OOD methods.
Optimizes data labeling for causal effect estimation with missing outcomes.
problem Estimating causal effects with missing outcome data and budget constraints.
method Optimizes batch sampling probability to minimize variance of causal inference estimator.
result Achieves lower mean-squared error with fewer labeled data points.
This paper analyzes and improves active learning techniques for real-world projects.
problem Reducing labelling effort in machine learning models with real-world constraints.
method Systematic study of active learning issues, proposing techniques to address model convergence, annotation error, and dataset imbalance.
result Presentation of two techniques to speed up active learning: partial uncertainty sampling and larger query size.
LambdaNet infers TypeScript types using graph neural networks.
problem Automatic inference of TypeScript type annotations.
method Graph Neural Network for type dependency graph analysis.
result LambdaNet outperforms existing methods by 14%.
Method curates cost-effective, high-quality datasets using AI models.
problem Costly manual labeling of datasets.
method Probably Approximately Correct Labels (PACL) method.
result Curates high-quality datasets with low overall labeling error.
New method identifies wrongly predicted samples for active learning.
problem Identifying important samples for machine learning models.
method A sample selection criterion based on model prediction and its effect on generalization error.
result State-of-the-art results and better rates at identifying wrongly predicted samples.
Study shows annotation instrument design affects model performance in hate speech detection.
problem Impact of annotation instrument design on model performance in hate speech detection.
method Collected annotations from five experimental conditions of an annotation instrument, fine-tuned BERT models on each dataset, evaluated performance on holdout portion.
result Significant differences in model performance and annotations across conditions.
The paper extends influence functions to sequence tagging tasks for better model interpretability.
problem Lack of interpretability methods for sequence tagging models.
method Define and compute influence of training instance segments on test segment predictions.
result The segment influence method tracks with true influence and identifies annotation errors.
New method uses imperfect LLM annotations for valid statistical inference in social science.
problem Inaccurate large language model annotations in social science research.
method Design-based supervised learning (DSL) combining imperfect LLM surrogates with gold-standard labels.
result DSL provides valid statistical inference with comparable predictive accuracy to existing methods.
Learning-based methods for visual segmentation have made progress on particular types of segmentation tasks, but are limited by the necessary supervision, the narrow definitions of fixed tasks, and the lack of control during inference for correcting errors. To remedy the rigidity and annotation burden of standard appro…
Active learning selects both observations and annotation precision for Gaussian Processes.
problem Costly annotation in supervised learning.
method Proposes an active learning algorithm that selects observations and annotation precision, using a modified BALD objective.
result Empirically shows the benefits of adjusting annotation precision in active learning.
One of the problems on the way to successful implementation of neural networks is the quality of annotation. For instance, different annotators can annotate images in a different way and very often their decisions do not match exactly and in extreme cases are even mutually exclusive which results in noisy annotations a…
The paper analyzes how conformal prediction works with contaminated reference data.
problem The impact of contamination on the validity and power of conformal prediction methods.
method The paper analyzes the impact of contamination on the validity of conformal methods and proposes a data-cleaning framework to enhance power.
result The proposed data-cleaning framework can effectively enhance power while maintaining type-I error control.
Generative model combines multi-dimensional annotations for more accurate ground truth estimation.
problem Inaccurate ground truth estimation from naive annotators' multi-dimensional annotations.
method Proposes a joint multi-dimensional model for global and time-series annotation fusion using Expectation-Maximization algorithm.
result More accurate ground truth estimates through joint modeling of multiple dimensions.
Face recognition system trained with noisy labels.
problem Label noise in training deep learning classifiers.
method Review and apply recent methods to manage noisy annotations.
result Improved performance of face recognition system with noisy labels.
Survey on AL strategies for cost-effective annotation in classification.
problem Real-world AL challenges due to human annotators' limitations.
method Categorizes 60 real-world AL strategies considering multiple annotators, query types, and cost schemes.
result General real-world AL strategy introduced for categorization of 60 strategies.
Robustness to Subpopulation Shift with Domain Label Noise via Regularized Annotation of Domainscs.LG
RAD improves robustness to domain annotation noise without explicit domain annotations.
problem Robustness to domain annotation noise in training data.
method Regularized Annotation of Domains (RAD) for last layer retraining.
result RAD outperforms state-of-the-art methods even with 5% noise in training data.
Words That Unite The World: A Unified Framework for Deciphering Central Bank Communications Globallycs.CL
Unified framework deciphers global central bank communications.
problem Misinterpretations of central bank communications can disproportionately impact vulnerable populations.
method Developed the World Central Banks (WCB) dataset, annotated and reviewed sentences, defined tasks, and benchmarked models.
result A model trained on aggregated data across banks outperforms models trained on individual bank data.
Accurate annotation of medical image is the crucial step for image AI clinical application. However, annotating medical image will incur a great deal of annotation effort and expense due to its high complexity and needing experienced doctors. To alleviate annotation cost, some active learning methods are proposed. But …
Machine learning approaches hold great potential for the automated detection of lung nodules in chest radiographs, but training the algorithms requires vary large amounts of manually annotated images, which are difficult to obtain. Weak labels indicating whether a radiograph is likely to contain pulmonary nodules are t…
PTBCC improves accuracy in multi-class annotation aggregation by learning from prototype confusion matrices.
problem Inaccurate and insufficient confusion matrices for annotators in multi-class classification tasks.
method PTBCC (ProtoType learning-driven Bayesian Classifier Combination) uses prototype confusion matrices to capture annotator expertise.
result PTBCC achieves up to 15% accuracy improvement and 3% higher average accuracy compared to existing methods.
ActiveLab improves classifier accuracy with fewer annotations by re-labeling.
problem Imperfect labels from multiple annotators in real-world data.
method ActiveLab automatically decides when to re-label examples for better classifier training.
result ActiveLab trains more accurate classifiers with fewer annotations.
Bayesian methods improve text annotation quality.
problem Inconsistent and unreliable human annotations in natural language processing.
method Two semi-supervised Bayesian methods: a deep learning model and an ensemble method.
result Bayesian methods enhance the reliability and performance of BERT models.
Human annotations serve an important role in computational models where the target constructs under study are hidden, such as dimensions of affect. This is especially relevant in machine learning, where subjective labels derived from related observable signals (e.g., audio, video, text) are needed to support model trai…
Paper proposes a weak supervision technique for CNN semantic segmentation of lung diseases using partially annotated data.
problem Creating annotated datasets for semantic segmentation of lung diseases is laborious and time-consuming.
method Proposes a weak supervision technique that utilizes partially annotated datasets to improve CNN semantic segmentation accuracy.
result Significantly improved segmentation accuracy using partially annotated datasets.
Q&A Label Learningcs.LG
A new Q&A labeling method for assigning labels in machine learning.
problem Assigning labels to instances in supervised machine learning.
method Developed a Q&A labeling method involving a question generator and an annotator.
result The derived label generative model is consistent with previous studies.
The Impact of Annotation Guidelines and Annotated Data on Extracting App Features from App Reviewscs.IR
Annotation guidelines used to guide the annotation of training and evaluation datasets can have a considerable impact on the quality of machine learning models. In this study, we explore the effects of annotation guidelines on the quality of app feature extraction models. As a main result, we propose several changes to…