Method curates cost-effective, high-quality datasets using AI models.
problem Costly manual labeling of datasets.
method Probably Approximately Correct Labels (PACL) method.
result Curates high-quality datasets with low overall labeling error.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “label quality”
Crowdsourcing is a relatively economic and efficient solution to collect annotations from the crowd through online platforms. Answers collected from workers with different expertise may be noisy and unreliable, and the quality of annotated data needs to be further maintained. Various solutions have been attempted to ob…
TBAL reduces manual annotation but requires validated data.
problem Creating large, high-quality labeled datasets.
method Threshold-based auto-labeling using human validation data.
result Sample complexity bounds on validation data needed.
FOCUS addresses label quality disparity in FL for healthcare applications.
problem Label quality disparity in federated learning for healthcare applications.
method FOCUS maintains a small set of benchmark samples and computes the mutual cross-entropy between local and benchmark datasets to quantify label credibility. It then adjusts client weights based on credibility values.
result FOCUS effectively reduces the impact of noisy labels from clients, improving model performance.
Crowdlab uses classifiers to estimate consensus labels and annotator quality.
problem Leveraging multiple annotators for data classification.
method Weighted ensemble approach using any trained classifier.
result Superior estimates for consensus labels and annotator quality.
The paper investigates how dataset quality and heterogeneity affect model confidence in machine learning.
problem Understanding how dataset quality and heterogeneity impact model confidence in machine learning.
method The study uses theoretical explanations and experimental demonstrations to investigate the effects of dataset size, label noise, and class heterogeneity on model confidence.
result Label noise reduces model confidence, while reduced dataset size increases it, and class heterogeneity leads to inconsistent confidence across classes.
In the recent years, we have witnessed the development of multi-label classification methods which utilize the structure of the label space in a divide and conquer approach to improve classification performance and allow large data sets to be classified efficiently. Yet most of the available data sets have been provide…
Paper proposes Universum GANs to improve GANs with limited labeled data.
problem Limited labeled data makes supervised learning challenging.
method Proposes Universum GANs with evolving discriminator loss.
result Improved discriminator accuracy and high quality data generation.
The scalability of the labeling process and the attainable quality of labels have become limiting factors for many applications of machine learning. The programmatic creation of labeled datasets via the synthesis of noisy heuristics provides a promising avenue to address this problem. We propose to improve modeling of …
Generative models have made immense progress in recent years, particularly in their ability to generate high quality images. However, that quality has been difficult to evaluate rigorously, with evaluation dominated by heuristic approaches that do not correlate well with human judgment, such as the Inception Score and …
Majority of state-of-the-art deep learning methods are discriminative approaches, which model the conditional distribution of labels given inputs features. The success of such approaches heavily depends on high-quality labeled instances, which are not easy to obtain, especially as the number of candidate classes increa…
Enhanced nearest neighbor improves accuracy in crowdsourced data.
problem Noise in crowdsourced labels degrades classification accuracy.
method Developed two algorithms to estimate worker quality and an enhanced nearest neighbor classifier.
result Proposed methods achieve the same regret as oracle version based on expert data.
Uncertainty of labels in clinical data resulting from intra-observer variability can have direct impact on the reliability of assessments made by deep neural networks. In this paper, we propose a method for modelling such uncertainty in the context of 2D echocardiography (echo), which is a routine procedure for detecti…
Enhances GNNs by improving input data quality from topology and labels.
problem Poor quality of graph data limits GNN performance.
method Improves graph data quality using model outputs for better semi-supervised node classification.
result SEG consistently improves GNN performance across various datasets.
Iterative Label Improvement: Robust Training by Confidence Based Filtering and Dataset Partitioningcs.LG
Improved deep learning models with less labelled data and better label quality.
problem High costs and effort in training deep neural networks with label errors.
method Iterative label improvement using confidence-based filtering and dataset partitioning.
result Significant improvement in label quality and model accuracy.
NeuCrowd creates high-quality samples from crowdsourced labels to improve representation learning.
problem Limited and inconsistent crowdsourced labels for representation learning.
method Unified framework that generates high-quality n-tuplet samples and learns a neural sampling network.
result NeuCrowd outperforms state-of-the-art baselines in prediction accuracy and AUC.
GCNs help in diagnosing label scarcity and feature quality on graphs.
problem Understanding when GCNs improve node classification.
method Simulated label scarcity, feature ablation, and per-class analysis.
result GCNs provide largest gains under extreme label scarcity, matching original performance with noisy features, but hurt when homophily is low and features are strong.
As a new approach to train generative models, \emph{generative adversarial networks} (GANs) have achieved considerable success in image generation. This framework has also recently been applied to data with graph structures. We propose labeled-graph generative adversarial networks (LGGAN) to train deep generative model…
Survey on deep learning robust training methods for noisy labels.
problem Dealing with noisy labels in deep learning models.
method Comprehensive review of 62 robust training methods categorized by their approach.
result Analysis of noise rate estimation and evaluation methodologies.
Study on noise models for noisy labels in NLP.
problem Quality of noise models from noisy labels.
method Theoretical analysis and synthetic dataset creation.
result Expected error of noise models derived.
Solving real-world problems, particularly with deep learning, relies on the availability of abundant, quality data. In this paper we develop a novel framework that maximises the utility of time-series datasets that contain only small quantities of expertly-labelled data, larger quantities of weakly (or coarsely) labell…
Scarcity of labeled data is a bottleneck for supervised learning models. A paradigm that has evolved for dealing with this problem is data programming. An existing data programming paradigm allows human supervision to be provided as a set of discrete labeling functions (LF) that output possibly noisy labels to input in…
Training deep neural networks requires massive amounts of training data, but for many tasks only limited labeled data is available. This makes weak supervision attractive, using weak or noisy signals like the output of heuristic methods or user click-through data for training. In a semi-supervised setting, we can use a…
In this work we addressed the issue of applying a stochastic classifier and a local, fuzzy confusion matrix under the framework of multi-label classification. We proposed a novel solution to the problem of correcting label pairwise ensembles. The main step of the correction procedure is to compute classifier-specific c…
Study shows resampling labels improves classifier performance in noisy data.
problem Balancing sample size vs label reliability in noisy data.
method Comparing different validation strategies and analyzing MNIST database with varying noise levels.
result Classifier performance declines with high incorrect labels, highlighting the importance of resampling.
The paper proposes a method to detect and filter noisy or mislabeled data using pointwise mutual information.
problem Detecting and filtering noisy or mislabeled data in deep learning models.
method A mutual information-based framework quantifying statistical dependencies between inputs and labels.
result The method effectively filters low-quality samples, improving classification accuracy by up to 15%.
Magnetic resonance (MR) imaging offers a wide variety of imaging techniques. A large amount of data is created per examination which needs to be checked for sufficient quality in order to derive a meaningful diagnosis. This is a manual process and therefore time- and cost-intensive. Any imaging artifacts originating fr…
Weakly supervised data are widespread and have attracted much attention. However, since label quality is often difficult to guarantee, sometimes the use of weakly supervised data will lead to unsatisfactory performance, i.e., performance degradation or poor performance gains. Moreover, it is usually not feasible to man…
Class labels have been empirically shown useful in improving the sample quality of generative adversarial nets (GANs). In this paper, we mathematically study the properties of the current variants of GANs that make use of class label information. With class aware gradient and cross-entropy decomposition, we reveal how …
Automated quality control for seismic data reduces human labor and time.
problem Costly and time-consuming manual QC of seismic data.
method Active learning to select and label relevant seismic data.
result Active learning technique reduces QC time and improves accuracy.
Learning exists in the context of data, yet notions of confidence typically focus on model predictions, not label quality. Confident learning (CL) is an alternative approach which focuses instead on label quality by characterizing and identifying label errors in datasets, based on the principles of pruning noisy data, …
In this work, we addressed the issue of applying a stochastic classifier and a local, fuzzy confusion matrix under the framework of multi-label classification. We proposed a novel solution to the problem of correcting label pairwise ensembles. The main step of the correction procedure is to compute classifier- specific…
Reduces test set maintenance effort by 80-100%.
problem Lack of proper and up-to-date test sets in real-world scenarios.
method Simple technique to reduce labeling effort.
result Significant reduction in test set maintenance effort (80-100%).
FUSE improves verification quality without ground truth labels.
problem Verification of model outputs using imperfect judges and reward models.
method Ensembling verifiers without ground truth labels using spectral algorithms.
result FUSE matches or improves upon semi-supervised alternatives in test-time scaling experiments.
New measure assesses time series pre-training data quality without labels.
problem Challenges in collecting diverse pre-training datasets for time series classification.
method Contrastive-learning-based foundation model and contrastive accuracy measure.
result Contrastive accuracy correlates with model performance on downstream tasks.
Collecting large training datasets, annotated with high-quality labels, is costly and time-consuming. This paper proposes a novel framework for training deep convolutional neural networks from noisy labeled datasets that can be obtained cheaply. The problem is formulated using an undirected graphical model that represe…
An automated metric to evaluate dialogue quality is vital for optimizing data driven dialogue management. The common approach of relying on explicit user feedback during a conversation is intrusive and sparse. Current models to estimate user satisfaction use limited feature sets and rely on annotation schemes with low …
SMART is an open source web application designed to help data scientists and research teams efficiently build labeled training data sets for supervised machine learning tasks. SMART provides users with an intuitive interface for creating labeled data sets, supports active learning to help reduce the required amount of …
Although shill bidding is a common auction fraud, it is however very tough to detect. Due to the unavailability and lack of training data, in this study, we build a high-quality labeled shill bidding dataset based on recently collected auctions from eBay. Labeling shill biding instances with multidimensional features i…
Crowdsourcing utilizes the wisdom of crowds for collective classification via information (e.g., labels of an item) provided by labelers. Current crowdsourcing algorithms are mainly unsupervised methods that are unaware of the quality of crowdsourced data. In this paper, we propose a supervised collective classificatio…
Cross-prediction improves inference from small labeled datasets.
problem Valid inference from small labeled datasets with imperfect predictions.
method Imputes missing labels via machine learning and debiases predictions.
result Inferences achieve desired error probability and are more powerful.
There is a rapidly increasing interest in crowdsourcing for data labeling. By crowdsourcing, a large number of labels can be often quickly gathered at low cost. However, the labels provided by the crowdsourcing workers are usually not of high quality. In this paper, we propose a minimax conditional entropy principle to…
The paper proposes a method to assess and improve data quality using GBDT training dynamics.
problem Improving data quality in datasets with noisy labels and varying contributions.
method Metrics computed from training dynamics of Gradient Boosting Decision Trees (GBDTs).
result The method achieved the best results compared to other approaches.
CheXpert++: Approximating the CheXpert labeler for Speed,Differentiability, and Probabilistic Outputcs.LG
CheXpert++ improves CheXpert's accuracy and usability for medical radiology reports.
problem Infeasibility of obtaining ground truth labels for medical data.
method BERT-based approximation of CheXpert, addressing speed, differentiability, and probabilistic output.
result Achieves 99.81% parity with CheXpert, significantly faster, differentiable, and probabilistic.
In many classification problems unlabelled data is abundant and a subset can be chosen for labelling. This defines the context of active learning (AL), where methods systematically select that subset, to improve a classifier by retraining. Given a classification problem, and a classifier trained on a small number of la…
Proposes a method to generate high-quality candlestick data for financial trading.
problem Lack of labeled financial trading data.
method Modified Local Search Attack Sampling method for candlestick data augmentation.
result Generated high-quality data that are hard to distinguish by humans.
Deep learning for supervised learning has achieved astonishing performance in various machine learning applications. However, annotated data is expensive and rare. In practice, only a small portion of data samples are annotated. Pseudo-ensembling-based approaches have achieved state-of-the-art results in computer visio…
New methods for handling time-varying label noise in time series classification.
problem Temporal label noise in time series classification tasks.
method Proposed methods to estimate temporal label noise function directly from data.
result Our methods lead to state-of-the-art performance under diverse types of temporal label noise.