Cluster annotation reduces active learning human interactions.
problem Reducing human interactions in active learning.
method Cluster annotation to label multiple samples at once.
result 82% and 87% less human interactions for CIFAR-10 and EuroSAT datasets.
Selective clustering annotated using modes of projections (SCAMP) is a new clustering algorithm for data in Rp. SCAMP is motivated from the point of view of non-parametric mixture modeling. Rather than maximizing a classification likelihood to determine cluster assignments, SCAMP casts clustering as a searc…
This paper clusters networks with annotated time-series data using kernel-ARMA and Grassmannian geometry.
problem Clustering networks with annotated time-series data, including state, node, and subnetwork clustering.
method Extract features from time-series data using kernel-ARMA, map onto Grassmannian, and cluster using Riemannian geometry.
result The proposed framework outperforms state-of-the-art clustering schemes on brain-network data.
Crowdsourcing has been proven to be an effective and efficient tool to annotate large datasets. User annotations are often noisy, so methods to combine the annotations to produce reliable estimates of the ground truth are necessary. We claim that considering the existence of clusters of users in this combination step c…
A framework clusters vehicle motion trajectories efficiently.
problem Costly manual annotation of vehicle motion data.
method Five-stage framework: align, embed, extract, embed, cluster.
result Framework achieves promising results on real-world dataset.
Study explores how to explain deep neural networks using interactive naming.
problem Explaining deep neural networks' decisions in human-understandable terms.
method Developed an interactive naming interface for human annotators to cluster activation maps into visual concepts.
result Significant agreement among annotators about visual concepts, many activation maps have recognizable concepts.
Efficiently annotates hierarchical structure in images using 2AFC testing and deep metric learning.
problem Lack of efficient methods for hierarchical annotation of high-dimensional data like images.
method Two-alternative-forced-choice (2AFC) testing and deep metric learning for embedding data in semantic space.
result Successfully hierarchically clusters data, achieving finer granularity than original labels.
Proposes a self-training method for automatic data annotation.
problem Lack of large annotated datasets for supervised deep learning.
method Deep Bayesian Self-Training using variational inference and clustering.
result Superior performance over standard Self-Training baselines.
New system constructs cell-type taxonomy across multiple samples.
problem Challenges in matching clusters from different datasets.
method Combines Optimal Transport with Relaxed Marginal Constraints (OT-RMC) for simultaneous alignment of clusters across multiple samples.
result Highly accurate annotation of cell types and sample-level feature extraction.
The hidden Markov model (HMM) is a widely-used generative model that copes with sequential data, assuming that each observation is conditioned on the state of a hidden Markov chain. In this paper, we derive a novel algorithm to cluster HMMs based on the hierarchical EM (HEM) algorithm. The proposed algorithm i) cluster…
Paper proposes an alternative to anchor points for learning with noisy labels.
problem Learning with noisy labels is challenging due to inaccurate labels.
method Estimates transition matrix using clusterability condition and noisy labels.
result Estimation of transition matrix is more accurate and efficient than anchor points.
Paper develops active learning for clustering unknown pairwise similarities.
problem Learning positive and negative pairwise similarities efficiently.
method Generic active learning framework for correlation clustering.
result Demonstrates effectiveness of query strategies in clustering.
ProtoTransfer learns from unlabeled data to classify unseen tasks.
problem Few-shot classification with limited labeled data.
method Self-supervised prototypical transfer learning.
result ProtoTransfer outperforms unsupervised meta-learning methods.
A deep generative model with a relational model tackles noisy pairwise comparisons for semi-supervised clustering.
problem Noisy pairwise comparisons on a small subset of data for clustering.
method Deep generative model (DGM) and statistical relational model, sharing latent variables, Bayesian variant, stochastic variational inference.
result Outperforms previous crowdsourced clustering methods on synthetic and real-world datasets.
Unsupervised learning classifies transient noise in gravitational wave detectors.
problem Transient noise interferes with gravitational wave signals, causing instability.
method Combines variational autoencoder and invariant information clustering.
result Consistent classification with Gravity Spy project labels.
In this demo paper, we introduce the DARPA D3M program for automatic machine learning (ML) and JPL's MARVIN tool that provides an environment to locate, annotate, and execute machine learning primitives for use in ML pipelines. MARVIN is a web-based application and associated back-end interface written in Python that e…
Method extracts time-localized clusters to explain deep learning models in ECG analysis.
problem Limited understanding of deep learning models in ECG analysis.
method Extracts time-localized clusters from model's internal representations.
result Enhances trust in AI-driven diagnostics and reveals clinically relevant patterns.
Unsupervised deep learning detects image quality issues without annotations.
problem Anomaly detection in images with quality issues.
method Use of deep generative models, specifically normalizing flows, for anomaly detection.
result Accurate semantic labeling and quality trends observed in images.
BELA infers labels for unlabeled data at lower cost.
problem Efficiently labeling large unlabeled datasets.
method Supervised splitting with bias-reduction techniques.
result BELA outperforms existing adaptive labeling strategies.
Approach for training deep nets with unlabeled patches.
problem Training deep neural networks with detailed expert annotations.
method Cluster-based learning from weakly labeled bags in latent space.
result Improved performance on Camelyon dataset.
Graph neural network predicts optimal coarse-grained mapping operators.
problem Optimal coarse-grained mapping operators selection for molecular dynamics simulations.
method Graph Neural Network (DSGPM) trained on expert-annotated data.
result DSGPM outperforms state-of-the-art methods in graph segmentation.
This work creates a system for understanding human movement in spaces.
problem Simplify communication and interaction between robots and humans in spatial tasks.
method Uses unsupervised learning with neural autoencoding to learn continuous representations of spatio-temporal trajectory data.
result Proposes a method to form prototypical representations of movement based on spatial context.
Study shows annotation instrument design affects model performance in hate speech detection.
problem Impact of annotation instrument design on model performance in hate speech detection.
method Collected annotations from five experimental conditions of an annotation instrument, fine-tuned BERT models on each dataset, evaluated performance on holdout portion.
result Significant differences in model performance and annotations across conditions.
Improves machine learning model training with noisy labels using active learning and clustering.
problem Training machine learning models with noisy oracles and limited data.
method Uses clustering and importance sampling to select batches of samples, incorporates model uncertainty, and introduces a denoising layer.
result Significant improvement in model accuracy on benchmark datasets compared to existing methods.
New method to estimate doctors' effort in annotating medical images.
problem High effort and expense in annotating medical images.
method Proposes a new criterion to evaluate effort, uses active learning and U-shape network for annotation strategy, and fine annotation platform to reduce effort.
result State-of-the-art segmentation performance achieved with only 60% annotation candidates, reducing effort by 44-47%.
Active learning selects both observations and annotation precision for Gaussian Processes.
problem Costly annotation in supervised learning.
method Proposes an active learning algorithm that selects observations and annotation precision, using a modified BALD objective.
result Empirically shows the benefits of adjusting annotation precision in active learning.
One of the problems on the way to successful implementation of neural networks is the quality of annotation. For instance, different annotators can annotate images in a different way and very often their decisions do not match exactly and in extreme cases are even mutually exclusive which results in noisy annotations a…
BP pretreatment reduces multi-label classification time.
problem Efficiently annotate large label sets for extreme multi-label classification.
method Divide instances into clusters, attach most relevant labels, train on pairs of clusters.
result BP reduces prediction time significantly without sacrificing accuracy.
Generative model combines multi-dimensional annotations for more accurate ground truth estimation.
problem Inaccurate ground truth estimation from naive annotators' multi-dimensional annotations.
method Proposes a joint multi-dimensional model for global and time-series annotation fusion using Expectation-Maximization algorithm.
result More accurate ground truth estimates through joint modeling of multiple dimensions.
Survey on AL strategies for cost-effective annotation in classification.
problem Real-world AL challenges due to human annotators' limitations.
method Categorizes 60 real-world AL strategies considering multiple annotators, query types, and cost schemes.
result General real-world AL strategy introduced for categorization of 60 strategies.
RAD improves robustness to domain annotation noise without explicit domain annotations.
problem Robustness to domain annotation noise in training data.
method Regularized Annotation of Domains (RAD) for last layer retraining.
result RAD outperforms state-of-the-art methods even with 5% noise in training data.
The study challenges the notion that partial data annotation is inferior, suggesting it can sometimes outperform complete annotation.
problem The inefficiency and high cost of completely annotating structured data.
method Information theoretic formulation applied to three diverse structured learning tasks.
result Learning from partial structures can sometimes outperform learning from complete ones.
DAMI uses interpretable regions to select informative samples for deep learning models.
problem Efficiently identifying informative samples for deep learning models with minimal annotation cost.
method Inspired by piece-wise linear interpretability in DNN, DAMI selects samples on different linearly separable regions.
result DAMI outperforms state-of-the-art approaches in tabular data.
A new method uses triplet embeddings to improve human annotation for hidden constructs.
problem Improving human annotation for hidden constructs in machine learning.
method Proposes a novel annotation approach using triplet embeddings to lift absolute annotations to relative comparisons.
result Successfully represents synthetic hidden constructs in time under noisy sampling conditions.
Proposes ConstraintMatch for semi-supervised clustering with unconstrained data.
problem Leveraging unconstrained data alongside constraints for clustering models.
method Semi-supervised context with pseudo-constraining and pseudo-labeling mechanisms.
result Demonstrates effectiveness of ConstraintMatch over baselines.
PTBCC improves accuracy in multi-class annotation aggregation by learning from prototype confusion matrices.
problem Inaccurate and insufficient confusion matrices for annotators in multi-class classification tasks.
method PTBCC (ProtoType learning-driven Bayesian Classifier Combination) uses prototype confusion matrices to capture annotator expertise.
result PTBCC achieves up to 15% accuracy improvement and 3% higher average accuracy compared to existing methods.
ActiveLab improves classifier accuracy with fewer annotations by re-labeling.
problem Imperfect labels from multiple annotators in real-world data.
method ActiveLab automatically decides when to re-label examples for better classifier training.
result ActiveLab trains more accurate classifiers with fewer annotations.
Bayesian methods improve text annotation quality.
problem Inconsistent and unreliable human annotations in natural language processing.
method Two semi-supervised Bayesian methods: a deep learning model and an ensemble method.
result Bayesian methods enhance the reliability and performance of BERT models.
Study improves app feature extraction models with new annotation guidelines and data.
problem Improving the quality and usefulness of app feature extraction models.
method Exploring the effects of annotation guidelines and annotated data on app feature extraction models.
result New annotation guidelines lead to less noisy and more informative app features.
Paper proposes an efficient method for bounding box annotation in object detection.
problem Manual annotation of bounding boxes is tedious and resource-intensive.
method Iterative training of object detector on small batches of labeled images, with human annotator correcting errors.
result Significant reduction in human annotation effort, up to 75%.
Paper proposes a weak supervision technique for CNN semantic segmentation of lung diseases using partially annotated data.
problem Creating annotated datasets for semantic segmentation of lung diseases is laborious and time-consuming.
method Proposes a weak supervision technique that utilizes partially annotated datasets to improve CNN semantic segmentation accuracy.
result Significantly improved segmentation accuracy using partially annotated datasets.
New metric assesses image dataset complexity.
problem Assessing the complexity of image classification datasets.
method Cumulative Spectral Gradient (CSG) derived from spectral clustering.
result CSG correlates with CNN test accuracy and dataset separability.
Improves off-policy evaluation with imperfect annotations.
problem Limited dataset coverage for evaluating new policies.
method Doubly robust estimators combining IS and DM, incorporating counterfactual annotations.
result Using annotations within the DM component yields the most desirable theoretical results.
A popular approach for large scale data annotation tasks is crowdsourcing, wherein each data point is labeled by multiple noisy annotators. We consider the problem of inferring ground truth from noisy ordinal labels obtained from multiple annotators of varying and unknown expertise levels. Annotation models for ordinal…
Paper proposes UIS-RNN for fully supervised speaker diarization.
problem Speaker diarization with unknown number of speakers and time-stamped labels.
method Parameter-sharing RNNs with interleaved states, ddCRP for clustering.
result 7.6% diarization error rate on NIST SRE 2000 CALLHOME.
Crowdlab uses classifiers to estimate consensus labels and annotator quality.
problem Leveraging multiple annotators for data classification.
method Weighted ensemble approach using any trained classifier.
result Superior estimates for consensus labels and annotator quality.
Unsupervised segmentation learns features without labels, improving accuracy.
problem Discover and localize semantically meaningful categories in images without annotations.
method Separates feature learning from cluster compactification; distills unsupervised features into discrete semantic labels using a contrastive loss function.
result Significant improvement over prior state of the art on semantic segmentation challenges.
Method learns true labels from noisy annotators using regularization.
problem Learning from noisy labels in supervised learning.
method Regularized estimation of annotator confusion matrices.
result Method outperforms state-of-the-art methods in image classification.