A new method for active learning works well across all label budgets.
problem Active learning methods perform poorly in both low and high label budgets.
method Uncertainty Herding: a simple, computationally fast method that optimizes uncertainty coverage.
result Uncertainty Herding nearly optimizes distribution-level coverage and performs well across various active learning tasks.
Frequently, acquiring training data has an associated cost. We consider the situation where the learner may purchase data during training, subject TO a budget. IN particular, we examine the CASE WHERE each feature label has an associated cost, AND the total cost OF ALL feature labels acquired during training must NOT e…
In crowd labeling, a large amount of unlabeled data instances are outsourced to a crowd of workers. Workers will be paid for each label they provide, but the labeling requester usually has only a limited amount of the budget. Since data instances have different levels of labeling difficulty and workers have different r…
Clarinet uses complementary labels to train classifiers with less source data.
problem Training classifiers with true-label data from source domain is costly.
method Proposes CLARINET to train classifiers with complementary-label source data and unlabeled target data.
result CLARINET significantly outperforms baselines in unsupervised domain adaptation.
Due to concerns about human error in crowdsourcing, it is standard practice to collect labels for the same data point from multiple internet workers. We here show that the resulting budget can be used more effectively with a flexible worker assignment strategy that asks fewer workers to analyze easy-to-label data and m…
As machine learning transitions increasingly towards real world applications controlling the test-time cost of algorithms becomes more and more crucial. Recent work, such as the Greedy Miser and Speedboost, incorporate test-time budget constraints into the training procedure and learn classifiers that provably stay wit…
Optimizes AI learning with limited human feedback budgets.
problem Optimizing allocation of a fixed annotation budget for AI learning.
method Preference-Calibrated Active Learning (PCAL) using semi-parametric inference.
result Proves asymptotic optimality and robustness of the PCAL estimator.
Efficient method for generating adversarial examples with limited query budget.
problem Developing black-box adversarial attacks with limited information.
method Bayesian Optimization in a structured low-dimensional subspace.
result Significantly higher attack success rate with fewer queries.
PDO optimizes LLM prompts without labels, improving performance.
problem Optimizing prompts for LLMs without access to labeled data.
method Pairwise preference feedback, dueling bandits, Thompson Sampling, mutation.
result PDO identifies stronger prompts than label-free methods.
Study shows flipping a small subset of labels can severely damage machine learning models.
problem Adversarial attacks on distributed machine learning models.
method Formalized label flipping attacks, proposed a greedy algorithm, demonstrated with logistic regression models.
result A budget of only 0.1% of labels at each training step can reduce model accuracy by 6%, and some models can perform worse than random guessing when up to 25% of labels are flipped.
Budgeted deferral framework reduces expert query costs in machine learning.
problem High cost of querying all experts in deferral algorithms.
method Introduces budgeted deferral framework for selective expert querying.
result Empirical results show substantial reduction in training costs without sacrificing accuracy.
CBDA improves active learning for semantic segmentation, especially with imbalanced classes.
problem Class imbalance degrades performance in domain adaptive active learning.
method Class Balanced Dynamic Acquisition (CBDA) selects more balanced labels for active learning.
result CBDA increases minority class performance and outperforms baselines by 0.6-2.4 mIoU.
Crowdsourcing platforms provide marketplaces where task requesters can pay to get labels on their data. Such markets have emerged recently as popular venues for collecting annotations that are crucial in training machine learning models in various applications. However, as jobs are tedious and payments are low, errors …
MLDemon monitors ML systems post-deployment, improving reliability with real-time performance estimates and expert labels.
problem Ensuring reliability of machine learning systems post-deployment, especially when user inputs differ from training data.
method Integrates unlabeled and on-demand labeled data to monitor ML model performance in real-time, deciding when to acquire expert labels.
result Outperforms existing approaches in temporal datasets with diverse distribution drifts, providing theoretical optimality for distribution drifts.
A cost-effective approach to label acquisition using active learning markets.
problem Improving model fitting and training for predictive analytics.
method Formalizing market clearing as an optimisation problem, integrating budget constraints and improvement thresholds, using two active learning strategies with distinct pricing mechanisms.
result Superior performance with fewer labels acquired compared to conventional methods.
Framework ranks sectors influenced by Indian Union Budgets.
problem Real-time analysis of budgetary impacts on sector-specific equity performance.
method Fine-tuned embeddings and language models for sector identification and performance ranking.
result 0.997 NDCG score in predicting sector ranks based on post-budget performances.
Extends risk control to adaptive data collection, anytime-valid guarantees.
problem Ensuring safety of machine learning models with critical risk measures.
method Sequential risk controlling prediction sets (RCPS) for adaptive data collection and active labeling.
result Anytime-valid guarantees for risk control in sequential data collection.
Cost-efficient feature selection for multi-label classification in medicine.
problem Feature selection in multi-label classification with cost constraints.
method Sequential feature selection maximizing conditional mutual information, followed by cost-free feature selection using shadow features.
result The method effectively reduces prediction costs in medical applications.
Adaptive sampling detects local concept drift with limited labels.
problem Detecting local concept drift in dynamic environments with scarce labels.
method Combines residual-based exploration and exploitation with EWMA monitoring.
result Superior performance in label efficiency and drift detection accuracy.
Optimizes data labeling for causal effect estimation with missing outcomes.
problem Estimating causal effects with missing outcome data and budget constraints.
method Optimizes batch sampling probability to minimize variance of causal inference estimator.
result Achieves lower mean-squared error with fewer labeled data points.
We propose Disentanglement based Active Learning (DAL), a new active learning technique based on self-supervision which leverages the concept of disentanglement. Instead of requesting labels from human oracle, our method automatically labels the majority of the datapoints, thus drastically reducing the human labeling b…
Active inference uses machine learning to prioritize data labeling for more efficient statistical inference.
problem Efficiently collecting data points for statistical inference with limited labels.
method A machine learning-assisted approach that identifies uncertain data points for labeling.
result Achieves the same level of accuracy with fewer samples, resulting in smaller confidence intervals and more powerful p-values.
New approach for feature evolution in streaming data with limited storage.
problem Rarely-provided labels in feature evolving streams.
method Incorporates manifold regularization and a buffer to adapt to different storage budgets.
result Preserves the performance of feature evolving learning across different storage budgets.
A new active learning method considers both uncertainty and diversity to minimize labeling and decision costs.
problem Classical AL approaches fail to capture data distribution in unlabeled data, leading to mislabeling of outliers.
method CBAL considers classification uncertainty and instance diversity, using a min-max approach to minimize labeling and decision costs.
result Extensive experiments show CBAL outperforms state-of-the-art AL approaches.
Online Active Learning (OAL) aims to manage unlabeled datastream by selectively querying the label of data. OAL is applicable to many real-world problems, such as anomaly detection in health-care and finance. In these problems, there are two key challenges: the query budget is often limited; the ratio between classes i…
IUPM monitors machine learning models under gradual shifts using optimal transport and active labeling.
problem Gradual distribution shifts lead to unnoticed accuracy declines in machine learning models.
method Incremental Uncertainty-aware Performance Monitoring (IUPM) using optimal transport and active labeling.
result IUPM outperforms existing baselines in gradual shift scenarios and guides label acquisition more effectively.
AM-PPI uses multiple predictors to reduce label cost in healthcare AI.
problem Reduces label cost in post-deployment monitoring of healthcare AI.
method Combines model predictions with a small labeled sample, routing each instance to a cost-appropriate subset of predictors.
result Produces narrower confidence intervals than single-predictor methods.
Machine learning (ML) is increasingly being used in high-stakes applications impacting society. Therefore, it is of critical importance that ML models do not propagate discrimination. Collecting accurate labeled data in societal applications is challenging and costly. Active learning is a promising approach to build an…
Recently, crowdsourcing has emerged as an effective paradigm for human-powered large scale problem solving in various domains. However, task requester usually has a limited amount of budget, thus it is desirable to have a policy to wisely allocate the budget to achieve better quality. In this paper, we study the princi…
Strict partial order is a mathematical structure commonly seen in relational data. One obstacle to extracting such type of relations at scale is the lack of large-scale labels for building effective data-driven solutions. We develop an active learning framework for mining such relations subject to a strict order. Our a…
ActiveLab improves classifier accuracy with fewer annotations by re-labeling.
problem Imperfect labels from multiple annotators in real-world data.
method ActiveLab automatically decides when to re-label examples for better classifier training.
result ActiveLab trains more accurate classifiers with fewer annotations.
Class imbalance is an intrinsic characteristic of multi-label data. Most of the labels in multi-label data sets are associated with a small number of training examples, much smaller compared to the size of the data set. Class imbalance poses a key challenge that plagues most multi-label learning methods. Ensemble of Cl…
Algorithm adapts to shifting domains with minimal label queries.
problem Adaptive learning in online machine learning systems with domain shifts.
method Adaptive algorithm balancing regret and label queries for hidden domains.
result Achieves lower regret compared to uniform and greedy queries.
This paper tackles data-efficient CEE with scarce labelled data, proposing a method to progressively reduce generalization risk.
problem Data scarcity in CEE tasks, especially in high-stake domains like medical treatment effect prediction.
method Develops a principled label acquisition pipeline (MACAL) for CEE tasks, focusing on reducing generalization risk progressively.
result Proposes Model Agnostic Causal Active Learning (MACAL) algorithm for batch-wise label acquisition.
Fair active learning selects data points to balance model accuracy and fairness.
problem Ensuring fairness in machine learning models used in high-stakes applications.
method Designing algorithms for fair active learning that select data points to balance model accuracy and fairness, focusing on demographic parity.
result Demonstrated the effectiveness of the proposed fair active learning approach over benchmark datasets.
OLALA automates document layout annotation by selecting ambiguous regions for labeling.
problem Efficiently annotating complex document layouts with limited resources.
method Object-Level Active Learning framework that selects ambiguous regions for labeling and uses semi-automatic correction.
result OLALA significantly boosts model performance and improves annotation efficiency.
Study proposes an active subsampling method for estimating individualized thresholds in high-dimensional data.
problem Estimating optimal individualized thresholds in high-dimensional data with limited labeled samples.
method Developed a K-step active subsampling algorithm to iteratively select and label the most informative data points.
result Revealed a phase transition phenomenon in the estimation of θ with respect to the smoothness of the conditional density. Active inference framework improves U-statistic estimation efficiency.
problem Costly acquisition of labels for U-statistics. method Active inference framework with optimal sampling rule.
result Substantial gains in estimation efficiency over baseline methods.
Study shows resampling labels improves classifier performance in noisy data.
problem Balancing sample size vs label reliability in noisy data.
method Comparing different validation strategies and analyzing MNIST database with varying noise levels.
result Classifier performance declines with high incorrect labels, highlighting the importance of resampling.
This paper tackles label-efficient evaluation in extreme class imbalance.
problem Challenges in obtaining a sufficient sample for accurate evaluation in tasks with extreme class imbalance.
method Develops a framework for online evaluation based on adaptive importance sampling.
result Establishes strong consistency and a central limit theorem for performance estimates.
Online active regression algorithms minimize label queries for efficient data regression.
problem Efficiently predict data points with minimal label queries in an online setting.
method Proposed online algorithms for active regression under ℓ_p loss, achieving (1+ε)-approximation with minimal label queries.
result Achieves (1+ε)-approximation with only ε^(-1) d log(nκ) label queries, matching offline methods in performance.
Paper tackles active learning under human label variation, proposing a new framework.
problem Active learning assumes a single ground truth, ignoring human label variation.
method Survey and propose a conceptual framework for HLV-aware active learning.
result Lay a conceptual foundation for HLV-aware active learning.
A new method selects models for ensemble learning to maximize mutual information, outperforming existing approaches.
problem Selecting models for ensemble learning to improve performance and reduce correlation issues.
method Formulate budgeted ensemble selection as maximizing mutual information, use Gaussian-copula to model correlated errors, propose a greedy mutual-information selection algorithm.
result Our method consistently outperforms strong baselines across multiple datasets.
In unsupervised domain adaptation (UDA), classifiers for the target domain (TD) are trained with clean labeled data from the source domain (SD) and unlabeled data from TD. However, in the wild, it is difficult to acquire a large amount of perfectly clean labeled data in SD given limited budget. Hence, we consider a new…
Measures policy-violating content prevalence with ML-assisted sampling and LLM labeling.
problem Accurate measurement of content violations that are often rare and costly to label.
method Design-based measurement system using ML-assisted probability sampling and LLM labeling.
result Produces unbiased prevalence estimates with confidence intervals and dashboard drilldowns.
DP-CGAN generates private synthetic data and labels.
problem Preserving privacy in synthetic data generation.
method Differentially private conditional GAN (DP-CGAN) with clipping and perturbation.
result DP-CGAN generates visually and empirically promising results on MNIST with low privacy cost.
Active WeaSuL uses active learning to improve weak supervision for better model performance.
problem Limited labelled data in machine learning.
method Combines active learning with weak supervision to improve probabilistic labels.
result Active WeaSuL outperforms weak supervision and active learning with limited labelled data.
Active feature selection uses mutual information to choose fewer labels for better feature selection.
problem Selecting features with limited labeled data.
method Uses active feature selection with mutual information criterion, optimizing label selection for better feature quality.
result Algorithm selects features with higher mutual information using fewer labels than the data set size.