Unified query framework for active metric learning and classification.
problem Combining representation learning and task-specific goals in machine learning.
method Adaptive selection of nearest neighbor queries using information theoretic criterion.
result Actively selected nearest neighbor queries outperform recent methods in deep metric learning and classification.
Proposes an active RBI framework using Rényi information measures for more informed decision-making.
problem Optimal latent variable estimates in real-time settings with streaming noisy observations.
method Unified inference and query selection steps through Rényi entropy and α-divergence; new objective called Momentum for exploration.
result Analytically demonstrates superior performance compared to conventional methods like mutual information.
FAQ efficiently evaluates LLMs with statistical guarantees using adaptive query selection.
problem Efficiently evaluating many LLMs on a large suite of benchmarks is expensive.
method FAQ uses Bayesian factor models, adaptive sampling, and proactive active inference to select queries.
result FAQ delivers up to 5x effective sample size gains over baselines, matching CI width with fewer queries.
The problem of active diagnosis arises in several applications such as disease diagnosis, and fault diagnosis in computer networks, where the goal is to rapidly identify the binary states of a set of objects (e.g., faulty or working) by sequentially selecting, and observing, (noisy) responses to binary valued queries. …
We investigate the problem of active learning on a given tree whose nodes are assigned binary labels in an adversarial way. Inspired by recent results by Guillory and Bilmes, we characterize (up to constant factors) the optimal placement of queries so to minimize the mistakes made on the non-queried nodes. Our query se…
Unified method for deep active learning improves performance and efficiency.
problem Improving deep active learning performance and efficiency.
method Unified and principled approach using Wasserstein distance for querying and training.
result Consistently better empirical performance and time-efficient query strategy compared to baselines.
Designs an online selective sampling approach for choosing which model to use.
problem Active model selection for pre-trained classifiers in unlabeled data streams.
method Online selective sampling approach to query and label examples.
result High probability of outputting the best model with minimal label queries.
ActiveHNE improves HNE performance by using supervised information effectively.
problem Challenges in heterogeneous network embedding due to diverse node types and relationships.
method Develops ActiveHNE framework with DHNE and AQHN components, using graph convolutional neural network and multi-armed bandit mechanism.
result Demonstrates effectiveness and reduction in query cost compared to existing methods.
Proposes an offline active learning method for graphs.
problem Active learning on graphs where labeling is expensive.
method Two-stage biased sampling strategy considering informativeness and representativeness.
result Establishes a theoretical relationship between generalization error and number of nodes selected.
RadGrad selects expert queries based on agent error and risk to reduce costs.
problem Mitigating covariate shift in sequential decision prediction problems.
method Selective querying based on prediction of agent error and risk.
result Improves upon existing safety-aware algorithms and matches DAgger's performance.
CAMS selects best pre-trained model for unlabeled data points.
problem Efficiently utilizing pre-trained models and unlabeled data.
method Contextual active model selection algorithm with two components: contextual model selection and active query.
result CAMS requires less than 10% labeling effort compared to existing methods, achieving similar or better accuracy.
InfoTuple efficiently selects larger tuple queries for ranking multiple objects, improving efficiency and consistency.
problem Efficiently selecting and ranking multiple objects for similarity learning.
method Adaptive selection method using mutual information maximization.
result InfoTuple outperforms state-of-the-art methods on synthetic and human response datasets.
Study reduces memory needs for active learning with enriched queries.
problem Expensive labeling costs in active learning.
method Introduces bounded memory active learning through enriched queries, introduces lossless sample compression.
result Can learn classifiers with bounded memory and query optimality.
New framework for efficient query-based imitation learning.
problem Aligning agent policy with human expert behavior without prior knowledge.
method Adversarial reward query with successor representation.
result Significantly outperforms uncertainty-based methods in query efficiency.
This paper proposes an active metric learning method for clustering with pairwise constraints.
problem Clustering with pairwise constraints and improving clustering performance.
method Active metric learning method that queries informative instance pairs and updates the learned metric sequentially.
result The proposed method enhances clustering performance and provides a tighter error bound.
Study learns linear utility functions from comparisons, showing learnability gaps between passive and active learning.
problem Learn linear utility functions from pairwise comparison queries.
method Analyzes passive and active learning settings, considering noise-free and noisy query responses.
result Efficient learnability of linear utilities in passive learning, but not for utility parameters without strong assumptions.
This paper investigates and evaluates support vector machine active learning algorithms for use with imbalanced datasets, which commonly arise in many applications such as information extraction applications. Algorithms based on closest-to-hyperplane selection and query-by-committee selection are combined with methods …
Standard myopic active learning assumes that human annotations are always obtainable whenever new samples are selected. This, however, is unrealistic in many real-world applications where human experts are not readily available at all times. In this paper, we consider the single shot setting: all the required samples s…
This paper proposes LDM for efficient deep active learning.
problem Improving model performance by selecting informative unlabeled data.
method Least Disagree Metric (LDM) and its estimator for deep learning models.
result LDM-based active learning achieves state-of-the-art performance.
In many real-world machine learning applications, unlabeled data are abundant whereas class labels are expensive and scarce. An active learner aims to obtain a model of high accuracy with as few labeled instances as possible by effectively selecting useful examples for labeling. We propose a new selection criterion tha…
ALINE optimizes Bayesian inference and data acquisition by strategically querying informative data.
problem Strategic acquisition of informative data for Bayesian inference in challenging tasks.
method Unified framework combining amortized Bayesian inference and active data acquisition using a transformer architecture trained via reinforcement learning.
result ALINE delivers both instant and accurate inference along with efficient selection of informative points.
Marich extracts high-fidelity models from public data with minimal queries.
problem Creating an accurate replica of a target ML model using few queries.
method Sequentially selects informative queries to maximize entropy and reduce model mismatch.
result Extracted models achieve 60-95% of target model's accuracy with 1,000-8,500 queries.
Active learning selects high-quality examples for text-to-SQL systems.
problem Efficiently annotate large language models for text-to-SQL systems.
method Formalizes example selection as a constrained experimental design problem over semantic query embeddings, proposing a stratified greedy algorithm that maximizes heteroscedastic mutual information.
result Proposed method significantly reduces labeling effort while maintaining high text-to-SQL retrieval accuracy.
Interactive IL algorithm queries noisy expert for feedback, reducing sample size.
problem Imitation learning with noisy expert feedback requires many samples.
method Selective sampling algorithm with online regression oracle.
result Interactive IL algorithm reduces query complexity and improves performance.
Active learning aims to obtain a classifier of high accuracy by using fewer label requests in comparison to passive learning by selecting effective queries. Many active learning methods have been developed in the past two decades, which sample queries based on informativeness or representativeness of unlabeled data poi…
Supervised machine learning methods usually require a large set of labeled examples for model training. However, in many real applications, there are plentiful unlabeled data but limited labeled data; and the acquisition of labels is costly. Active learning (AL) reduces the labeling cost by iteratively selecting the mo…
Obtaining labels can be costly and time-consuming. Active learning allows a learning algorithm to intelligently query samples to be labeled for efficient learning. Fisher information ratio (FIR) has been used as an objective for selecting queries in active learning. However, little is known about the theory behind the …
Agent learns from an expert, adapting to constraints in concept learning.
problem Insufficient query selection in active learning for realistic human domains.
method Imitation learning to reason about both internal goals and external constraints.
result Agent outperforms other active learners under most constrained conditions.
In standard passive imitation learning, the goal is to learn a target policy by passively observing full execution trajectories of it. Unfortunately, generating such trajectories can require substantial expert effort and be impractical in some cases. In this paper, we consider active imitation learning with the goal of…
We point out an issue with Theorem 5 appearing in "Group-based active query selection for rapid diagnosis in time-critical situations". Theorem 5 bounds the expected number of queries for a greedy algorithm to identify the class of an item within a constant factor of optimal. The Theorem is based on correctness of a re…
Designers of AI agents often iterate on the reward function in a trial-and-error process until they get the desired behavior, but this only guarantees good behavior in the training environment. We propose structuring this process as a series of queries asking the user to compare between different reward functions. Thus…
This paper improves active learning by using robust divergences for committee disagreement.
problem Active learning with high measurement costs.
method Query by committee with Bregman divergence (including Kullback-Leibler divergence as a special case).
result The proposed method is more robust and performs as well as or better than conventional methods.
Graph embedding provides an efficient solution for graph analysis by converting the graph into a low-dimensional space which preserves the structure information. In contrast to the graph structure data, the i.i.d. node embedding can be processed efficiently in terms of both time and space. Current semi-supervised graph…
DMLE improves active learning by correcting MLE for sample dependencies.
problem Dependencies among samples in active learning affect model parameter estimation.
method Dependency-aware Maximum Likelihood Estimation (DMLE).
result DMLE achieves superior performance across multiple benchmark datasets.
Mindful active learning improves activity recognition using wearable sensors.
problem Activity recognition using wearable sensors with human cognitive and physical limitations.
method Introduces mindful active learning, a framework that considers human memory and query budget.
result EMMA framework achieves higher accuracy than traditional methods, especially with limited query budgets and weak human memory.
Proposes a new OAL algorithm for imbalanced data with limited labels.
problem Handling imbalanced unlabeled datastream with limited query budget.
method Integrates asymmetric losses and queries strategies, uses second-order optimization, and applies sketching technique.
result Demonstrates improved performance and efficiency in class imbalance.
Fair active learning selects data points to balance model accuracy and fairness.
problem Ensuring fairness in machine learning models used in high-stakes applications.
method Designing algorithms for fair active learning that select data points to balance model accuracy and fairness.
result Demonstrated the effectiveness and efficiency of fair active learning algorithms over benchmark datasets.
Fair active learning selects data points to balance model accuracy and fairness.
problem Ensuring fairness in machine learning models used in high-stakes applications.
method Designing algorithms for fair active learning that select data points to balance model accuracy and fairness, focusing on demographic parity.
result Demonstrated the effectiveness of the proposed fair active learning approach over benchmark datasets.
Segmentation is essential for medical image analysis tasks such as intervention planning, therapy guidance, diagnosis, treatment decisions. Deep learning is becoming increasingly prominent for segmentation, where the lack of annotations, however, often becomes the main limitation. Due to privacy concerns and ethical co…
We present a novel active learning algorithm for community detection on networks. Our proposed algorithm uses a Maximal Expected Model Change (MEMC) criterion for querying network nodes label assignments. MEMC detects nodes that maximally change the community assignment likelihood model following a query. Our method is…
Improved active output selection reduces calibration time by 10% or more.
problem Efficiently calibrate models with noisy data.
method Improved active output selection strategy considering noise estimate.
result At least 10% fewer measurements needed compared to existing strategies.
Survey on AL strategies for cost-effective annotation in classification.
problem Real-world AL challenges due to human annotators' limitations.
method Categorizes 60 real-world AL strategies considering multiple annotators, query types, and cost schemes.
result General real-world AL strategy introduced for categorization of 60 strategies.
We propose a new active learning strategy designed for deep neural networks. The goal is to minimize the number of data annotation queried from an oracle during training. Previous active learning strategies scalable for deep networks were mostly based on uncertain sample selection. In this work, we focus on examples ly…
In this paper, we consider active information acquisition when the prediction model is meant to be applied on a targeted subset of the population. The goal is to label a pre-specified fraction of customers in the target or test set by iteratively querying for information from the non-target or training set. The number …
Feature missing is a serious problem in many applications, which may lead to low quality of training data and further significantly degrade the learning performance. While feature acquisition usually involves special devices or complex process, it is expensive to acquire all feature values for the whole dataset. On the…
Meta-AAD uses deep reinforcement learning to improve anomaly detection by selecting the most informative instances.
problem High false-positive rate in anomaly detection, especially in high-stake applications.
method Meta-AAD leverages deep reinforcement learning to train a meta-policy for query selection, optimizing the number of discovered anomalies.
result Meta-AAD significantly outperforms state-of-the-art re-ranking strategies and unsupervised baselines on 24 benchmark datasets.
Proposes a graph-based method for active learning in regression.
problem Active learning for regression is understudied and faces challenges with noisy labels and parameter uncertainty.
method Formulates active regression as a bipartite graph optimization problem.
result Demonstrates the effectiveness of the proposed graph-based approach on benchmark data sets.
Budgeted deferral framework reduces expert query costs in machine learning.
problem High cost of querying all experts in deferral algorithms.
method Introduces budgeted deferral framework for selective expert querying.
result Empirical results show substantial reduction in training costs without sacrificing accuracy.