We study the problem of interactively learning a binary classifier using noisy labeling and pairwise comparison oracles, where the comparison oracle answers which one in the given two instances is more likely to be positive. Learning from such oracles has multiple applications where obtaining direct labels is harder bu…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “oracle labeling”
SoQal reduces oracle label requests in active learning by up to 35%.
problem Exploiting unlabelled data in healthcare requires costly oracle labeling.
method Dynamic questioning strategy to minimize oracle label requests.
result SoQal reduces oracle label requests by up to 35%.
New oracle uses uncertainty for active classification with noisy feedback.
problem Improving query complexity in interactive binary classifier learning.
method Proposes a new pairwise comparison oracle that considers uncertainty and an adaptive labeling algorithm.
result Demonstrates improved performance and efficiency compared to existing methods.
We investigate active learning with access to two distinct oracles: Label (which is standard) and Search (which is not). The Search oracle models the situation where a human searches a database to seed or counterexample an existing solution. Search is stronger than Label while being natural to implement in many situati…
New research shows label refinement and weak training have limitations for aligning LLMs.
problem Limitations of refinement methods for aligning large language models.
method Analyzed probabilistic assumptions and alternative approaches to label refinement and weak training.
result Label refinement and weak training suffer from irreducible error, leaving a performance gap.
An active learner is given a hypothesis class, a large set of unlabeled examples and the ability to interactively query labels to an oracle of a subset of these examples; the goal of the learner is to learn a hypothesis in the class that fits the data well by making as few label queries as possible. This work addresses…
SoQal uses selective oracle questioning to improve active learning of cardiac signals.
problem Active learning of cardiac signals is challenging due to limited labelled data and high annotation costs.
method Proposes a framework combining selective oracle questioning and Bayesian active learning by consistency.
result SoQal outperforms baseline methods in active learning of cardiac signals, even with noisy oracles.
Study how noisy labels affect semi-supervised learning.
problem Effect of noisy labels on semi-supervised learning performance.
method Proposed an algorithm derived from a continuous relaxation of the Maximum A Posteriori (MAP) estimator for a Degree Corrected Stochastic Block Model (DC-SBM).
result Our approach achieves promising performance even with very noisy labeled data.
Sufficient supervised information is crucial for any machine learning models to boost performance. However, labeling data is expensive and sometimes difficult to obtain. Active learning is an approach to acquire annotations for data from a human oracle by selecting informative samples with a high probability to enhance…
Active learning selects most informative unlabeled samples for labeling.
problem Efficiently label unlabeled data in applications with scarce labeled data.
method Formulated as open-set recognition, uses VNNs to identify uncertain samples.
result Achieved state-of-the-art results on MNIST, CIFAR-10, and CIFAR-100.
Noisy labeled data is more a norm than a rarity for self-generated content that is continuously published on the web and social media. Due to privacy concerns and governmental regulations, such a data stream can only be stored and used for learning purposes in a limited duration. To overcome the noise in this on-line s…
In most classification tasks there are observations that are ambiguous and therefore difficult to correctly label. Set-valued classifiers output sets of plausible labels rather than a single label, thereby giving a more appropriate and informative treatment to the labeling of ambiguous instances. We introduce a framewo…
Paper addresses hybrid learning with constrained adversaries, achieving optimal performance.
problem Hybrid learning problem with i.i.d. features and adversarial labels.
method Structured adversarial setting, efficient algorithm with ERM oracle.
result Oracle-efficient algorithm with regret scaling with Rademacher complexity.
Efficient algorithms recover two sparse models from a mix of linear queries.
problem Recovering two sparse models from a mix of linear queries.
method Efficient algorithms for query complexity problem.
result Improved query complexity for model recovery.
Crowdsourcing is an effective tool for human-powered computation on many tasks challenging for computers. In this paper, we provide finite-sample exponential bounds on the error rate (in probability and in expectation) of hyperplane binary labeling rules under the Dawid-Skene crowdsourcing model. The bounds can be appl…
Method discovers symmetries in data with neural networks.
problem Discovering symmetries in high-dimensional data.
method Fully connected neural networks trained with a loss function for symmetry.
result Generators for symmetries in latent space.
Develops NPMC method for noisy labels, improving multiclass classification accuracy.
problem Asymmetric misclassification costs and label noise in multiclass classification.
method Empirical likelihood approach using exponential tilting density ratio model.
result Root n consistent and asymptotically normal estimators for clean labels and noise mechanism.
Representation learning systems typically rely on massive amounts of labeled data in order to be trained to high accuracy. Recently, high-dimensional parametric models like neural networks have succeeded in building rich representations using either compressive, reconstructive or supervised criteria. However, the seman…
Crowdsourcing has become an effective and popular tool for human-powered computation to label large datasets. Since the workers can be unreliable, it is common in crowdsourcing to assign multiple workers to one task, and to aggregate the labels in order to obtain results of high quality. In this paper, we provide finit…
A Bayesian approach to multilabel classification using tree-based models.
problem Challenges in multilabel classification due to complex label relationships and correlations.
method Bayesian Additive Regression Trees (BART) framework for modeling multilabel classification.
result Improved predictive performance compared to other models, including an oracle model.
We propose Disentanglement based Active Learning (DAL), a new active learning technique based on self-supervision which leverages the concept of disentanglement. Instead of requesting labels from human oracle, our method automatically labels the majority of the datapoints, thus drastically reducing the human labeling b…
New active learning method for kernel selection improves efficiency and accuracy.
problem Real-world applications where acquiring true labels is costly or time-consuming.
method Active Multiple Kernel Learning (AMKL) with adaptive kernel selection (AMKL-AKS).
result AMKL-AKS achieves optimal sublinear regret and better performance with fewer labeled data.
This paper investigates the computational complexity of sparse label propagation which has been proposed recently for processing network structured data. Sparse label propagation amounts to a convex optimization problem and might be considered as an extension of basis pursuit from sparse vectors to network structured d…
Study on tradeoffs between mistakes and ERM oracle calls in online and transductive learning.
problem Analyzing online and transductive learning with limited ERM and weak consistency oracle access.
method Proves lower bounds and upper bounds on mistakes and oracle calls, considering realizable and agnostic cases.
result Achieves optimal mistake bounds with weak consistency queries for certain concept classes.
The paper tackles fast rates in batch active learning with pool-based data.
problem Reduced adaptivity in batch active learning leads to suboptimal results.
method Proposes a stage-wise greedy algorithm that balances informativeness and diversity.
result The algorithm's excess risk matches minimax rates in standard statistical learning settings.
This paper presents privileged multi-label learning (PrML) to explore and exploit the relationship between labels in multi-label learning problems. We suggest that for each individual label, it cannot only be implicitly connected with other labels via the low-rank constraint over label predictors, but also its performa…
A generative model may generate utter nonsense when it is fit to maximize the likelihood of observed data. This happens due to "model error," i.e., when the true data generating distribution does not fit within the class of generative models being learned. To address this, we propose a model of active distribution lear…
Annotating large unlabeled datasets can be a major bottleneck for machine learning applications. We introduce a scheme for inferring labels of unlabeled data at a fraction of the cost of labeling the entire dataset. Our scheme, bounded expectation of label assignment (BELA), greedily queries an oracle (or human labeler…
Fair Active Learningcs.LG
Machine learning (ML) is increasingly being used in high-stakes applications impacting society. Therefore, it is of critical importance that ML models do not propagate discrimination. Collecting accurate labeled data in societal applications is challenging and costly. Active learning is a promising approach to build an…
Generative model improves safety in self-driving simulators and human motion generation.
problem Improving generative models for constrained domains like safety-critical applications.
method Developed Gen-neG, a denoising diffusion model that uses oracle-assisted guidance.
result Empirically validated Gen-neG for collision avoidance and safety-guarded human motion generation.
The Oracle of DLphics.LG
We present a novel technique based on deep learning and set theory which yields exceptional classification and prediction results. Having access to a sufficiently large amount of labelled training data, our methodology is capable of predicting the labels of the test data almost always even if the training data is entir…
We consider an active learning setting where the algorithm has access to a large pool of unlabeled data and a small pool of labeled data. In each iteration, the algorithm chooses few unlabeled data points and obtains their labels from an oracle. In this paper, we consider a probabilistic querying procedure to choose th…
In active learning, the user sequentially chooses values for feature and an oracle returns the corresponding label . In this paper, we consider the effect of feature noise in active learning, which could arise either because itself is being measured, or it is corrupted in transmission to the oracle, or the o…
New algorithm reduces online learning error for unknown feature distributions.
problem Oracle-efficient hybrid online learning with unknown feature and label distributions.
method Computational efficient online predictor using ERM oracle for finite-VC and fat-shattering classes.
result Oracle-efficient sublinear regret bounds for hybrid online learning with unknown feature generation.
New stability bounds for GD in overparameterised shallow nets without NTK assumptions.
problem Generalisation and excess risk bounds for shallow neural networks.
method Oracle inequalities and stability analysis of GD without kernelisation.
result Oracle type bounds reveal GD's generalisation is controlled by an interpolating network with shortest GD path.
Data de-duplication is the task of detecting multiple records that correspond to the same real-world entity in a database. In this work, we view de-duplication as a clustering problem where the goal is to put records corresponding to the same physical entity in the same cluster and putting records corresponding to diff…
In many applications the process of generating label information is expensive and time consuming. We present a new method that combines active and semi-supervised deep learning to achieve high generalization performance from a deep convolutional neural network with as few known labels as possible. In a setting where a …
Bayesian-guided method selects optimal design from large candidate pool.
problem Optimizing complex structures with high-fidelity evaluations.
method Bayesian active learning with surrogate modeling.
result Optimal design identified with minimal oracle evaluations.
Paper tackles dynamic label shift in online learning, achieving optimal performance.
problem Adapting to changing class marginals in online supervised and unsupervised learning.
method Develops novel algorithms reducing adaptation to online regression, achieving optimal dynamic regret.
result Achieves superior performance in various online label shift scenarios.
Fair Active Learningcs.LG
Fair active learning selects data points to balance model accuracy and fairness.
problem Ensuring fairness in machine learning models used in high-stakes applications.
method Designing algorithms for fair active learning that select data points to balance model accuracy and fairness, focusing on demographic parity.
result Demonstrated the effectiveness of the proposed fair active learning approach over benchmark datasets.
New method improves transductive learning predictions with multiplicative oracle inequalities.
problem Improving transductive learning predictions with known covariates.
method Median of Level-Set Aggregation (MLSA) for transductive LOO prediction.
result Proved multiplicative oracle inequality for LOO error.
We propose in this paper a general framework for deriving loss functions for structured prediction. In our framework, the user chooses a convex set including the output space and provides an oracle for projecting onto that set. Given that oracle, our framework automatically generates a corresponding convex and smooth l…
Proposes a weighted conformal approach for cluster label uncertainty.
problem Cluster label uncertainty in unlabeled data.
method Develops a conformal inference algorithm to correct label mismatch.
result Improves confidence set size in nonlinear and high-dimensional clustering.
Embed-KCPD segments text without labels, outperforming baselines.
problem Unsupervised text segmentation is challenging due to lack of boundary labels.
method Embed-KCPD represents sentences as embeddings and estimates boundaries via penalized KCPD.
result Embed-KCPD outperforms strong unsupervised baselines across benchmarks.
Classification algorithms have been widely adopted to detect anomalies for various systems, e.g., IoT, cloud and face recognition, under the common assumption that the data source is clean, i.e., features and labels are correctly set. However, data collected from the wild can be unreliable due to careless annotations o…
Counterfactual learning from observational data involves learning a classifier on an entire population based on data that is observed conditioned on a selection policy. This work considers this problem in an active setting, where the learner additionally has access to unlabeled examples and can choose to get a subset o…
Entity resolution (ER) presents unique challenges for evaluation methodology. While crowdsourcing platforms acquire ground truth, sound approaches to sampling must drive labelling efforts. In ER, extreme class imbalance between matching and non-matching records can lead to enormous labelling requirements when seeking s…
Study examines how uncertainty visualization affects analyst trust in automated classification systems.
problem The impact of uncertainty on analyst trust in automated classification systems.
method Empirical study evaluating different active learning query policies and visualizations.
result Query policy significantly influences analyst trust in automated classification systems.