This paper considers the challenge of evaluating a set of classifiers, as done in shared task evaluations like the KDD Cup or NIST TREC, without expert labels. While expert labels provide the traditional cornerstone for evaluating statistical learners, limited or expensive access to experts represents a practical bottl…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “expert labeling”
XLabel tool reduces medical experts' workload by 40% and explains its decisions.
problem Efficiently labeling large electronic health records.
method Visual-interactive tool using Explainable Boosting Machine (EBM) for classification and explanation.
result EBM achieves high accuracy and explainability, even with mislabeled data.
New algorithms reduce label collection for online prediction with expert advice.
problem Efficiently predicting binary sequences with expert advice using fewer labels.
method Adaptive selective sampling for exponentially weighted forecasters.
result Label complexity scales roughly as the square root of the number of rounds for a scenario with a strictly better expert.
Active WeaSuL uses active learning to improve weak supervision for better model performance.
problem Limited labelled data in machine learning.
method Combines active learning with weak supervision to improve probabilistic labels.
result Active WeaSuL outperforms weak supervision and active learning with limited labelled data.
System uses conformal prediction to help experts make accurate decisions without understanding when to trust it.
problem Helping experts make accurate decisions in multiclass classification tasks.
method Develops an automated decision support system using conformal prediction to provide precise prediction sets and an efficient search method.
result System improves expert predictions by providing precise prediction sets and forcing experts to predict from these sets.
Logistic regression can handle noisy labels effectively when labels are imperfectly assigned by multiple experts.
problem Label noise in supervised classification due to manual labelling by multiple experts.
method Using approximate posterior probabilities of class membership from multiple experts to train logistic regression models.
result Logistic regression can be robust to label noise when classification difficulty is the only source of errors.
Adding uninformative labels improves tumor segmentation in low-data mammography.
problem Improving tumor segmentation in mammography with limited data.
method Used seemingly uninformative labels from non-expert annotators to turn a multi-label task into a multi-class problem.
result Performance gains in tumor segmentation are achieved in low-data settings with additional uninformative labels.
The development of real-time affect detection models often depends upon obtaining annotated data for supervised learning by employing human experts to label the student data. One open question in annotating affective data for affect detection is whether the labelers (i.e., human experts) need to be socio-culturally sim…
Budgeted deferral framework reduces expert query costs in machine learning.
problem High cost of querying all experts in deferral algorithms.
method Introduces budgeted deferral framework for selective expert querying.
result Empirical results show substantial reduction in training costs without sacrificing accuracy.
Scarcity of labeled data is one of the most frequent problems faced in machine learning. This is particularly true in relation extraction in text mining, where large corpora of texts exists in many application domains, while labeling of text data requires an expert to invest much time to read the documents. Overall, st…
A novel framework for regression with multiple experts, addressing challenges in infinite and continuous label spaces.
problem Challenges in regression with multiple experts due to the infinite and continuous nature of the label space.
method Introduces a novel framework for regression with deferral, analyzing both single-stage and two-stage scenarios with new surrogate loss functions.
result Proves -consistency bounds for both single-stage and two-stage methods, providing stronger guarantees than Bayes consistency.
Efficient label acquisition processes are key to obtaining robust classifiers. However, data labeling is often challenging and subject to high levels of label noise. This can arise even when classification targets are well defined, if instances to be labeled are more difficult than the prototypes used to define the cla…
ExpertNet uses noisy labels to improve deep learning robustness.
problem Improving deep learning robustness against noisy labels.
method ExpertNet framework combining Amateur and Expert models, iteratively learning from noisy labels and images.
result ExpertNet achieves robust classification with as little as 20-50% training data, outperforming state-of-the-art models.
A new method combines experts' opinions to train regression models with noisy labels.
problem Training regression models with noisy labels from multiple experts.
method Estimate each labeler's expertise and combine opinions using learned weights.
result Empirically outperforms existing techniques on simulated and real data.
New method improves conformal prediction for ambiguous labels.
problem Uncertainty in expert voting labels affects CP guarantees.
method Develops Monte Carlo CP procedures approximating true posterior.
result Monte Carlo CP closes coverage gap with expert annotations.
Active learning (AL) is a learning paradigm where an active learner has to train a model (e.g., a classifier) which is in principal trained in a supervised way, but in AL it has to be done by means of a data set with initially unlabeled samples. To get labels for these samples, the active learner has to ask an oracle (…
Inherent risk scoring is an important function in anti-money laundering, used for determining the riskiness of an individual during onboarding fraudulent transactions occur. It is, however, often fraught with two challenges: (1) inconsistent notions of what constitutes as high or low risk by experts a…
Advice-efficient prediction with expert advice (in analogy to label-efficient prediction) is a variant of prediction with expert advice game, where on each round of the game we are allowed to ask for advice of a limited number out of experts. This setting is especially interesting when asking for advice of ever…
LEAQI learns to query an expert less often for noisy guidance.
problem Active imitation learning with noisy guidance to reduce expert query complexity.
method Developed LEAQI, a difference classifier that predicts expert disagreement with noisy heuristic.
result Significantly fewer queries to the expert with comparable or better accuracy.
Efficient Human-in-the-Loop Active Learning: A Novel Framework for Data Labeling in AI Systemsstat.ML
A new AI framework optimizes expert labeling of unlabeled data.
problem Efficiently labeling unlabeled data for AI systems.
method Human-in-the-loop active learning framework with innovative query schemes.
result Higher accuracy and lower loss compared to other methods.
VAEs struggle with surjective multimodal data, especially class labels describing images.
problem VAEs struggle to capture variability in surjective multimodal data.
method Theoretical and empirical demonstration of VAEs with a mixture of experts posterior.
result VAEs with a mixture of experts posterior can disregard variation in surjective multimodal data.
Enhanced nearest neighbor improves accuracy in crowdsourced data.
problem Noise in crowdsourced labels degrades classification accuracy.
method Developed two algorithms to estimate worker quality and an enhanced nearest neighbor classifier.
result Proposed methods achieve the same regret as oracle version based on expert data.
New decision support systems use prediction sets to help experts update their predictions, improving performance.
problem Challenges in explaining and updating predictions from decision support systems.
method Developed a methodology leveraging nested structure of prediction sets and counterfactual monotonicity to improve performance.
result Limiting experts' agency leads to better performance in decision support systems based on prediction sets.
The paper identifies and analyzes subjective class issues in user-generated data.
problem Subjective labels in user-generated data can lead to biased and manipulated results.
method Defined subjective and objective classes, proposed a framework for detecting subjective labels.
result Data mining practitioners can detect and avoid subjective class issues early in their projects.
We explore the problem of learning under selective labels in the context of algorithm-assisted decision making. Selective labels is a pervasive selection bias problem that arises when historical decision making blinds us to the true outcome for certain instances. Examples of this are common in many applications, rangin…
AutoElicit uses LLMs to quickly create expert priors for predictive models.
problem Creating accurate priors for predictive models is time-consuming and costly.
method AutoElicit extracts knowledge from LLMs to construct priors for predictive models.
result AutoElicit yields priors that reduce error and save labelling effort.
New method reduces labeler costs by aggregating predictions from local classifiers.
problem Reduce labeler costs in multiclass classification.
method Model -class classification using smaller classifiers trained on subsets of tasks.
result Near-optimal scheme for designing classifier configurations reduces labeler costs.
We study how to effectively leverage expert feedback to learn sequential decision-making policies. We focus on problems with sparse rewards and long time horizons, which typically pose significant challenges in reinforcement learning. We propose an algorithmic framework, called hierarchical guidance, that leverages the…
CNNs identify stock market trend endpoints based on expert opinion.
problem Finding optimal entry and exit points for stock market trends.
method Three CNN submodels sequentially identify changepoints, locate them, and classify trends as upward, downward, or flat.
result CNNs can identify long-term trends based on expert opinion, offering a new approach to stock market analysis.
ARGUE combines expert networks for anomaly detection.
problem Anomaly detection without labeled data.
method Gated mixture-of-experts architecture combining expert networks.
result Prior knowledge about normal data distribution is valuable.
Study reduces human labeling in LLM-based classification systems.
problem Minimizing human intervention in training LLM-based classification systems.
method Active learning framework with Conservative Hull-based Classifier (CHC), Center-based Classifier (CC), and Generalized Hull-based Classifier (GHC).
result CHC achieves regret and is minimax optimal for . GHC bridges the gap between different regimes.
Proposes a Varying-Coefficient MoE model for analyzing dynamic data.
problem Inadequate constant coefficients in MoE models for dynamic settings.
method Varying-Coefficient Mixture of Experts (VCMoE) model with varying coefficients in gating and expert models.
result Established identifiability and consistency of the VCMoE model.
Online L2D algorithm for multiclass classification with varying experts.
problem Handling streaming data, changing expert availability, and shifting expert distribution.
method First online L2D algorithm with and regret guarantees.
result Effective extension of standard L2D to settings with varying expert availability and reliability.
Quantitative CT predicts ILD patterns and prognosis.
problem Diagnosing and predicting prognosis of fibrosing ILD patterns.
method High-resolution CT texture features, TM model for classification and survival analysis.
result TM model outperforms histogram-based model in distinguishing UIP from non-UIP patterns and allows for survival group partitioning.
Deep learning automates biofouling detection in ship hull images.
problem Automating biofouling assessment from ship hull images.
method Deep learning model trained on expert-annotated images.
result Deep learning model agrees with expert annotations.
Stress research is a rapidly emerging area in thefield of electroencephalography (EEG) based signal processing.The use of EEG as an objective measure for cost effective andpersonalized stress management becomes important in particularsituations such as the non-availability of mental health facilities.In this study, lon…
MLDemon monitors ML systems post-deployment, improving reliability with real-time performance estimates and expert labels.
problem Ensuring reliability of machine learning systems post-deployment, especially when user inputs differ from training data.
method Integrates unlabeled and on-demand labeled data to monitor ML model performance in real-time, deciding when to acquire expert labels.
result Outperforms existing approaches in temporal datasets with diverse distribution drifts, providing theoretical optimality for distribution drifts.
Transformer models show robustness across domains with domain adversarial training.
problem Domain adaptation from multiple sources with no labeled data.
method Domain adversarial training and mixture of experts.
result Domain adversarial training improves representation but not performance.
Framework detects and ranks suspicious market manipulation using temporal convolutions and expert assessment.
problem Detecting and deterring rogue agents in financial markets.
method Weakly supervised learning, expert assessment, similarity search.
result Promising preliminary results in detecting and ranking suspicious market manipulation.
Improved transfer learning with expert models, reducing compute and speeding up performance.
problem Improving sample efficiency and reducing computational requirements for new tasks.
method Training a diverse set of experts using existing label structures and performance proxies to select the relevant expert for each target task.
result Significant speed-up of 2-3 orders of magnitude compared to competing approaches.
Disease phenotyping algorithms process observational clinical data to identify patients with specific diseases. Supervised phenotyping methods require significant quantities of expert-labeled data, while unsupervised methods may learn non-disease phenotypes. To address these limitations, we propose the Semi-Supervised …
Study on learning to defer to multiple experts with consistent surrogates and confidence calibration.
problem Addressing the open problems of consistent surrogates, confidence calibration, and ensembling of experts.
method Derive two consistent surrogates (softmax and OvA) and propose a conformal inference technique for choosing experts.
result The OvA-based loss does not cause mis-calibration propagation, while the softmax-based loss does.
Improves accuracy and fairness in prediction systems with multiple domain experts.
problem Designing unbiased and accurate deferral systems with multiple experts.
method Proposes a framework for learning a classifier and deferral system that chooses to defer to multiple human experts.
result Significantly improves accuracy and fairness of final predictions compared to baselines.
WEST uses EHRs and expert cases to improve rare disease phenotyping.
problem Limited labeled data for rare diseases.
method Weakly supervised transformer model trained on probabilistic silver-standard labels.
result WEST outperforms existing methods in phenotype classification and subphenotyping.
Consider a regression problem where there is no labeled data and the only observations are the predictions of experts over many samples . With no knowledge on the accuracy of the experts, is it still possible to accurately estimate the unknown responses ? Can one still detect the leas…
CheXpert++: Approximating the CheXpert labeler for Speed,Differentiability, and Probabilistic Outputcs.LG
CheXpert++ improves CheXpert's accuracy and usability for medical radiology reports.
problem Infeasibility of obtaining ground truth labels for medical data.
method BERT-based approximation of CheXpert, addressing speed, differentiability, and probabilistic output.
result Achieves 99.81% parity with CheXpert, significantly faster, differentiable, and probabilistic.
We present a novel approach for supervised domain adaptation that is based upon the probabilistic framework of Gaussian processes (GPs). Specifically, we introduce domain-specific GPs as local experts for facial expression classification from face images. The adaptation of the classifier is facilitated in probabilistic…
Conventional learning with expert advice methods assumes a learner is always receiving the outcome (e.g., class labels) of every incoming training instance at the end of each trial. In real applications, acquiring the outcome from oracle can be costly or time consuming. In this paper, we address a new problem of active…