Bayesian approach tackles active learning with abstention feedbacks, achieving near-optimality.
problem Active learning with abstention feedbacks, especially in classification problems.
method Bayesian approach, incorporating estimated average abstention rate into greedy criteria.
result Developed two new greedy algorithms achieving near-optimality guarantees.
We study pool-based active learning with abstention feedbacks, where a labeler can abstain from labeling a queried example with some unknown abstention rate. This is an important problem with many useful applications. We take a Bayesian approach to the problem and develop two new greedy algorithms that learn both the c…
Improves reward bounds for prediction with expert advice using abstention.
problem Prediction with expert advice under bandit feedback with abstention.
method CBA algorithm exploiting abstention to improve reward bounds.
result Achieved significant improvement in reward bounds for general confidence-rated predictors.
Study selective classification with limited feedback in online learning.
problem Resource-limited and safety-critical domains where feedback is only received on abstentions.
method Versioning-based schemes for minimizing mistakes and abstentions.
result Constructed schemes that make few mistakes and minimal excess abstentions.
An assistant learns to mediate decisions between humans and experts, balancing risk and learning.
problem Learning to mediate decisions between imperfect humans and expert knowledge.
method Formalizes online decision mediation, proposes a policy to balance immediate loss and future generalization.
result Consistent gains over benchmarks in decision-making performance.
Dynamic abstention improves LLM accuracy by selectively terminating unpromising reasoning.
problem LLMs waste compute on incorrect responses, leading to inefficiency.
method Formal reinforcement learning framework with abstention reward parameter.
result Dynamic abstention outperforms natural baselines in selective accuracy.
Paper introduces new loss functions for multi-class abstention learning.
problem Learning with multi-class classification and the ability to abstain.
method Developed new families of surrogate losses for abstention.
result Proved strong consistency guarantees for new surrogate losses.
Motivated by Supervised Opinion Analysis, we propose a novel framework devoted to Structured Output Learning with Abstention (SOLA). The structure prediction model is able to abstain from predicting some labels in the structured output at a cost chosen by the user in a flexible way. For that purpose, we decompose the p…
Adaptive algorithm identifies best arm with abstention, showing phase transition from polynomial to exponential error probability.
problem Bayesian best-arm identification with abstention to reduce undetected error.
method Adaptive algorithm PGWS that optimally uses abstention budget.
result Introducing any positive abstention budget induces an exponential decay in undetected error probability.
A framework for ranking with abstention, offering theoretical guarantees and practical effectiveness.
problem Making predictions with limited cost when uncertain.
method Introduces a novel ranking framework with abstention, analyzing theoretical consistency bounds.
result Extensive theoretical analysis including H-consistency bounds for linear and neural network models. This work builds a fair classification algorithm that abstains from making predictions.
problem Building a fair classification algorithm that incorporates human decision-making and avoids disparities.
method Formalizes the problem of risk minimization under fairness and abstention constraints, derives the optimal classifier, and proposes a post-processing algorithm using unlabeled data.
result The proposed algorithm achieves fairness and abstention guarantees independently of the initial classifier, provided sufficient unlabeled data is available.
Online prediction improves with abstention, reducing regret to a fixed bound.
problem Sequential prediction with expert advice.
method Introducing abstention with a cost smaller than 0.5, achieving independent regret bounds.
result Achieves expected regret bounds independent of time horizon, with bounds of order log(N)/(1-2c).
New algorithms for multi-class classification with abstention.
problem Learning with abstention in multi-class classification.
method Introducing new surrogate losses and analyzing predictor-rejector framework.
result Strong consistency guarantees for abstention loss function.
Efficient active learning with abstention reduces label complexity exponentially.
problem Achieving high accuracy with minimal labels.
method Developed a computationally efficient active learning algorithm with abstention.
result Achieves polylog(1/ε) label complexity, reducing by an exponential factor.
MOBA model optimizes error rates for two-class cost-sensitive problems.
problem Cost-sensitive classification with ambiguous labels and varying costs.
method Multi-objective bounded-abstention model using genetic algorithm.
result MOBA achieves lower expected costs with known costs and better trade-offs without.
New algorithm tackles multi-armed bandits with abstention, achieving optimal regret bounds.
problem Optimizing decisions with abstention in multi-armed bandits.
method Developed algorithms meeting information-theoretic lower bounds on regret.
result Achieved asymptotically and minimax optimal regret bounds.
New algorithms for binary classification with abstention, achieving near-optimal performance.
problem Binary classification with abstention in various sampling models.
method Proposed active learning algorithms for three abstention settings, analyzed in a non-parametric framework.
result Upper-bounds and matching lower-bounds on the excess risk of the algorithms, demonstrating near-optimality.
Proposes a classifier with bounded abstention rate for binary classification.
problem Binary classification with abstention rate constraints.
method Characterizes Bayes optimal classifier, proposes plug-in classifier with abstention region, and develops computationally efficient algorithm.
result Proposed classifier achieves high probability of satisfying abstention constraint and is minimax near-optimal.
A deep abstaining classifier tackles label noise in deep learning.
problem Label noise in deep learning training data.
method Proposes a loss function allowing deep neural networks to abstain from making predictions on confusing samples.
result Deep abstaining classifier (DAC) improves robust learning in various types of label noise.
We study active learning where the labeler can not only return incorrect labels but also abstain from labeling. We consider different noise and abstention conditions of the labeler. We propose an algorithm which utilizes abstention responses, and analyze its statistical consistency and query complexity under fairly nat…
A new policy learning method allows policies to abstain when uncertain, improving safety and applicability.
problem Risk of making decisions without full confidence in uncertain predictions.
method Policy learning with abstention, identifying near-optimal policies and constructing an abstention rule.
result Improved safety and applicability in policy learning, with theoretical guarantees.
A conformal procedure improves CoT reasoning by aggregating reasoning paths and calibrating abstention rules.
problem Aggregation uncertainty in chain-of-thought reasoning makes correct answers less reliable.
method Introduces a conformal procedure for CoT reasoning that uses weighted score aggregation and abstention rules.
result Achieves higher selective accuracy with abstention, reducing confident-error rate.
Paper introduces a method to evaluate abstaining classifiers by considering missing predictions as counterfactuals.
problem Lack of a principled approach to evaluate and compare abstaining classifiers, especially when missing predictions are important.
method Develops a novel approach to treat abstentions as missing data, defining counterfactual scores and using causal inference methods to estimate them.
result Shows that under certain conditions, counterfactual scores can be identified and estimated efficiently, improving evaluation of abstaining classifiers.
Confidential Guardian prevents model abstention from being used to discriminate.
problem Dishonest institutions can exploit machine learning model abstention to unfairly deny services.
method Confidential Guardian uses zero-knowledge proofs to verify model confidence and detect suppression.
result Confidential Guardian effectively prevents the misuse of cautious predictions.
In safety-critical applications of machine learning, it is often important to abstain from making predictions on low confidence examples. Standard abstention methods tend to be focused on optimizing top-k accuracy, but in many applications, accuracy is not the metric of interest. Further, label shift (a shift in class …
The paper introduces a method for multi-label classification that allows partial predictions.
problem Handling multi-label classification with the option to abstain from predictions.
method Formalized MLC with abstention as a generalized loss minimization problem.
result Initial results for Hamming loss, rank loss, and F-measure.
Develops abstention procedure for nonparametric regression via variance testing.
problem Prediction with selective abstention in error-critical machine learning.
method Nonparametric heteroskedastic regression via testing hypothesis on conditional variance.
result Non-asymptotic risk bounds and convergence regimes for the estimator.
This thesis tackles learning with multi-class abstention and multi-expert deferral, improving model reliability and efficiency.
problem Improving model reliability and efficiency in large language models (LLMs) by leveraging multiple experts.
method Developed new surrogate losses and consistency guarantees for multi-class classification and regression with deferral.
result Strong consistency guarantees for surrogate losses in multi-class classification and regression with deferral.
Algorithm learns without knowing distribution, reducing error.
problem Sequential prediction with adversarial injections and abstentions.
method Boosting procedure of weak learners for general VC classes.
result Sublinear error guarantees for general VC classes.
Algorithm predicts with optimal loss by abstaining from uncertain test examples.
problem Predicting with training data not matching test data.
method Transductive abstention algorithm using labeled and unlabeled test examples.
result Optimal prediction loss guarantees with additional term for abstaining cost.
Unified approach adjusts classifiers to meet system-level constraints.
problem Multi-class classification under system-level constraints.
method Post-processing approach using linearly constrained stochastic program and entropic regularization.
result Finite-sample guarantees for risk and constraint satisfaction.
Selective classification can worsen accuracy disparities between groups.
problem Selective classification can magnify existing accuracy disparities between various groups.
method Study of margin distribution and distributionally-robust models.
result Selective classification can uniformly improve each group on distributionally-robust models.
A new model for sequential prediction handles adversarial examples by allowing abstention.
problem Sequential prediction algorithms fail with adversarial examples, leading to incorrect predictions.
method Proposes a new model that allows abstention from predictions on adversarial examples, scaling error with VC dimension.
result A learner's error scales with the VC dimension of the hypothesis class, matching the stochastic setting.
A method to control false membership rate in unsupervised mixture models.
problem Controlling misclassification in ambiguous datasets.
method Develops a plug-in procedure with theoretical guarantees on FMR.
result The false membership rate does not exceed the pre-defined nominal level α. Optimal cutoff interval for risk scores improves binary classification accuracy.
problem Improving binary classification accuracy with abstention.
method Determines optimal cutoff interval for risk scores, refraining from decisions outside this interval.
result Minimizes classification margin and maximizes accuracy within the interval.
Selective regression allows abstention to improve fairness criteria.
problem Selective regression can exacerbate disparities between subgroups.
method Proposes new fairness criteria and two approaches to mitigate performance disparity.
result Proposed fairness criteria ensures performance improvement for every subgroup with reduced coverage.
This paper presents a novel approach for detection of liver abnormalities in an automated manner using ultrasound images. For this purpose, we have implemented a machine learning model that can not only generate labels (normal and abnormal) for a given ultrasound image but it can also detect when its prediction is like…
A new approach to make classifiers safer by allowing them to abstain from making decisions on adversarial inputs.
problem Making machine learning systems robust against adversarial attacks, especially in safety-critical applications.
method Introducing a novel objective function and a simple baseline for adversarial robustness with abstention, followed by CARL (Combined Abstention Robustness Learning) for joint classifier and abstention region learning.
result Training with CARL results in a more accurate, robust, and efficient classifier than a simple baseline.
Paper analyzes robustness of non-Lipschitz networks, proving powerful adversarial attacks but offering solutions.
problem Adversarial attacks on deep networks, especially non-Lipschitz networks.
method Developed an attack model that abstracts the challenge of adversarial robustness, proving the power of such attacks and offering solutions.
result Proves powerful adversarial attacks on non-Lipschitz networks but offers solutions with abstention.
This thesis enhances ML reliability by selectively abstaining from predictions when uncertain.
problem Improving reliability in machine learning systems, especially in high-stakes domains.
method Exploiting uncertainty signals from training trajectories to develop lightweight, post-hoc abstention methods compatible with differential privacy.
result A robust trajectory-based approach to selective prediction that maintains high accuracy under privacy noise.
New insights into cascade feedback linearization of control systems.
problem Obtaining a cascade feedback linearization for invariant control systems.
method Introducing truncated versions of operators from the calculus of variations to prove new theorems.
result Established new geometry and foundational theorems for future work.
New method learns from either positive or negative feedback alone.
problem Limited applicability of existing preference optimization methods in scenarios with only unpaired feedback.
method Decouples learning from positive and negative feedback, using expectation-maximization (EM) to optimize probability of positive outcomes and explicitly incorporate negative examples.
result Stable learning from negative feedback alone demonstrated.
BRACE addresses noncompliance in bandits, offering methods for recommendation and treatment policies.
problem Noncompliance in bandit problems complicates learning objectives and treatment effects.
method BRACE formalizes objective-choice, identifies direct-control regimes, and proposes a phase-doubling algorithm for IV inversion.
result BRACE delivers valid policy values and structural uncertainty, even under weak identification and homogeneity failure.
User preferences for items can be inferred from either explicit feedback, such as item ratings, or implicit feedback, such as rental histories. Research in collaborative filtering has concentrated on explicit feedback, resulting in the development of accurate and scalable models. However, since explicit feedback is oft…
This paper improves image retrieval accuracy through novel relevance feedback methods.
problem Improving image retrieval accuracy in Content-Based Image Retrieval (CBIR).
method Novel addition to feature re-weighting and classification techniques, focusing on 0-th iteration improvement.
result Significantly improved retrieval accuracy from relevance feedback.
Recommender systems recommend items more accurately by analyzing users' potential interest on different brands' items. In conjunction with users' rating similarity, the presence of users' implicit feedbacks like clicking items, viewing items specifications, watching videos etc. have been proved to be helpful for learni…
Study evaluates new models using human feedback from another model.
problem Evaluate a new model using human feedback collected for another model.
method Formalize problem, propose model-based and model-free estimators, analyze unbiasedness, and empirically evaluate.
result Proposed estimators can predict absolute values, rank, and optimize evaluated policies.
Develops a new model for RLHF accounting for partially observed states and intermediate feedback.
problem Lack of models for partially observed states and intermediate feedback in RLHF.
method PORRL model with cardinal and dueling feedback methods.
result Demonstrates improved learning and alignment with new model-based and model-free methods.