Conformal prediction sets improve human decision making by quantifying model uncertainty.
problem Humans signal uncertainty and offer alternatives when unsure, but machine learning models often lack this feature.
method Conducted a randomized controlled trial with human subjects given conformal prediction sets.
result Human accuracy improves when given conformal prediction sets compared to fixed-size prediction sets.
Humans are the final decision makers in critical tasks that involve ethical and legal concerns, ranging from recidivism prediction, to medical diagnosis, to fighting against fake news. Although machine learning models can sometimes achieve impressive performance in these tasks, these tasks are not amenable to full auto…
When might human input help (or not) when assessing risk in fairness domains? Dressel and Farid (2018) asked Mechanical Turk workers to evaluate a subset of defendants in the ProPublica COMPAS data for risk of recidivism, and concluded that COMPAS predictions were no more accurate or fair than predictions made by human…
Conformal prediction helps quantify uncertainty but its use by humans is unclear.
problem Uncertainty quantification in predictions for human decision making.
method Decision theoretic framework for evaluating predictive uncertainty.
result Conformal prediction sets and human decision making goals are in tension.
This work advances collaborative decision making by combining human and AI strengths in uncertainty quantification.
problem Current AI lacks robust decision-making capabilities under uncertainty, especially in high-stakes contexts.
method Introduces Human AI Collaborative Uncertainty Quantification (HACUQ) framework, formalizing AI-human collaboration and developing calibration algorithms.
result Optimal collaborative prediction sets follow a two-threshold structure, and online adaptation algorithms can adapt to evolving human behavior.
Recent research in psycholinguistics has provided increasing evidence that humans predict upcoming content. Prediction also affects perception and might be a key to robustness in human language processing. In this paper, we investigate the factors that affect human prediction by building a computational model that can …
Robot learns to imitate human interactions through deep learning.
problem Teaching robots to coordinate actions with human partners.
method Deep learning framework for motion embedding, prediction, and trajectory generation.
result Importance of predictive and adaptive components for successful imitation.
Study finds visual explanations do not significantly improve human accuracy or trust in model predictions.
problem Measuring the impact of visual explanations on human accuracy and trust in model predictions.
method Randomized controlled trial with image-based age prediction task, varying levels of explanation quality.
result Visual explanations do not significantly alter human accuracy or trust in the model.
Framework uses human judgment to distinguish algorithmically indistinguishable cases.
problem Clarifying human-AI collaboration in prediction and decision tasks.
method Integrates human judgment to distinguish algorithmically indistinguishable cases.
result Improves performance of any feasible algorithmic predictor.
Human-AI teaming suffers from calibration issues.
problem Human-AI teaming
method Assume calibrated models and humans
result Existing methods for combination do not preserve human's calibration.
New metrics predict human sentence comprehension across languages.
problem Predicting human sentence comprehension using computational models.
method Developed sentence-level metrics using multilingual large language models.
result Achieved high accuracy in predicting human sentence reading speeds.
Combines human and model predictions for improved accuracy.
problem Improving classification accuracy when both human and model predictions are imperfect.
method Uses confusion matrices and calibration to combine probabilistic model outputs with human class-level predictions.
result Human-model combinations consistently outperform either alone, with accuracy gains even with limited human input.
Human decision-making underlies all economic behavior. For the past four decades, human decision-making under uncertainty has continued to be explained by theoretical models based on prospect theory, a framework that was awarded the Nobel Prize in Economic Sciences. However, theoretical models of this kind have develop…
CPATTA uses conformal prediction for efficient test-time adaptation.
problem Low data selection efficiency in existing ATTA methods.
method Conformal Prediction, online weight-update algorithm, domain-shift detector, staged update scheme.
result CPATTA consistently outperforms state-of-the-art methods by 5% in accuracy.
DMGNN predicts 3D human motions using adaptive multiscale graphs.
problem Predicting 3D skeleton-based human motions accurately.
method Dynamic multiscale graph neural networks (DMGNN) with adaptive multiscale graphs and MGCU.
result DMGNN outperforms state-of-the-art methods in short and long-term predictions.
Predictive Q-learning algorithm for IoT networks with human operators.
problem Resilient and predictive actions for IoT networks with faulty components.
method Predictive and resilient Q-learning algorithm considering historical data and human operator feedback.
result Optimal scheduling policies avoiding attacked locations and faults.
Model predicts human food choices based on demographics.
problem Predicting human food choices from demographic data.
method Non-deterministic model based on NHANES dataset and behavioral studies.
result Generates synthetic data similar to original dataset.
Explainable AI improves human decision accuracy but does not enhance it significantly.
problem Improving human decision-making through explainable AI.
method Comparing human decision accuracy with and without AI predictions, including or excluding explanations.
result Providing AI predictions improves human decision accuracy, but explanations do not significantly enhance it.
We present an approach to interactive-predictive neural machine translation that attempts to reduce human effort from three directions: Firstly, instead of requiring humans to select, correct, or delete segments, we employ the idea of learning from human reinforcements in form of judgments on the quality of partial tra…
Improves prediction accuracy in document classification by measuring uncertainty.
problem Ensuring limited human resources focus on uncertain predictions in text classification.
method Proposes a neural-network-based model using dropout-entropy for uncertainty measurement and metric learning on feature representations.
result Significant improvement in overall prediction accuracy, from 0.78 to 0.92, when 30% of most uncertain predictions are handed over to human experts.
Develops a test to assess if human experts add value to predictions.
problem Detecting if human expertise adds value to predictions.
method Statistical framework and hypothesis test to assess independence of expert predictions from outcomes.
result Physicians' decisions for AGIB patients incorporate information not available to a screening tool.
New research shows machine-assisted decisions can still be unfair even when the algorithm is fair.
problem Ensuring fairness in decisions made with machine-assisted human input.
method Formal model and lab experiment to analyze how machine predictions affect human decisions.
result Excluding information about protected groups from machine predictions can increase disparities.
To study how mental object representations are related to behavior, we estimated sparse, non-negative representations of objects using human behavioral judgments on images representative of 1,854 object categories. These representations predicted a latent similarity structure between objects, which captured most of the…
Improved antibody humanness prediction using patent data.
problem Predicting the immunogenic response of antibody therapeutics.
method Multi-stage, multi-loss training process with contrastive learning and cross-entropy loss.
result The model outperforms baselines on five out of six inference tasks, improving humanness prediction.
New definition of interpretability for deep neural networks.
problem Vague definition of interpretability for deep neural networks.
method Proposed a new definition of human predictability for interpretability of DNNs.
result Our definition will help to the research of interpretability of DNNs.
Study on predictable forward processes in trading without frequent evaluations.
problem Trading performance evaluation times not matching trading times.
method Solving a linear functional equation to construct predictable forward processes.
result Predictable forward processes are inherently myopic and optimal strategies do not use future information.
Current deep learning results on video generation are limited while there are only a few first results on video prediction and no relevant significant results on video completion. This is due to the severe ill-posedness inherent in these three problems. In this paper, we focus on human action videos, and propose a gene…
Machine learning predicts plant phenotypes from soil microbiome data.
problem Predicting plant phenotypes from soil microbiome data.
method Two models (random forest and Bayesian neural network) were used to predict plant phenotypes from soil properties and microbial population density.
result Human decisions and normalization strategies significantly impact model performance.
Paper introduces a new method for generating diverse human motion predictions.
problem Stochastic human motion prediction with limited flexibility.
method Stochastically combines root variations with previous pose information in a recurrent network.
result Model generates more diverse motion sequences than existing techniques.
New framework predicts diverse, contextually plausible 3D human motions.
problem Predicting multiple plausible future 3D poses given observed poses.
method Developed a new variational framework that conditions latent variable on past observation to encourage relevant information.
result Our approach generates motions of higher quality and preserves contextual information.
Study finds resolution impacts human classification performance in MNIST data.
problem Understanding factors affecting human classification performance in machine learning.
method Empirical study of MNIST data at various resolutions.
result Derived a quantitative relationship between resolution and human classification performance.
The paper studies how to allocate human validation in AI-assisted tasks to minimize errors.
problem Heterogeneous reliability of AI-generated signals across tasks, products, and customer segments.
method Tuned prediction-powered inference, upper confidence bounds policy, Neyman square-root rule.
result The proposed policy outperforms uniform and epsilon-greedy allocation, closing most of the gap to the oracle when reliability is heterogeneous.
TraLFM models human mobility patterns from traffic trajectories.
problem Understanding human mobility patterns from traffic data.
method Latent factor modeling of sequential, personal, and temporal factors.
result TraLFM significantly outperforms state-of-the-art methods in latent factor analysis and next location prediction.
Proposes a generic prediction architecture for autonomous vehicles considering both rational and irrational driving behaviors.
problem Accurately predicting future behaviors of surrounding vehicles for safe autonomous vehicle planning.
method Combines learning-based and planning-based models to address rationalities in human behavior.
result Stable prediction performance under various unseen driving scenarios.
Conformal prediction sets can lead to unfair outcomes.
problem Disparate impact in decision-making with conformal prediction sets.
method Experiments with human participants to demonstrate disparate impact and propose equalizing set sizes across groups.
result Providing prediction sets that satisfy Equalized Coverage increases disparate impact compared to marginal coverage.
Model predicts multi-modal sequences using N-curves.
problem Capturing multi-modal data in sequential data.
method Neural network model based on Mixture Density Networks with Bézier curves.
result Smooth multi-mode predictions without Monte Carlo simulation.
At the core of interpretable machine learning is the question of whether humans are able to make accurate predictions about a model's behavior. Assumed in this question are three properties of the interpretable output: coverage, precision, and effort. Coverage refers to how often humans think they can predict the model…
Framework improves human decision-making by learning representations.
problem Improving human decision-making performance conflated with machine accuracy.
method Mind Composed with Machine framework, incorporating human decision-making model into representation learning.
result Empirically demonstrated successful application to various tasks and representational forms.
Study reveals AI's spontaneous topic changes in text prediction.
problem AI's inability to spontaneously switch topics like humans.
method Defined topic as Token Priority Graphs (TPGs) and analyzed self-attention models.
result AI can only switch topics if lower-priority tokens outnumber higher-priority ones.
This paper evaluates deferring systems using causal inference.
problem Evaluating the impact of deferring systems on model accuracy is challenging.
method The paper uses a causal inference framework to evaluate deferring systems, distinguishing between scenarios with and without access to human predictions.
result The approach allows identifying causal effects of deferring strategies on predictive accuracy.
Optimal allocation of human effort to correct AI assessments in decision-making.
problem How to allocate costly human effort to correct noisy or biased AI-generated assessments.
method Decision-theoretic framework treating AI assessments as signals and human judgments as costly information. Developed estimation procedures under nonparametric and linear models.
result Our approach substantially outperforms LLM-only predictions and achieves performance comparable to full human review while using only 20-30% of the human information.
Recent studies in the field of human vision science suggest that the human responses to the stimuli on a visual display are non-deterministic. People may attend to different locations on the same visual input at the same time. Based on this knowledge, we propose a new stochastic model of visual attention by introducing…
Gradient-based algorithm improves model performance with triage.
problem Improving model accuracy with human expert involvement.
method Formal characterization of triage, optimal triage policy as deterministic threshold rule, gradient-based algorithm.
result Gradient-based algorithm finds triage policies and models with increasing performance.
Bayesian model predicts sequences better than LSTMs by identifying underlying rules.
problem Current RNNs struggle to generalize from limited training data and identify underlying rules in sequences.
method Bayesian model that learns underlying concepts from sequences and generalizes to new data.
result Bayesian model predicts sequences better than traditional LSTMs.
Paper predicts stock prices using ML and human intelligence.
problem Uncertainty in stock price prediction.
method ML models (LSTM, ARIMA, CNN-LSTM, GRU, LSTM-GRU) augmented with Superforecasters predictions.
result Superforecasters' predictions improve stock price prediction accuracy.
Study uses AI to refine loan assessments, improving credit default predictions.
problem Improving credit default prediction accuracy using AI-refined text.
method Comparative analysis of human-written and AI-refined loan assessments using deep learning techniques.
result AI-refined texts significantly enhance credit default predictions, especially when combined with structured data.
New models predict mobility flows as well as complex machine learning but are simpler and interpretable.
problem Incomplete understanding and modeling of human mobility flows.
method Developed simple machine-learned, closed-form models of mobility.
result These models predict mobility flows more accurately than gravity or complex machine/deep learning models.
Autonomous vehicles (AVs) are on the road. To safely and efficiently interact with other road participants, AVs have to accurately predict the behavior of surrounding vehicles and plan accordingly. Such prediction should be probabilistic, to address the uncertainties in human behavior. Such prediction should also be in…