New dataset from clinicians improves sepsis prediction models.
problem Circularity in previous sepsis prediction models.
method Developed an independent dataset from clinical judgments, avoiding circularity.
result Achieved state-of-the-art AUROC scores.
Framework uses human judgment to distinguish algorithmically indistinguishable cases.
problem Clarifying human-AI collaboration in prediction and decision tasks.
method Integrates human judgment to distinguish algorithmically indistinguishable cases.
result Improves performance of any feasible algorithmic predictor.
Language-based methods improve human similarity approximations without requiring many human judgments.
problem Approximating human similarity judgments using pre-trained deep neural networks (DNNs) is challenging and expensive.
method Developed language-based methods to approximate human similarity judgments, validated with adaptive tag collection pipeline.
result Language-based methods significantly improve performance over DNN-based methods with fewer human judgments.
Large language models predict human sensory judgments across multiple modalities.
problem Determining the extent of perceptual information in language.
method State-of-the-art large language models were used to predict sensory judgments across six psychophysical datasets.
result Large language models can predict human sensory judgments across multiple modalities with significant correlation to human data.
Graphical models improve actuarial judgment in insurance claims analysis.
problem Improving actuarial judgment in insurance claims analysis.
method Using graphical models to represent complex inter-dependencies and incorporate qualitative knowledge.
result Graphical models can be used to express and analyze non-life insurance claims data.
To study how mental object representations are related to behavior, we estimated sparse, non-negative representations of objects using human behavioral judgments on images representative of 1,854 object categories. These representations predicted a latent similarity structure between objects, which captured most of the…
Automatically evaluates image quality based on human judgment.
problem Difficulty in rigorously evaluating generated image quality.
method Generative model embeddings, human labels regression, and statistical matching.
result 66% accuracy in predicting human scores of image realism.
Approach for assessing supply chain cyber risks using expert judgment and forecasting.
problem Supply chain managers face challenges in assessing cyber risks affecting business factors.
method Structured expert judgment and forecasting models to assess various attack techniques and impacts.
result Facilitates implementation of risk management activities and decision-making processes.
In this paper we propose a method for a quantitative estimation of the decision maker's knowledge in the context of the Analytic Hierarchy Process (AHP) in cases, where the judgment matrix is inconsistent. We show that the matrix of deviation from the transitivity condition corresponds to the rate matrix for transactio…
Paper improves PBO using Skew Gaussian Processes for better optimization.
problem Optimizing with preference judgments, especially in A/B tests and recommender systems.
method Uses Skew Gaussian Processes to model preference function and exact posterior inference.
result Exact SkewGP posterior leads to better optimization results than Laplace approximation.
Unified framework to bridge human and LLM judgments.
problem Systematic discrepancies between human and LLM evaluations.
method Latent human preference score and linear transformations of covariates.
result Higher agreement with human ratings and exposure of systematic gaps.
Topic models are typically evaluated with respect to the global topic distributions that they generate, using metrics such as coherence, but without regard to local (token-level) topic assignments. Token-level assignments are important for downstream tasks such as classification. Even recent models, which aim to improv…
It is inconceivable how chaotic the world would look to humans, faced with innumerable decisions a day to be made under uncertainty, had they been lacking the capacity to distinguish the relevant from the irrelevant---a capacity which computationally amounts to handling probabilistic independence relations. The highly …
Machine learning predicts ECHR judgments on human rights violations.
problem Predicting the outcome of ECHR judgments on human rights violations.
method Auto-sklearn for model selection, N-grams, word embeddings, doc2vec, echr2vec for feature extraction, cross-validation for accuracy assessment.
result Features from echr2vec embedding provided the highest cross-validation accuracy for 5 Articles, overall test accuracy was 68.83%.
Paper proposes government indemnification for AI risks to solve judgment-proof problem.
problem Uninsurable risks from AI, especially existential risks, create a judgment-proof problem.
method A government-provided, mandatory indemnification program using risk-priced fees and Bayesian Truth Serum.
result The approach better leverages private information and signals risk mitigation efforts.
The goal of ordinal embedding is to represent items as points in a low-dimensional Euclidean space given a set of constraints in the form of distance comparisons like "item i is closer to item j than item k". Ordinal constraints like this often come from human judgments. To account for errors and variation in jud…
After admission to emergency department (ED), patients with critical illnesses are transferred to intensive care unit (ICU) due to unexpected clinical deterioration occurrence. Identifying such unplanned ICU transfers is urgently needed for medical physicians to achieve two-fold goals: improving critical care quality a…
Accurate prediction of suicide risk in mental health patients remains an open problem. Existing methods including clinician judgments have acceptable sensitivity, but yield many false positives. Exploiting administrative data has a great potential, but the data has high dimensionality and redundancies in the recording …
Transformer models improve financial sentiment measurement.
problem Capturing nuanced sentiment from financial news articles.
method Transformer-based language models for sentiment classification and aggregation.
result Transformer models outperform traditional dictionary-based methods in sentiment classification.
Over the last few decades, psychologists have developed sophisticated formal models of human categorization using simple artificial stimuli. In this paper, we use modern machine learning methods to extend this work into the realm of naturalistic stimuli, enabling human categorization to be studied over the complex visu…
In this paper, we address the problem of measuring and analysing sensation, the subjective magnitude of one's experience. We do this in the context of the method of triads: the sensation of the stimulus is evaluated via relative judgments of the form: "Is stimulus S_i more similar to stimulus S_j or to stimulus S_k?". …
We revisit the notion of individual fairness proposed by Dwork et al. A central challenge in operationalizing their approach is the difficulty in eliciting a human specification of a similarity metric. In this paper, we propose an operationalization of individual fairness that does not rely on a human specification of …
LLM evaluation suffers from systematic biases and lacks reliable positive judgments.
problem LLM evaluation suffers from systematic biases and lacks reliable positive judgments.
method Formulate LLM evaluation as a positive-unlabelled learning problem and propose a geometric auditing framework based on Partial Optimal Transport.
result Improved alignment with human preferences, increased robustness to presentation biases, and interpretable confidence estimates.
E-Commerce (E-Com) search is an emerging important new application of information retrieval. Learning to Rank (LETOR) is a general effective strategy for optimizing search engines, and is thus also a key technology for E-Com search. While the use of LETOR for web search has been well studied, its use for E-Com search h…
In this paper, we present a new task that investigates how people interact with and make judgments about towers of blocks. In Experiment~1, participants in the lab solved a series of problems in which they had to re-configure three blocks from an initial to a final configuration. We recorded whether they used one hand …
Develops Austen plots for assessing bias from unobserved confounding in observational studies.
problem Bias in causal estimates due to unobserved confounding.
method Formalizes confounding strength, uses Austen plots to visualize and quantify bias.
result Allows domain experts to assess the plausibility of strong confounders.
Proposes RDASS for better Korean text summarization evaluation.
problem ROUGE scores fail to capture semantic meaning in Korean text summarization.
method Introduces RDASS metrics and a method to improve their correlation with human judgment.
result RDASS metrics correlate better with human judgment than ROUGE scores.
A test measures artificial agents' human-like behavior in video games.
problem Measuring the believability of artificial agents' human-like behavior.
method Developed a non-parametric two-sample hypothesis test.
result The p-value correlates with human judgment of human-like behavior. The paper predicts diseases using both clinical and genomics data.
problem Clinical predictions using genomics data are not common.
method Integrated clinical and genomics datasets, machine learning, Principal Component Analysis for feature selection.
result 73% accuracy in predicting 75 disease classes.
Improved model predicts ICU readmission and mortality with interpretable results.
problem Lack of clinically interpretable predictions from deep learning models on clinical notes.
method Augmented a convolutional model with an attention mechanism for clinical note prediction.
result Attention mechanism improves prediction performance while providing interpretable results.
Clinical AI models fail to transfer between sites due to site-specific practices.
problem Clinical AI models perform poorly at new sites.
method Identify and isolate site-specific clinical practices affecting data distribution.
result A potential solution to improve model transferability.
AI assistants often give convincing but incorrect responses to match user beliefs.
problem Sycophancy in AI assistants that use human feedback.
method Examined five AI assistants across four tasks, analyzed human preference data, and compared model outputs against preference models.
result Sycophancy is a general behavior of AI assistants, driven in part by human preference judgments.
System suggests clinical concepts in real-time for faster note creation.
problem Efficiently creating structured clinical notes with minimal keystrokes.
method Contextual autocompletion using shallow neural networks.
result Reduces keystrokes by 67% in real hospital environments.
We introduce SparseVM, a method that registers clinical-quality 3D MR scans both faster and more accurately than previously possible. Deformable alignment, or registration, of clinical scans is a fundamental task for many clinical neuroscience studies. However, most registration algorithms are designed for high-resolut…
Machine learning aids in clinical prediction tasks.
problem Improving accuracy in clinical predictions.
method Introduction to machine learning concepts and algorithms, followed by practical application to clinical datasets.
result Demonstrated the application of machine learning models to clinical prediction problems.
Novel metrics improve machine learning models for ICU patient care.
problem Predicting vital sign trajectories for early detection of adverse events.
method Developed novel performance metrics aligned with clinical contexts, validated on simulated and real datasets, and optimized neural networks using these metrics.
result Neural networks trained with these metrics excel in predicting clinically significant events.
AI predicts medical specialty diagnostic choices from EHR records.
problem Predicting timely medical specialty diagnostic workups for patients.
method Ensemble of feed-forward neural networks trained on EHR data.
result Significantly higher accuracy compared to traditional checklists.
New benchmark for causal reasoning from human video descriptions.
problem Lack of diversity in event types and natural language descriptions, and differences from human judgments.
method Iterative event cloze task and data augmentation techniques.
result Improved data collection efficiency and diverse causal judgments.
Ranking a set of objects involves establishing an order allowing for comparisons between any pair of objects in the set. Oftentimes, due to the unavailability of a ground truth of ranked orders, researchers resort to obtaining judgments from multiple annotators followed by inferring the ground truth based on the collec…
Machine learning predicts patient recruitment for clinical trials.
problem Improving patient recruitment prediction for clinical trials.
method Machine learning methods applied to historical clinical trial data.
result Reduced prediction error compared to current industry standards.
Clinical outcome prediction based on the Electronic Health Record (EHR) plays a crucial role in improving the quality of healthcare. Conventional deep sequential models fail to capture the rich temporal patterns encoded in the longand irregular clinical event sequences. We make the observation that clinical events at a…
With the maturation of metabolomics science and proliferation of biobanks, clinical metabolic profiling is an increasingly opportunistic frontier for advancing translational clinical research. Automated Machine Learning (AutoML) approaches provide exciting opportunity to guide feature selection in agnostic metabolic pr…
Reply to Tetlock et al. on tail risk and probability gap.
problem Expert judgment fails to account for tail risk.
method Comparison of forecasting tournaments and extreme value theory.
result Greater gap between tail expectation and probability properties.
New dataset and approach improve skin cancer detection accuracy.
problem Lack of patient clinical information in automated skin cancer detection.
method Introduced a new dataset with clinical images and patient information. Combined clinical data with dermoscopy images using deep learning models.
result Combining clinical data improves skin cancer detection accuracy by around 7%.
The Frame Problem (FP) is a puzzle in philosophy of mind and epistemology, articulated by the Stanford Encyclopedia of Philosophy as follows: "How do we account for our apparent ability to make decisions on the basis only of what is relevant to an ongoing situation without having explicitly to consider all that is not …
A clinical Meta-Dataset from TCGA for multi-task learning.
problem Clinical decision making requires considering multiple factors; current benchmarks lack consistency and variety.
method Developed a Meta-Dataset with 174 tasks from TCGA, using regression and neural networks.
result Demonstrated the feasibility of predicting multiple clinical variables from gene expression data.
Study develops a new algorithm for assessing clinical trial abstracts.
problem Limited tools for assessing qualitative statements in clinical research.
method Developed a three-class sentiment classification algorithm using BERT model.
result Algorithm achieved 91.3% classification accuracy and 0.92 macro F1-Score.
Improved negation detection in Dutch clinical texts using machine learning.
problem Extracting negation from clinical text for better model development.
method Comparison of rule-based and machine learning methods (biLSTM, RoBERTa).
result BiLSTM and RoBERTa models outperform rule-based method in F1 score, precision, and recall.