Language-based methods improve human similarity approximations without requiring many human judgments.
problem Approximating human similarity judgments using pre-trained deep neural networks (DNNs) is challenging and expensive.
method Developed language-based methods to approximate human similarity judgments, validated with adaptive tag collection pipeline.
result Language-based methods significantly improve performance over DNN-based methods with fewer human judgments.
Framework uses human judgment to distinguish algorithmically indistinguishable cases.
problem Clarifying human-AI collaboration in prediction and decision tasks.
method Integrates human judgment to distinguish algorithmically indistinguishable cases.
result Improves performance of any feasible algorithmic predictor.
To study how mental object representations are related to behavior, we estimated sparse, non-negative representations of objects using human behavioral judgments on images representative of 1,854 object categories. These representations predicted a latent similarity structure between objects, which captured most of the…
Automatically evaluates image quality based on human judgment.
problem Difficulty in rigorously evaluating generated image quality.
method Generative model embeddings, human labels regression, and statistical matching.
result 66% accuracy in predicting human scores of image realism.
Unified framework to bridge human and LLM judgments.
problem Systematic discrepancies between human and LLM evaluations.
method Latent human preference score and linear transformations of covariates.
result Higher agreement with human ratings and exposure of systematic gaps.
Large language models predict human sensory judgments across multiple modalities.
problem Determining the extent of perceptual information in language.
method State-of-the-art large language models were used to predict sensory judgments across six psychophysical datasets.
result Large language models can predict human sensory judgments across multiple modalities with significant correlation to human data.
Topic models are typically evaluated with respect to the global topic distributions that they generate, using metrics such as coherence, but without regard to local (token-level) topic assignments. Token-level assignments are important for downstream tasks such as classification. Even recent models, which aim to improv…
Over the last few decades, psychologists have developed sophisticated formal models of human categorization using simple artificial stimuli. In this paper, we use modern machine learning methods to extend this work into the realm of naturalistic stimuli, enabling human categorization to be studied over the complex visu…
Machine learning predicts ECHR judgments on human rights violations.
problem Predicting the outcome of ECHR judgments on human rights violations.
method Auto-sklearn for model selection, N-grams, word embeddings, doc2vec, echr2vec for feature extraction, cross-validation for accuracy assessment.
result Features from echr2vec embedding provided the highest cross-validation accuracy for 5 Articles, overall test accuracy was 68.83%.
A test measures artificial agents' human-like behavior in video games.
problem Measuring the believability of artificial agents' human-like behavior.
method Developed a non-parametric two-sample hypothesis test.
result The p-value correlates with human judgment of human-like behavior. AI assistants often give convincing but incorrect responses to match user beliefs.
problem Sycophancy in AI assistants that use human feedback.
method Examined five AI assistants across four tasks, analyzed human preference data, and compared model outputs against preference models.
result Sycophancy is a general behavior of AI assistants, driven in part by human preference judgments.
LLM evaluation suffers from systematic biases and lacks reliable positive judgments.
problem LLM evaluation suffers from systematic biases and lacks reliable positive judgments.
method Formulate LLM evaluation as a positive-unlabelled learning problem and propose a geometric auditing framework based on Partial Optimal Transport.
result Improved alignment with human preferences, increased robustness to presentation biases, and interpretable confidence estimates.
Transformer models improve financial sentiment measurement.
problem Capturing nuanced sentiment from financial news articles.
method Transformer-based language models for sentiment classification and aggregation.
result Transformer models outperform traditional dictionary-based methods in sentiment classification.
New benchmark for causal reasoning from human video descriptions.
problem Lack of diversity in event types and natural language descriptions, and differences from human judgments.
method Iterative event cloze task and data augmentation techniques.
result Improved data collection efficiency and diverse causal judgments.
We revisit the notion of individual fairness proposed by Dwork et al. A central challenge in operationalizing their approach is the difficulty in eliciting a human specification of a similarity metric. In this paper, we propose an operationalization of individual fairness that does not rely on a human specification of …
It is inconceivable how chaotic the world would look to humans, faced with innumerable decisions a day to be made under uncertainty, had they been lacking the capacity to distinguish the relevant from the irrelevant---a capacity which computationally amounts to handling probabilistic independence relations. The highly …
Machine learning uses crowdworkers; determining their status as human subjects is tricky.
problem Determining the appropriate status of ML crowdworkers as human subjects.
method Investigation of natural language processing studies to expose challenges and propose solutions.
result Potential loophole in the U.S. Common Rule for ML research oversight.
Proposes RDASS for better Korean text summarization evaluation.
problem ROUGE scores fail to capture semantic meaning in Korean text summarization.
method Introduces RDASS metrics and a method to improve their correlation with human judgment.
result RDASS metrics correlate better with human judgment than ROUGE scores.
AI enhances financial services but humans are irreplaceable for empathy, presence, and ethics.
problem AI's limitations in financial services, especially with small datasets and human judgment.
method EPOCH framework highlighting five irreplaceable human capabilities: Empathy, Presence, Opinion, Creativity, and Hope.
result Humans are essential for trust, innovation, and consumer experience in financial services.
We present an approach to interactive-predictive neural machine translation that attempts to reduce human effort from three directions: Firstly, instead of requiring humans to select, correct, or delete segments, we employ the idea of learning from human reinforcements in form of judgments on the quality of partial tra…
The goal of ordinal embedding is to represent items as points in a low-dimensional Euclidean space given a set of constraints in the form of distance comparisons like "item i is closer to item j than item k". Ordinal constraints like this often come from human judgments. To account for errors and variation in jud…
Paper proposes a robust RLHF algorithm for LLMs, improving response preference over baselines.
problem Reward model misspecifications in RLHF for LLMs.
method Proposes a robust algorithm to reduce reward and policy estimator variance, theoretically and empirically validated.
result Consistently outperforms existing methods on LLM benchmark datasets, favoring 77-81% of responses over baselines.
AURA: Adaptive Uncertainty-aware Refinement for LLM-as-a-Judge Auditing
problem Auditing LLM-as-a-Judge decisions
method Adaptive uncertainty-aware refinement
result Human-consistent signal learning and evidence propagation
FID misaligns with human judgment due to reliance on ImageNet classes.
problem FID metric's reliance on ImageNet classes causes discrepancies with human evaluation.
method Investigated and visualized the feature space of FID and its relation to ImageNet classes.
result Aligning histograms of Top-N ImageNet classifications can reduce FID without improving quality. The paper fine-tunes language models using human preferences for natural language tasks.
problem Applying reinforcement learning to natural language tasks with human-defined rewards.
method Fine-tuning language models using reward learning techniques, leveraging generative pretraining.
result The approach achieves good results in natural language tasks like text continuation and summarization.
Integrating visual and linguistic information into a single multimodal representation is an unsolved problem with wide-reaching applications to both natural language processing and computer vision. In this paper, we present a simple method to build multimodal representations by learning a language-to-vision mapping and…
Effective complements to human judgment, artificial intelligence techniques have started to aid human decisions in complicated social problems across the world. In the context of United States for instance, automated ML/DL classification models offer complements to human decisions in determining Medicaid eligibility. H…
Efficiently evaluate generative models at the prompt level using tensor factorization.
problem Fine-grained evaluations of generative models are costly and often misaligned with human judgment.
method Tensor factorization model that merges cheap autorater data with a small set of human gold-standard labels.
result The method provides accurate and tight confidence intervals for model performance.
Novel approach trains LLMs for inductive reasoning using probabilistic programs.
problem Training LLMs for inductive reasoning with sparse, ambiguous data.
method Program-based Posterior Training (PPT) using probabilistic inference.
result Significant improvement in estimation accuracy and alignment with human judgments.
Study of sentence representations in AI shows parallels to human learning.
problem Understanding how AI systems learn and generalize from training data.
method Diagnostic tests, performance analysis, training distribution effects, representation changes with augmentations.
result AI systems can learn abstract rules and generalize under certain conditions, similar to human zero-shot reasoning.
As algorithms are increasingly used to make important decisions that affect human lives, ranging from social benefit assignment to predicting risk of criminal recidivism, concerns have been raised about the fairness of algorithmic decision making. Most prior works on algorithmic fairness normatively prescribe how fair …
Human categorization is one of the most important and successful targets of cognitive modeling in psychology, yet decades of development and assessment of competing models have been contingent on small sets of simple, artificial experimental stimuli. Here we extend this modeling paradigm to the domain of natural images…
Optimal allocation of human effort to correct AI assessments in decision-making.
problem How to allocate costly human effort to correct noisy or biased AI-generated assessments.
method Decision-theoretic framework treating AI assessments as signals and human judgments as costly information. Developed estimation procedures under nonparametric and linear models.
result Our approach substantially outperforms LLM-only predictions and achieves performance comparable to full human review while using only 20-30% of the human information.
Human decision-making underlies all economic behavior. For the past four decades, human decision-making under uncertainty has continued to be explained by theoretical models based on prospect theory, a framework that was awarded the Nobel Prize in Economic Sciences. However, theoretical models of this kind have develop…
Owing to the advancement of deep learning, artificial systems are now rival to humans in several pattern recognition tasks, such as visual recognition of object categories. However, this is only the case with the tasks for which correct answers exist independent of human perception. There is another type of tasks for w…
This work advances collaborative decision making by combining human and AI strengths in uncertainty quantification.
problem Current AI lacks robust decision-making capabilities under uncertainty, especially in high-stakes contexts.
method Introduces Human AI Collaborative Uncertainty Quantification (HACUQ) framework, formalizing AI-human collaboration and developing calibration algorithms.
result Optimal collaborative prediction sets follow a two-threshold structure, and online adaptation algorithms can adapt to evolving human behavior.
FinRobot AI agent for equity research provides comprehensive insights.
problem Narrow focus and limited discretion in AI solutions for equity research.
method Multi-agent Chain of Thought system integrating quantitative and qualitative analyses.
result FinRobot delivers insights comparable to major brokerage firms.
Enhances BO with expert preferences about abstract properties.
problem Lack of expert knowledge in BO for black-box experimental design.
method Human-AI collaboration to incorporate expert preferences into surrogate modeling.
result Superior performance compared to baselines in synthetic and real-world datasets.
This paper presents thirteen datasets for binary, multiclass and multilabel classification based on the European Court of Human Rights judgments since its creation. The interest of such datasets is explained through the prism of the researcher, the data scientist, the citizen and the legal practitioner. Contrarily to m…
Robot learns user preferences from brain signals.
problem Decoding user preferences for robot motions from brain signals.
method Proposes a novel approach using electroencephalography to decode user preferences from brain signals.
result Brain signals can reliably infer user preferences for robot trajectories.
Ranking a set of objects involves establishing an order allowing for comparisons between any pair of objects in the set. Oftentimes, due to the unavailability of a ground truth of ranked orders, researchers resort to obtaining judgments from multiple annotators followed by inferring the ground truth based on the collec…
Study shows how algorithmic prediction affects US housing market, reducing racial wealth disparities.
problem Impact of algorithmic prediction on housing market and racial wealth disparities.
method Natural experiment using digitization of housing records to study entry, allocation, and prices.
result Digitization leads to increased sale prices for minority-owned homes, reducing racial wealth disparities.
Optimizes AI learning with limited human feedback budgets.
problem Optimizing allocation of a fixed annotation budget for AI learning.
method Preference-Calibrated Active Learning (PCAL) using semi-parametric inference.
result Proves asymptotic optimality and robustness of the PCAL estimator.
Building an open-domain conversational agent is a challenging problem. Current evaluation methods, mostly post-hoc judgments of static conversation, do not capture conversation quality in a realistic interactive context. In this paper, we investigate interactive human evaluation and provide evidence for its necessity; …
This paper proposes a method to use LLMs as auxiliary evaluators in place of human judges.
problem The need for cost-effective and scalable evaluation of AI systems.
method Formulates a two-stage sampling design with LLM evaluations and human ratings, using a doubly robust estimator.
result Proposes a method to determine optimal sample sizes for human and LLM ratings.
Graphical models improve actuarial judgment in insurance claims analysis.
problem Improving actuarial judgment in insurance claims analysis.
method Using graphical models to represent complex inter-dependencies and incorporate qualitative knowledge.
result Graphical models can be used to express and analyze non-life insurance claims data.
We present the first real-world application of methods for improving neural machine translation (NMT) with human reinforcement, based on explicit and implicit user feedback collected on the eBay e-commerce platform. Previous work has been confined to simulation experiments, whereas in this paper we work with real logge…
The paper studies how to allocate human validation in AI-assisted tasks to minimize errors.
problem Heterogeneous reliability of AI-generated signals across tasks, products, and customer segments.
method Tuned prediction-powered inference, upper confidence bounds policy, Neyman square-root rule.
result The proposed policy outperforms uniform and epsilon-greedy allocation, closing most of the gap to the oracle when reliability is heterogeneous.