Research
On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,695 papers · 148 categories

Trend · papers per month

316293124 · Oct 201919922001200920172026
48 results for human gold-standard

Efficiently evaluate generative models at the prompt level using tensor factorization.

problem Fine-grained evaluations of generative models are costly and often misaligned with human judgment.
method Tensor factorization model that merges cheap autorater data with a small set of human gold-standard labels.
result The method provides accurate and tight confidence intervals for model performance.

Meta-Router optimizes LLM selection using gold-standard and preference-based data.

problem Training a high-quality LLM router with combined data sources is challenging due to bias and scarcity.
method Developed an integrative causal router training framework to correct bias and improve routing accuracy.
result Our approach delivers more accurate routing and improves the trade-off between cost and quality.

Generative models have made immense progress in recent years, particularly in their ability to generate high quality images. However, that quality has been difficult to evaluate rigorously, with evaluation dominated by heuristic approaches that do not correlate well with human judgment, such as the Inception Score and …

2019-11-30abs ↗pdf ↗

Study scaling laws of reward model overoptimization in reinforcement learning.

problem Reward model overoptimization hinders true performance in reinforcement learning.
method Synthetic setup with fixed gold reward model; optimization using RL or best-of-nn sampling; analysis of scaling laws.
result Scaling laws of reward model overoptimization differ based on optimization method and scale smoothly with model parameters.

PPI++ outperforms gold-standard labels only if pseudo-labels are highly correlated.

problem Optimizing statistical estimation using noisy pseudo-labels.
method Exact finite-sample analysis of PPI++ on mean estimation problem.
result PPI++ has provably worse estimation error than gold-standard labels alone in some settings.

Polysomnography (PSG) is the gold standard for diagnosing sleep obstructive apnea (OSA). It allows monitoring of breathing events throughout the night. The detection of these events is usually done by trained sleep experts. However, this task is tedious, highly time-consuming and subject to important inter-scorer varia…

2019-06-20abs ↗pdf ↗

New method improves statistical inference using machine learning-imputed data.

problem Improving statistical inference with imputed data from machine learning.
method Two-phase sampling approach for Z-estimation with ML-imputed outcomes.
result Guaranteed efficiency matching or exceeding classical inference, regardless of prediction quality.

Semantic Web knowledge representation standards, and in particular RDF and OWL, often come endowed with a formal semantics which is considered to be of fundamental importance for the field. Reasoning, i.e., the drawing of logical inferences from knowledge expressed in such standards, is traditionally based on logical d…

2018-11-09abs ↗pdf ↗

Paper proposes a new method to measure model sensitivity using final model only.

problem Understanding model behavior using only the final trained model.
method Reframe TDA as measuring sensitivity, propose further training as gold standard, unify gradient-based methods.
result Gradient-based methods approximate further training but vary in quality.

Despite an explosion in the number of experimentally determined, atomically detailed structures of biomolecules, many critical tasks in structural biology remain data-limited. Whether performance in such tasks can be improved by using large repositories of tangentially related structural data remains an open question. …

2018-07-03abs ↗pdf ↗

Regardless of the gold-standard being considered as outdated, it provides valuable signs concerning the development of novel monetary standards, better adjusted to the current macroeconomic environment. By using a point of view of classical physics, the intent of this work is doing a review of the concept of monetary s…

2017-11-30abs ↗pdf ↗

We present an algorithm, AROFAC2, which detects the (CP-)rank of a degree 3 tensor and calculates its factorization into rank-one components. We provide generative conditions for the algorithm to work and demonstrate on both synthetic and real world data that AROFAC2 is a potentially outperforming alternative to the go…

2012-11-30abs ↗pdf ↗

The article explains how to estimate confusion matrices for classifiers using unlabeled data.

problem Estimating sensitivity and specificity of binary medical diagnostic tests without gold standard tests.
method Modifying diagnostic test solutions to estimate confusion matrices for classifiers on unlabeled data.
result The approach can be used to estimate accuracy statistics for supervised or unsupervised binary classifiers on unlabeled data.

New method uses imperfect LLM annotations for valid statistical inference in social science.

problem Inaccurate large language model annotations in social science research.
method Design-based supervised learning (DSL) combining imperfect LLM surrogates with gold-standard labels.
result DSL provides valid statistical inference with comparable predictive accuracy to existing methods.

AM-PPI uses multiple predictors to reduce label cost in healthcare AI.

problem Reduces label cost in post-deployment monitoring of healthcare AI.
method Combines model predictions with a small labeled sample, routing each instance to a cost-appropriate subset of predictors.
result Produces narrower confidence intervals than single-predictor methods.

We describe a method for parameter estimation in bipartite probabilistic graphical models for joint prediction of clinical conditions from the electronic medical record. The method does not rely on the availability of gold-standard labels, but rather uses noisy labels, called anchors, for learning. We provide a likelih…

2016-08-02abs ↗pdf ↗

Bandit structured prediction describes a stochastic optimization framework where learning is performed from partial feedback. This feedback is received in the form of a task loss evaluation to a predicted output structure, without having access to gold standard structures. We advance this framework by lifting linear ba…

2017-04-21abs ↗pdf ↗

Bayesian inference was once a gold standard for learning with neural networks, providing accurate full predictive distributions and well calibrated uncertainty. However, scaling Bayesian inference techniques to deep neural networks is challenging due to the high dimensionality of the parameter space. In this paper, we …

2019-07-17abs ↗pdf ↗

Differential privacy, a notion of algorithmic stability, is a gold standard for measuring the additional risk an algorithm's output poses to the privacy of a single record in the dataset. Differential privacy is defined as the distance between the output distribution of an algorithm on neighboring datasets that differ …

2019-07-03abs ↗pdf ↗

Study causal effects on humans in mixed human-AI systems with unobserved unit types.

problem Estimating causal effects on humans in systems with unobserved unit types and interaction networks.
method Assumed human-AI prior, causal message passing (CMP) framework, subpopulation analysis.
result Consistently recover human-specific causal effects using subpopulations with varying expected human composition and treatment exposure.

Pairs trading strategy fails to outperform market benchmarks, but performs well during bear markets.

problem The validity of pairs trading as a profitable strategy in modern markets.
method Used common distance and cointegration methods on US equities from 1990 to 2020, including the Covid-19 crisis.
result The pairs trading strategy does not consistently outperform market benchmarks, but performs well during bear markets.

Two new methods improve forecasting of functional time series data.

problem Forecasting of functional time-dependent data.
method Functional Singular Spectrum Analysis (FSFA) based forecasting methods.
result Our methods outperform existing algorithms for periodic stochastic processes.

Gradient matching is a promising tool for learning parameters and state dynamics of ordinary differential equations. It is a grid free inference approach, which, for fully observable systems is at times competitive with numerical integration. However, for many real-world applications, only sparse observations are avail…

2017-05-19abs ↗pdf ↗

When machine predictors can achieve higher performance than the human decision-makers they support, improving the performance of human decision-makers is often conflated with improving machine accuracy. Here we propose a framework to directly support human decision-making, in which the role of machines is to reframe pr…

2019-05-29abs ↗pdf ↗

Conformal prediction sets improve human decision making by quantifying model uncertainty.

problem Humans signal uncertainty and offer alternatives when unsure, but machine learning models often lack this feature.
method Conducted a randomized controlled trial with human subjects given conformal prediction sets.
result Human accuracy improves when given conformal prediction sets compared to fixed-size prediction sets.

This paper argues for more realistic human models in RL.

problem Current RL models oversimplify human feedback, ignoring personal, contextual, and dynamic aspects.
method Calls for interdisciplinary research on human feedback in RL.
result Realistic human models are needed for robust human-in-the-loop RL systems.

While we would like agents that can coordinate with humans, current algorithms such as self-play and population-based training create agents that can coordinate with themselves. Agents that assume their partner to be optimal or similar to them can converge to coordination protocols that fail to understand and be unders…

2019-10-13abs ↗pdf ↗

This paper presents a method to generate synthetic data with differential privacy to protect user privacy while maintaining data trends.

problem Protecting user privacy while using real data for AI applications.
method Develops a practical guide to generating synthetic data using differential privacy.
result Synthetic data can be generated to preserve trends of real data while ensuring strong privacy protections.