Research
On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,742 papers · 148 categories

Trend · papers per month

1122 · Oct 202419922001200920172026
10 results for LLM-as-judge

RACER optimizes LLM-as-judge accuracy with dynamic reasoning selection.

problem Balancing reasoning accuracy with computational cost in LLM-as-judge settings.
method Formulates routing as a constrained distributionally robust optimization problem, accounting for distribution shift via KL-divergence uncertainty set.
result RACER achieves superior accuracy-cost trade-offs under distribution shift.

R-AutoEval+ improves model evaluation efficiency and reliability using adaptive synthetic data.

problem Accurate model selection from AI candidates using real-world data is costly and impractical at scale.
method R-AutoEval+ uses adaptive prediction-powered inference to correct bias in autoevaluators while maintaining or improving sample efficiency.
result R-AutoEval+ provides finite-sample reliability guarantees and enhanced sample efficiency compared to conventional methods.

Large language models correlate in errors, even with different architectures and providers.

problem Lack of empirical evidence on whether different large language models differ meaningfully.
method Empirical evaluation of over 350 large language models using two leaderboards and a resume-screening task.
result Large language models have substantial correlation in errors, even with distinct architectures and providers.

Language model benchmarks often misrepresent true understanding, revealing vulnerabilities in evaluation methods.

problem Language model benchmarks fail to accurately reflect true language understanding and adaptability.
method Systematic analysis of NLP evaluation frameworks, identifying vulnerabilities in static benchmarks, human evaluation protocols, and LLM-as-judge frameworks.
result Current evaluation methods are unreliable and need improvement to accurately assess LLM performance.

UCFE benchmarks LLMs in financial tasks with human feedback.

problem Evaluating LLMs' financial task performance and user satisfaction.
method Hybrid approach combining human expert evaluations and dynamic interactions.
result Significant alignment between benchmark scores and human preferences (Pearson correlation coefficient of 0.78).

FinReflectKG - EvalBench benchmarks financial KG extraction from SEC 10-K filings.

problem Lack of universal benchmark and evaluation framework for financial KG construction.
method Agentic and holistic evaluation principles, deterministic commit-then-justify judging protocol, binary and ordinal evaluations.
result Reflection-based extraction outperforms single-pass extraction in comprehensiveness, precision, and relevance.

New models can't beat existing ones, so debiasing methods only slightly reduce needed labels.

problem Limiting scalability in model evaluation due to self-preferencing biases.
method Study of debiasing methods using a few high-quality labels to reduce model judgments.
result Debiasing methods can't decrease required ground truth labels by more than half when the judge is no more accurate than the model.