Tiny benchmarks reduce LLM evaluation costs by using fewer examples.
problem Expensive evaluation of LLMs with tens of thousands of examples.
method Developed evaluation tools and tiny versions of popular benchmarks.
result Accurately estimate LLM performance with just 100 curated examples.
Study constructs a Japanese financial LLM benchmark.
problem Need for domain-specific benchmarks for LLMs.
method Constructed a benchmark with multiple Japanese and financial domain tasks.
result GPT-4 outperforms other models in the benchmark.
New benchmarks show LLMs struggle with causal discovery.
problem Leveraging LLMs for causal discovery is unreliable due to dataset leakage.
method Developing science-grounded benchmarks and hybrid methods combining LLM predictions with statistical analysis.
result LLMs perform poorly on novel, real-world scientific studies compared to classical methods.
InvestorBench benchmarks LLM agents in financial tasks.
problem Lack of a comprehensive benchmark for LLM-based financial agents.
method Developed a benchmark with diverse financial tasks and datasets.
result Evaluated LLM agents' performance across various financial products and market environments.
PredictaBoard benchmarks LLM score predictors to assess their ability to anticipate errors.
problem Inconsistent performance of LLMs in common sense reasoning tasks.
method Collaborative benchmarking framework evaluating pairs of LLMs and assessors using rejection rate at different tolerance errors.
result Highlights the need to evaluate predictability alongside performance for safer AI systems.
Sloth predicts LLM performance using latent skills across families.
problem Variations in benchmark performance due to differences in training configurations and data processing across model families.
method Sloth uses publicly available benchmark data and assumes LLM performance is driven by latent skills influenced by model size and training tokens. It exploits correlations across benchmarks to provide accurate predictions.
result Sloth predicts LLM performance accurately and offers insights into scaling behaviors for complex tasks.
Benchmark assesses LLMs' causal inference skills, revealing significant limitations.
problem Lack of rigorous evaluation of LLMs' causal inference capabilities.
method CausalPitfalls benchmark with structured challenges and grading rubrics.
result Significant limitations in current LLMs' statistical causal inference.
New benchmarks focus on LLM risk in finance, not just accuracy.
problem Standard benchmarks ignore LLM financial risks, leading to unsafe deployment.
method Three-level agenda: model, workflow, and system stress-testing.
result Hidden weaknesses in LLMs are missed by traditional benchmarks.
A new sparse benchmark metabench identifies key abilities from large benchmarks.
problem Redundancy and compression in existing benchmarks.
method Data from 5000+ LLMs to identify most informative items, distilling a sparse benchmark.
result Sparse benchmark metabench captures underlying abilities with high accuracy.
Language model benchmarks often misrepresent true understanding, revealing vulnerabilities in evaluation methods.
problem Language model benchmarks fail to accurately reflect true language understanding and adaptability.
method Systematic analysis of NLP evaluation frameworks, identifying vulnerabilities in static benchmarks, human evaluation protocols, and LLM-as-judge frameworks.
result Current evaluation methods are unreliable and need improvement to accurately assess LLM performance.
UCFE benchmarks LLMs in financial tasks with human feedback.
problem Evaluating LLMs' financial task performance and user satisfaction.
method Hybrid approach combining human expert evaluations and dynamic interactions.
result Significant alignment between benchmark scores and human preferences (Pearson correlation coefficient of 0.78).
CausalGame benchmarks LLM agents' causal thinking in games.
problem Evaluating causal thinking in AI Scientists with LLMs.
method Interactive games with 14 scenarios incorporating selection bias, measurement error, and hidden confounders.
result None of the 30 LLM agents demonstrated reliable causal thinking, with the best model achieving only 68.0% survival.
Benchmark evaluates LLM trading agents by masking identifiers to prevent memory leaks.
problem Evaluate LLM trading agents without relying on market memory or noise.
method Data-side masking protocol, Barra-style performance attribution framework.
result LLM agents' returns are largely explained by market and style exposure, not stock selection.
FinTradeBench benchmarks LLMs for financial reasoning combining company fundamentals and market signals.
problem Challenges in evaluating financial reasoning models for LLMs.
method Developed a benchmark integrating company fundamentals and trading signals, using a calibration-then-scaling framework.
result Clear performance gap between LLMs, retrieval improves reasoning over textual fundamentals but not trading signals.
FinMaster benchmarks LLMs in financial tasks, revealing gaps in reasoning.
problem Challenges in financial tasks, including labor-intensive processes and low error tolerance.
method Developed a comprehensive financial benchmark (FinMaster) with three modules: FinSim, FinSuite, and FinEval.
result LLMs struggle with complex financial reasoning, showing significant accuracy drops.
Paper proposes FinAR-Bench to evaluate LLMs in financial analysis tasks.
problem Inaccurate financial analysis by LLMs leading to investment and regulatory issues.
method Proposes FinAR-Bench, a benchmark dataset with three steps: key info extraction, financial indicator calculation, and logical reasoning.
result LLMs perform better in key info extraction and indicator calculation but struggle with logical reasoning.
EDINET-Bench evaluates LLMs on complex financial tasks using Japanese financial statements.
problem Challenges in evaluating LLMs on financial tasks due to specialized expertise and scarce benchmarks.
method Developed EDINET-Bench, an open-source Japanese financial benchmark for LLMs on tasks like fraud detection and earnings forecasting.
result State-of-the-art LLMs perform only marginally better than logistic regression in financial tasks, highlighting the need for more realistic benchmarks.
This paper evaluates LLMs on large graph property estimation tasks.
problem Limited context length of LLMs limits their evaluation on large graphs.
method Developed EstGraph dataset and introduced four tasks for LLMs to estimate large graph properties.
result LLMs perform better on graph property estimation tasks when provided with context-rich prompts based on random walks.
Paper examines LLM capability benchmarks through construct validity, favoring nomological account.
problem Linking theoretical capabilities to empirical measurements in LLMs.
method Contrasts three frameworks: nomological, inferential, and causal.
result Nomological account provides best foundation for LLM research.
Benchmark assesses forecasting models' ability to use textual context.
problem Forecasting models struggle with integrating textual context.
method Introduces a benchmark with numerical and textual data, evaluates various models.
result LLM prompting method outperforms other models.
Look-Ahead-Bench evaluates financial LLMs for lookahead bias, revealing significant differences in model performance.
problem Measuring and mitigating lookahead bias in financial LLMs.
method Standardized benchmark evaluating model behavior in practical financial scenarios, analyzing performance decay across market regimes.
result Standard LLMs exhibit significant lookahead bias, while Pitinf models show improved generalization and reasoning abilities.
Bayesian approach quantifies uncertainty in LLM evaluations.
problem Statistical uncertainty in evaluating LLM behavior.
method Bayesian evaluation of LLM behavior using probabilistic text generation strategies.
result Bayesian approach provides useful uncertainty quantification about LLM behavior.
New framework selects high-quality pretraining data without training LLMs.
problem Slow progress in understanding pretraining data due to costly experiments.
method Statistical framework based on perplexity-benchmark correlations.
result Approach outperforms existing methods on multiple benchmarks.
PriceSeer benchmarks LLMs in real-time stock prediction.
problem Evaluating LLMs' stock prediction accuracy and robustness.
method Real-time benchmark with 110 U.S. stocks, internal and external information expansion.
result LLMs perform suboptimally in long-term predictions due to fake news and specific industries.
Study benchmarks LLMs in portfolio optimization tasks.
problem Evaluate financial decision-making of LLMs.
method Mathematically explicit portfolio optimization problems with multiple-choice questions.
result Distinct performance patterns among LLMs in different financial tasks.
Survey of financial LLMs in finance.
problem Limited research on Financial LLMs (FinLLMs).
method Chronological overview of PLMs, comparison of techniques, performance evaluations, and advanced tasks.
result Compilation of accessible datasets and benchmarks for AI research in finance.
Novel method diagnoses large language models' reasoning abilities.
problem Fine-grained evaluation of large language models' reasoning abilities.
method Adapting cognitive diagnosis models to LLMs, estimating mastery profiles and Q-matrix, incorporating textual information.
result Accurate parameter recovery and insights into LLMs' capabilities.
SPIN converts weak LLMs to strong ones using self-play.
problem Growing strong LLMs without human-annotated data.
method Self-play fine-tuning (SPIN) starting from a supervised fine-tuned model.
result SPIN significantly improves LLM performance across benchmarks.
AlphaForgeBench evaluates LLMs as quantitative researchers, not trading agents, to address instability in financial decision-making.
problem Behavioral instability of LLMs in sequential decision-making under financial uncertainty.
method Proposes AlphaForgeBench, a framework that requires LLMs to generate executable alpha factors and compose factor-based trading strategies.
result Eliminates execution-induced instability and provides a rigorous benchmark for evaluating financial reasoning.
LLM forecasting benchmarks suffer from information leakage, which confounds model performance.
problem LLM forecasting benchmarks suffer from information leakage.
method A retrieval-augmented LLM forecaster observes only decision-time information.
result The full pipeline obtains a median monthly Spearman rank IC of +0.154.
Study builds a Japanese financial-specific LLM through continual pre-training.
problem Lack of domain-specific Japanese financial LLMs.
method Continual pre-training on Japanese financial-focused datasets using a base Japanese LLM.
result Tuned model outperforms original model on Japanese financial benchmarks.
Benchmark tests LLMs on discovering physics laws in unconventional worlds.
problem Difficulties in distinguishing genuine reasoning from recall in LLMs across physics evaluations.
method Interactive benchmark with 22 worlds governed by various unconventional physics laws, requiring agents to design experiments and revise hypotheses.
result Strongest agents fail on worlds requiring latent structure discovery, highlighting limitations in long-term reasoning.
PromptEval estimates LLM performance across many prompts, improving reproducibility.
problem Limited prompt templates affect LLM benchmark reproducibility.
method Estimates performance distribution across many prompts using borrowed strength.
result PromptEval accurately estimates performance quantiles with practical budget.
Traditional methods outperform LLMs in forecasting corporate credit ratings.
problem Forecasting corporate credit ratings using LLMs.
method Comparison of traditional methods (XGBoost) and LLMs (LLaMA) on credit rating forecasting.
result XGBoost outperforms LLMs in forecasting corporate credit ratings.
Research benchmarks LLMs in medical domain to reduce hallucinations.
problem Hallucinations in medical LLMs can lead to incorrect information.
method Developed Med-HALT dataset and testing methods.
result Significant performance differences among LLMs identified.
Framework uses LLMs to automate strategy finding in quantitative finance.
problem Brittleness of traditional deep learning models in financial applications.
method Three-stage framework with prompt-engineered LLMs, multimodal agent-based evaluation, and dynamic weight optimization.
result Robust performance in Chinese & US markets, superior risk-adjusted performance.
A new framework evaluates large language models efficiently and accurately.
problem Evaluation of large language models is challenging due to stochasticity and heterogeneity of benchmarks.
method Interpretable and scalable framework based on Item Response Theory (IRT) and majorization-minimization principle.
result Our method achieves superior scalability and interpretability compared to existing approaches.
THERMOMETER calibrates LLMs efficiently for diverse tasks.
problem Calibrating large language models is challenging due to computational and versatility issues.
method THERMOMETER learns an auxiliary model for calibrating a LLM using data from multiple tasks.
result THERMOMETER produces better-calibrated responses for new tasks.
Efficiently predict LLM benchmarks using feature selection and regression.
problem Predicting full benchmark scores with minimal question subsets.
method Multiple regression with feature selection, using kernel ridge regression and mRMR.
result Improved prediction accuracy and ranking correlation across various benchmarks.
FLUID-LLM uses LLMs to predict fluid dynamics with improved accuracy.
problem Leveraging LLMs for CFD due to their pattern recognition abilities but struggles with fluid dynamics complexities.
method Combines pre-trained LLMs with spatiotemporal-aware encoding to predict unsteady fluid dynamics.
result Significant performance improvements in CFD predictions across various datasets.
BloombergGPT is a large language model trained on financial data, outperforming existing models on financial tasks.
problem Lack of specialized large language models for finance.
method Trained on a 363 billion token dataset augmented with 345 billion tokens from general datasets, using a 50 billion parameter model.
result BloombergGPT outperforms existing models on financial tasks without sacrificing performance on general LLM benchmarks.
AI-Trader benchmarks LLMs in live financial markets, revealing poor trading performance.
problem Challenges in real-time financial decision-making by autonomous agents.
method Fully automated, live evaluation benchmark with minimal human intervention.
result General intelligence does not translate to effective trading, highlighting limitations.
Unified framework to bridge human and LLM judgments.
problem Systematic discrepancies between human and LLM evaluations.
method Latent human preference score and linear transformations of covariates.
result Higher agreement with human ratings and exposure of systematic gaps.
SIREN protocol corrects optimistic winner's scores in LLM evaluation.
problem Optimistic winner's scores in LLM evaluation due to adaptive benchmarking.
method SIREN protocol that freezes post-search shortlist, separates selection and evaluation, and uses bootstrap for uncertainty quantification.
result SIREN provides valid confidence intervals for procedure-performance curves and deployment conclusions.
BTZSC benchmarks zero-shot text classification across diverse models.
problem Systematically comparing zero-shot text classification across various models.
method Comprehensive benchmark of 22 datasets, comparing NLI cross-encoders, embedding models, rerankers, and instruction-tuned LLMs.
result Rerankers and instruction-tuned LLMs outperform NLI cross-encoders, with rerankers setting a new state-of-the-art.
Fine-tunes LLMs to correct bias in predictions.
problem LLMs exhibit bias in predictions from data.
method Supervised fine-tuning with Low-Rank Adaptation (LoRA).
result Fine-tuning corrects bias in both controlled and real-world settings.
Automates GPU kernel optimization for diverse applications.
problem Lack of systematic evaluation for multi-scenario GPU kernel optimization.
method Introduces MSKernelBench and CUDAMaster for automated optimization.
result Significant speedups across various operators, outperforming existing tools.
LaRT models LLMs' response accuracy and CoT length to evaluate reasoning ability and speed.
problem Valid evaluation of Large Language Models (LLMs) via response accuracy and chain-of-thought length.
method Introduces Latency-Response Theory (LaRT) to jointly model response accuracy and CoT length using latent ability and latent speed.
result LaRT yields higher estimation accuracy and shorter confidence intervals for latent traits compared to IRT.