UCFE benchmarks LLMs in financial tasks with human feedback.
problem Evaluating LLMs' financial task performance and user satisfaction.
method Hybrid approach combining human expert evaluations and dynamic interactions.
result Significant alignment between benchmark scores and human preferences (Pearson correlation coefficient of 0.78).
Study constructs a Japanese financial LLM benchmark.
problem Need for domain-specific benchmarks for LLMs.
method Constructed a benchmark with multiple Japanese and financial domain tasks.
result GPT-4 outperforms other models in the benchmark.
InvestorBench benchmarks LLM agents in financial tasks.
problem Lack of a comprehensive benchmark for LLM-based financial agents.
method Developed a benchmark with diverse financial tasks and datasets.
result Evaluated LLM agents' performance across various financial products and market environments.
FinMaster benchmarks LLMs in financial tasks, revealing gaps in reasoning.
problem Challenges in financial tasks, including labor-intensive processes and low error tolerance.
method Developed a comprehensive financial benchmark (FinMaster) with three modules: FinSim, FinSuite, and FinEval.
result LLMs struggle with complex financial reasoning, showing significant accuracy drops.
Paper proposes FinAR-Bench to evaluate LLMs in financial analysis tasks.
problem Inaccurate financial analysis by LLMs leading to investment and regulatory issues.
method Proposes FinAR-Bench, a benchmark dataset with three steps: key info extraction, financial indicator calculation, and logical reasoning.
result LLMs perform better in key info extraction and indicator calculation but struggle with logical reasoning.
FinTradeBench benchmarks LLMs for financial reasoning combining company fundamentals and market signals.
problem Challenges in evaluating financial reasoning models for LLMs.
method Developed a benchmark integrating company fundamentals and trading signals, using a calibration-then-scaling framework.
result Clear performance gap between LLMs, retrieval improves reasoning over textual fundamentals but not trading signals.
EDINET-Bench evaluates LLMs on complex financial tasks using Japanese financial statements.
problem Challenges in evaluating LLMs on financial tasks due to specialized expertise and scarce benchmarks.
method Developed EDINET-Bench, an open-source Japanese financial benchmark for LLMs on tasks like fraud detection and earnings forecasting.
result State-of-the-art LLMs perform only marginally better than logistic regression in financial tasks, highlighting the need for more realistic benchmarks.
Benchmark evaluates financial misinformation detection models, revealing weaknesses without external context.
problem Detecting financial misinformation without external references.
method RFC Bench at paragraph level, two tasks: reference-free detection and comparison-based diagnosis.
result Performance improves with comparative context, revealing model weaknesses in reference-free settings.
BloombergGPT is a large language model trained on financial data, outperforming existing models on financial tasks.
problem Lack of specialized large language models for finance.
method Trained on a 363 billion token dataset augmented with 345 billion tokens from general datasets, using a 50 billion parameter model.
result BloombergGPT outperforms existing models on financial tasks without sacrificing performance on general LLM benchmarks.
Benchmark detects decision-time leakage in financial backtests.
problem Detecting decision-time leakage in financial machine-learning backtests.
method Toggles one evaluation convention at a time around a clean t+1-open reference, holding other factors fixed. result Inflation is highly selective, affecting specific features and execution methods.
UniFinEval benchmarks financial models across text, images, and videos.
problem Challenges in evaluating financial multimodal models across text, images, and videos.
method Proposes UniFinEval, a unified multimodal benchmark for financial scenarios.
result Gemini-3-pro-preview achieves best performance but still lags behind experts.
FinReflectKG - EvalBench benchmarks financial KG extraction from SEC 10-K filings.
problem Lack of universal benchmark and evaluation framework for financial KG construction.
method Agentic and holistic evaluation principles, deterministic commit-then-justify judging protocol, binary and ordinal evaluations.
result Reflection-based extraction outperforms single-pass extraction in comprehensiveness, precision, and relevance.
Optimal benchmark design varies based on costs in financial manipulation.
problem Manipulation of price benchmarks in finance.
method Analyzes empirical pattern and cost structures to determine optimal benchmark design.
result The optimal benchmark depends on the relative sizes of fixed and variable costs.
Study introduces KorFinMTEB for Korean financial texts, revealing model limitations.
problem Limited evaluation benchmarks for low-resource domains, especially Korean.
method Developed KorFinMTEB, a tailored benchmark for Korean financial texts.
result Models perform better on translated benchmarks than on domain-specific ones.
FinReflectKG benchmarks financial QA by linking relevant context from a financial KG, improving model performance and efficiency.
problem Efficiently retrieving and navigating relevant financial information across diverse sources and years.
method A benchmark built on a temporally indexed financial KG, generating QA pairs via pattern-specific prompts and quality control, evaluating retrieval scenarios.
result KG-guided retrieval yields substantial gains in correctness and token utilization, improving model performance by 24%.
Survey of financial LLMs in finance.
problem Limited research on Financial LLMs (FinLLMs).
method Chronological overview of PLMs, comparison of techniques, performance evaluations, and advanced tasks.
result Compilation of accessible datasets and benchmarks for AI research in finance.
Benchmarking deep learning models for financial time series, focusing on risk-adjusted performance.
problem Optimizing risk-adjusted performance in financial time series prediction.
method Evaluation of various deep learning architectures including linear models, RNNs, transformers, state space models, and sequence representation approaches.
result Hybrid models like VSN with LSTM and xLSTM achieve the highest overall Sharpe ratio and superior downside adjusted characteristics.
New model predicts financial market abnormalities using stock index uncertainties.
problem Forecasting abnormal financial fluctuations in the market.
method Quantitative analysis of mean and volatility uncertainties, constructing early warning indicators.
result Established a new abnormal fluctuations warning model.
AI-Trader benchmarks LLMs in live financial markets, revealing poor trading performance.
problem Challenges in real-time financial decision-making by autonomous agents.
method Fully automated, live evaluation benchmark with minimal human intervention.
result General intelligence does not translate to effective trading, highlighting limitations.
GANs generate realistic financial data for research.
problem Creating realistic financial datasets for research and benchmarking.
method Proposed a novel type of GAN and methods for preprocessing and evaluation.
result GANs can replicate financial datasets with high fidelity.
Paper introduces NumLLM for better financial text understanding with numeric variables.
problem Poor performance of existing financial large language models in numeric financial text.
method Constructed financial corpus, fine-tuned with LoRA modules, merged into foundation model.
result NumLLM achieves best performance on financial question-answering benchmark, especially with numeric questions.
Study benchmarks LLMs in portfolio optimization tasks.
problem Evaluate financial decision-making of LLMs.
method Mathematically explicit portfolio optimization problems with multiple-choice questions.
result Distinct performance patterns among LLMs in different financial tasks.
LOB-Bench benchmarks generative AI for financial data, outperforming traditional models.
problem Lack of consensus on evaluating generative AI models for financial data.
method Python-based benchmark with LOB statistics and market impact metrics.
result Generative autoregressive models outperform traditional models in LOB data.
Paper details how to smoothly transition from EONIA to ESTR without significant financial impact.
problem Transition from EONIA to ESTR impacts financial instruments, especially OTC derivatives.
method Detailed analysis of how clean discounting approach based on ESTR affects pricing of OIS, IRS, and XVAs.
result The transition to EONIA-free pricing framework is safe and consistent, ensuring complete elimination of EONIA.
FinanceBench benchmarks LLMs on financial QA, revealing limitations.
problem Evaluating LLMs' performance on financial question answering.
method Developed a comprehensive test suite (FinanceBench) with 10,231 questions, tested 16 models, and manually reviewed answers.
result Existing LLMs have significant limitations for financial QA, especially GPT-4-Turbo.
This paper benchmarks FinGPT for financial datasets using open-source large language models.
problem Challenges in integrating GPT-based models with financial datasets.
method Instruction Tuning paradigm for open-source large language models adapted for financial contexts.
result Demonstrates the effectiveness and adaptability of FinGPT in financial tasks.
FinRL-Meta offers market environments and benchmarks for financial reinforcement learning.
problem Challenges in creating high-quality market environments and benchmarks for financial reinforcement learning.
method DataOps paradigm, automatic pipeline, community-wise competitions, Jupyter/Python demos.
result Openly accessible FinRL-Meta library for data-driven financial reinforcement learning.
We created financial benchmarks for distribution shifts in crude oil prices and volatility.
problem Scarcity of task-labeled time-series benchmarks in finance.
method Transformed asset price data into volatility proxies, generated task labels based on distribution shifts, and made datasets publicly available.
result Inclusion of task labels improves continual learning algorithms' performance on real-world data.
FINCH dataset enables financial Text-to-SQL tasks, improving model evaluation.
problem Lack of a large-scale financial dataset for Text-to-SQL tasks.
method Curated financial dataset (FINCH), benchmarking reasoning and language models.
result Proposed FINCH Score for more accurate financial model evaluation.
We study the pricing and hedging of derivatives in incomplete financial markets by considering the local risk-minimization method in the context of the benchmark approach, which will be called benchmarked local risk-minimization. We show that the proposed benchmarked local risk-minimization allows to handle under extre…
FinStressTS creates synthetic benchmarks for financial forecasting, revealing model weaknesses.
problem Limited failure attribution in real-world financial benchmarks.
method Synthetic benchmark with 30 diagnostic environments linked to six mechanism families.
result Model performance varies by mechanism type, with autoregressive models often outperforming Transformers.
Benchmark evaluates AI-generated financial QA hallucinations, highlighting system vulnerabilities.
problem Ensuring factual accuracy of AI-generated financial QA outputs.
method Developed a benchmark dataset and evaluated six detection methods under clean and noisy conditions.
result LLM-based judges and embedding methods perform best, but degrade under noisy conditions.
InvestLM is a financial domain LLM tuned on LLaMA-65B for investment advice.
problem Improving financial text understanding and advice generation for investment.
method Curated financial instruction dataset, LLaMA-65B, less-is-more-for-alignment approach.
result InvestLM provides comparable responses to state-of-the-art commercial models.
FinPT uses large pretrained models to predict financial risks.
problem Outdated algorithms and lack of open financial benchmarks.
method Profile Tuning on large pretrained foundation models.
result Demonstrated effectiveness on FinBench datasets.
REALFIN benchmarks financial reasoning by removing implicit assumptions, revealing model weaknesses.
problem Models struggle when implicit assumptions are missing, leading to incorrect answers.
method Developed a bilingual benchmark that systematically removes essential premises from financial questions.
result General-purpose models over-commit, while finance-specialized models fail to identify missing premises.
CTBench benchmarks cryptocurrency time series generation for trading applications.
problem Lack of comprehensive benchmarks for cryptocurrency time series generation.
method Developed a comprehensive benchmark extsf{CTBench} with 13 metrics across 5 dimensions.
result Uncovered trade-offs between statistical fidelity and real-world profitability.
FinTMMBench benchmarks RAG systems for finance tasks across multiple data types and time periods.
problem Evaluating temporal-aware multi-modal retrieval augmented generation in finance.
method TMMHybridRAG method that converts and integrates data from various modalities and temporal information.
result Demonstrated effectiveness of TMMHybridRAG in diverse financial analysis tasks.
Study builds a Japanese financial-specific LLM through continual pre-training.
problem Lack of domain-specific Japanese financial LLMs.
method Continual pre-training on Japanese financial-focused datasets using a base Japanese LLM.
result Tuned model outperforms original model on Japanese financial benchmarks.
QuantBench benchmarks AI methods for quantitative investment.
problem Lack of a standardized benchmark for AI in quantitative investment.
method Developed an industrial-grade benchmark platform with standardization, flexibility, and full-pipeline coverage.
result Revealed critical research directions in AI for quantitative investment.
AlphaForgeBench evaluates LLMs as quantitative researchers, not trading agents, to address instability in financial decision-making.
problem Behavioral instability of LLMs in sequential decision-making under financial uncertainty.
method Proposes AlphaForgeBench, a framework that requires LLMs to generate executable alpha factors and compose factor-based trading strategies.
result Eliminates execution-induced instability and provides a rigorous benchmark for evaluating financial reasoning.
FLBench automates federated learning benchmarking.
problem Manual dataset partitioning fails to simulate real-world isolated data islands.
method Develops a federated learning benchmark suite with three domains.
result Automates evaluation of federated learning systems and algorithms.
Pretrained time-series models outperform train-from-scratch baselines in financial return forecasting.
problem Financial return forecasting
method Pretrained time-series foundation models
result Pretrained TSFMs dominate the ranking distribution, accounting for 8 of 10 task-level wins.
FinSurvival provides a large-scale financial survival modeling benchmark.
problem Lack of large-scale, realistic, and freely available datasets for benchmarking AI survival models.
method Derived 16 survival modeling tasks from cryptocurrency lending data using an automated pipeline.
result Demonstrated that existing AI survival models are not well-suited for these challenging tasks.
Look-Ahead-Bench evaluates financial LLMs for lookahead bias, revealing significant differences in model performance.
problem Measuring and mitigating lookahead bias in financial LLMs.
method Standardized benchmark evaluating model behavior in practical financial scenarios, analyzing performance decay across market regimes.
result Standard LLMs exhibit significant lookahead bias, while Pitinf models show improved generalization and reasoning abilities.
The authors seek financial datasets to benchmark feature engineering methods on US market data.
problem Improving predictive models for financial data science competitions.
method Feature engineering methods applied to multivariate time-series data from the US market.
result Predictive power of models tested against Numerai-Signals targets.
Study uses neural networks to predict firm earnings, outperforming benchmarks and analysts.
problem Limited coverage and biased estimates by financial analysts.
method Developed a neural network model using 40 years of financial data.
result Model outperforms benchmarks and analysts' forecasts for fiscal-year-end earnings predictions.
PortBench benchmarks LLMs for PM, revealing their weaknesses in diversification and robustness.
problem Lack of benchmarks for LLM-driven portfolio management, especially in diversification and robustness.
method Developed a comprehensive benchmark with a static QA dataset and a dynamic allocation pipeline, introducing metrics to evaluate correlation and robustness.
result 90% of LLMs fail to outperform a basic equal-weight allocation, highlighting their limitations in diversification and robustness.
Improved financial performance through better regime prediction.
problem Predicting financial market regimes for profitable trading.
method A novel method combining contrarian trading and frequent short positions.
result Significant performance improvements over four years across three asset classes.