Regurgitative training with synthetic data harms LLM performance.
problem The impact of training LLMs with synthetic data generated by other LLMs.
method Fine-tuning GPT-3.5 with synthetic and real data in a machine translation task.
result Regurgitative training significantly reduces LLM performance.
Traditional methods outperform LLMs in forecasting corporate credit ratings.
problem Forecasting corporate credit ratings using LLMs.
method Comparison of traditional methods (XGBoost) and LLMs (LLaMA) on credit rating forecasting.
result XGBoost outperforms LLMs in forecasting corporate credit ratings.
Paper presents a probabilistic model to improve LLM cascade performance.
problem Complexity of LLM cascades and their interaction error rates.
method Probabilistic model for joint performance distribution of LLMs.
result Improves area under the error-cost curve by 4.3% on average for cascades with k≥3 models.
PromptEval estimates LLM performance across many prompts, improving reproducibility.
problem Limited prompt templates affect LLM benchmark reproducibility.
method Estimates performance distribution across many prompts using borrowed strength.
result PromptEval accurately estimates performance quantiles with practical budget.
This study evaluates zero-shot LLMs in finance, finding ChatGPT performs well but fine-tuned models are better.
problem Evaluating zero-shot LLMs in financial tasks.
method Comparison of ChatGPT and fine-tuned models on annotated data.
result Fine-tuned models generally outperform zero-shot LLMs.
FLUID-LLM uses LLMs to predict fluid dynamics with improved accuracy.
problem Leveraging LLMs for CFD due to their pattern recognition abilities but struggles with fluid dynamics complexities.
method Combines pre-trained LLMs with spatiotemporal-aware encoding to predict unsteady fluid dynamics.
result Significant performance improvements in CFD predictions across various datasets.
Sloth predicts LLM performance using latent skills across families.
problem Variations in benchmark performance due to differences in training configurations and data processing across model families.
method Sloth uses publicly available benchmark data and assumes LLM performance is driven by latent skills influenced by model size and training tokens. It exploits correlations across benchmarks to provide accurate predictions.
result Sloth predicts LLM performance accurately and offers insights into scaling behaviors for complex tasks.
Survey of LLMs in finance tasks, including adoption and performance.
problem Utilizing large language models in financial tasks.
method Review of current approaches, decision framework for adoption.
result Synthesizes state-of-the-art for LLMs in finance.
Look-Ahead-Bench evaluates financial LLMs for lookahead bias, revealing significant differences in model performance.
problem Measuring and mitigating lookahead bias in financial LLMs.
method Standardized benchmark evaluating model behavior in practical financial scenarios, analyzing performance decay across market regimes.
result Standard LLMs exhibit significant lookahead bias, while Pitinf models show improved generalization and reasoning abilities.
AlphaPruning optimizes LLM pruning using HT-SR theory for better performance.
problem Improving pruning of large language models to reduce size without sacrificing performance.
method AlphaPruning uses HT-SR theory to allocate layerwise sparsity ratios more theoretically.
result AlphaPruning prunes LLaMA-7B to 80% sparsity with reasonable perplexity.
This paper fine-tunes LLMs for stock return prediction using financial news.
problem Improving stock return forecasting accuracy using LLMs.
method Fine-tuning LLMs with text and forecasting modules, comparing encoder-only and decoder-only models, and integrating token-level representations.
result LLMs' aggregated token-level embeddings enhance return predictions for long-only and long-short portfolios.
Improved financial sentiment analysis using LLMs with retrieval augmentation.
problem Limited performance of traditional NLP models in financial sentiment analysis.
method Retrieval-augmented Large Language Models (LLMs) with instruction tuning.
result Achieved 15% to 48% performance gain in accuracy and F1 score.
Tiny benchmarks reduce LLM evaluation costs by using fewer examples.
problem Expensive evaluation of LLMs with tens of thousands of examples.
method Developed evaluation tools and tiny versions of popular benchmarks.
result Accurately estimate LLM performance with just 100 curated examples.
New framework improves LLM performance by avoiding forgetting during sequential training stages.
problem Forgetting during sequential training stages of LLMs.
method Proposes a joint post-training framework with theoretical convergence guarantees.
result Empirically outperforms sequential post-training framework by up to 23%.
Study uses LLMs to improve Black-Litterman portfolio optimization.
problem Systematically generating investor views for Black-Litterman model.
method Translates LLM return forecasts and uncertainty into Black-Litterman inputs.
result LLM-driven portfolios outperform traditional baselines.
Unified LLM-agent with RL improves financial trading performance.
problem LLMs struggle with complex, multi-step financial tasks.
method Fusion of LLMs and gradient-based RL for policy optimization.
result Improves LLM performance in trading and other financial tasks.
LLMs predict SGD convergence without training on specific data.
problem Predicting SGD convergence without labeled data.
method Link between SGD and Markov chains, leveraging LLMs' understanding of dynamical systems.
result Zero-shot prediction of SGD convergence for new starting points.
Small LLMs outperform large ones on simple tasks without extra labelling costs.
problem Performance of large commercial models in simple classification tasks.
method Logistic Regression on small LLM embeddings.
result Small LLMs equal or outperform large LLMs in 'tens-of-shot' classification tasks.
New benchmarks show LLMs struggle with causal discovery.
problem Leveraging LLMs for causal discovery is unreliable due to dataset leakage.
method Developing science-grounded benchmarks and hybrid methods combining LLM predictions with statistical analysis.
result LLMs perform poorly on novel, real-world scientific studies compared to classical methods.
CARROT optimizes LLM routing by choosing the cheapest and most accurate model.
problem Optimizing query routing to the most cost-effective LLM for a given task.
method CARROT uses cost and performance estimates to select the best LLM for each query.
result CARROT is minimax optimal, selecting the best LLM for any query.
Statsformer validates and adapts LLM-derived semantic priors for improved supervised learning.
problem Unreliable semantic priors from LLMs can degrade supervised learning performance.
method Adapts LLM-derived feature scores into a family of learner-specific prior-injection mechanisms, calibrating their influence using out-of-fold validation.
result Improves prediction performance by adaptively downweighting unreliable LLM priors, ensuring a guardrailed statistical learning system.
LLMs outperform human analysts in predicting earnings direction.
problem Evaluating financial statements without narrative or industry-specific information.
method Trained GPT4 on standardized, anonymous financial statements and instructed to predict earnings direction.
result LLMs predict earnings directionally with accuracy comparable to narrowly trained ML models.
This paper simplifies fine-tuning for small LLMs, reducing barriers for developers.
problem Limited resources for fine-tuning large language models (LLMs) by individual developers and small organizations.
method Instruction-tuning datasets, small-sized LLMs (3B to 7B parameters), various training configurations and strategies.
result Improved model performance on benchmarks with specific training configurations, and insights into early termination and hyperparameter simplifications.
Study constructs a Japanese financial LLM benchmark.
problem Need for domain-specific benchmarks for LLMs.
method Constructed a benchmark with multiple Japanese and financial domain tasks.
result GPT-4 outperforms other models in the benchmark.
Best-of-∞ improves LLM performance by efficiently allocating inference-time computation.
problem Achieving optimal performance in test-time LLM ensembling with infinite budget.
method Adaptive generation scheme and weighted ensembles of LLMs, formulated as mixed-integer linear program.
result Optimal ensemble weighting improves performance over individual models.
PriceSeer benchmarks LLMs in real-time stock prediction.
problem Evaluating LLMs' stock prediction accuracy and robustness.
method Real-time benchmark with 110 U.S. stocks, internal and external information expansion.
result LLMs perform suboptimally in long-term predictions due to fake news and specific industries.
Valid certifies LLMs' domain adherence, bounding out-of-domain behavior.
problem Adversarial susceptibility of LLMs to generate out-of-domain outputs.
method VALID approach providing adversarial bounds as a certificate.
result Validates LLMs' domain adherence with meaningful certificates.
LOLA uses LLMs to optimize content delivery, outperforming traditional methods.
problem Identifying the most engaging headlines for user engagement.
method LOLA integrates LLMs with adaptive experimentation to optimize content delivery.
result LOLA outperforms traditional methods in optimizing user engagement.
InvestorBench benchmarks LLM agents in financial tasks.
problem Lack of a comprehensive benchmark for LLM-based financial agents.
method Developed a benchmark with diverse financial tasks and datasets.
result Evaluated LLM agents' performance across various financial products and market environments.
PredictaBoard benchmarks LLM score predictors to assess their ability to anticipate errors.
problem Inconsistent performance of LLMs in common sense reasoning tasks.
method Collaborative benchmarking framework evaluating pairs of LLMs and assessors using rejection rate at different tolerance errors.
result Highlights the need to evaluate predictability alongside performance for safer AI systems.
Compressed LLM embeddings improve noisy regression tasks without overfitting.
problem Noisy regression tasks with high signal-to-noise ratios.
method Comparison of embedding compression techniques using autoencoder hidden representations.
result Compression improves performance on noisy tasks like financial return prediction.
LLM Pro Finance Suite enhances financial NLP with instruction-tuned models.
problem Limited NLP capabilities for financial tasks in generalist models.
method Instruction-tuned large language models fine-tuned on financial data.
result Consistent improvement over state-of-the-art baselines in finance tasks.
Bayesian approach quantifies uncertainty in LLM-based systems.
problem Uncertainty quantification in LLM-based systems, especially for high-stakes applications.
method Interpreting prompts as parameters in a Bayesian model, using MHLP for inference.
result Improvements in predictive accuracy and uncertainty quantification on various benchmarks.
LLMs overestimate stock returns and are less accurate at predicting extreme outcomes.
problem Behavioral biases in LLMs' stock return forecasts.
method Comparison of LLM forecasts with crowd-sourced estimates and historical data.
result LLMs overestimate stock returns and are less accurate at predicting extreme outcomes.
Generative AI models enhance sector-based investment portfolios, but performance varies by market conditions.
problem Improving investment performance through better stock selection in volatile markets.
method Applied LLMs from OpenAI, Google, Anthropic, DeepSeek, and xAI to select and weight stocks within S&P 500 sectors.
result LLM-weighted portfolios outperform sector indices in stable markets but underperform in volatile ones.
Study builds a Japanese financial-specific LLM through continual pre-training.
problem Lack of domain-specific Japanese financial LLMs.
method Continual pre-training on Japanese financial-focused datasets using a base Japanese LLM.
result Tuned model outperforms original model on Japanese financial benchmarks.
New framework reduces LLM complexity by directly finetuning in Boolean domain.
problem Reducing the complexity of large language models (LLMs) while maintaining performance.
method Proposes a novel framework using multi-kernel Boolean parameters for direct finetuning in the Boolean domain.
result Significantly reduces complexity during both finetuning and inference, outperforming recent techniques.
Study compares LLMs vs classical models for financial sentiment analysis.
problem Improving sentiment analysis in financial market news.
method Comparative analysis of LLMs and classical models.
result LLMs outperform classical models in sentiment analysis of financial news.
Telescope detects LLM generated text by measuring token repetition probability.
problem Distinguishing LLM generated text from human writing.
method Telescope Perplexity, evaluating token repetition probability.
result Telescope Perplexity enables effective zero-shot LLM detection.
Survey examines LLMs in financial trading.
problem Using LLMs to outperform professional traders in finance.
method Comprehensive review of current research on LLMs in financial trading.
result LLMs can potentially outperform professional traders in backtesting.
Study evaluates LLMs for predicting Chinese stock movements using financial news sentiments.
problem Evaluating LLMs' ability to predict stock price movements using financial news sentiments.
method Standardized experimental procedure with three LLMs, each with unique performance enhancement methods.
result Developed quantitative trading strategies and conducted back-tests to assess LLMs' performance.
Study evaluates if LLMs have company-specific biases in financial sentiment analysis.
problem Evaluating if large language models exhibit company-specific biases in financial sentiment analysis.
method Comparing sentiment scores with and without company names, constructing economic models, and empirical analysis.
result LLMs show company-specific biases in sentiment analysis, impacting investor behavior and stock prices.
Fine-tuned open-source LLMs match or exceed closed-source models in social science research.
problem Limited scalability and high costs of large LLMs in social science research.
method Fine-tuning open-source models for specific tasks, exploring training set size effects, proposing hybrid workflow.
result Small, fine-tuned open-source LLMs achieve equal or superior performance to commercial alternatives.
Proposes a new algorithm for efficient online model selection of LLMs considering the increasing-then-converging trend.
problem Balancing cost and performance in choosing the best LLM among a diverse set of models.
method Introduces a time-increasing bandit algorithm (TI-UCB) that predicts model performance increases and balances exploration and exploitation.
result Achieves a logarithmic regret upper bound, indicating a fast convergence rate in model selection.
Improves feature selection in high-dimensional data using LLM-generated weights.
problem Inaccurate LLM-generated weights degrade feature selection performance.
method Integrates LLM-generated weights into prior inclusion probabilities using LLM Sparsity Prior (LSP).
result Improves prediction accuracy and identifies clinically relevant features.
Framework uses LLMs to automate strategy finding in quantitative finance.
problem Brittleness of traditional deep learning models in financial applications.
method Three-stage framework with prompt-engineered LLMs, multimodal agent-based evaluation, and dynamic weight optimization.
result Robust performance in Chinese & US markets, superior risk-adjusted performance.
Improved unlearning in LLMs using contrastive decoding.
problem Removing specific information from LLMs while preserving overall performance.
method Inference-time unlearning algorithm using contrastive decoding with two auxiliary models.
result Notable gains in forget quality and retained performance compared to prior approaches.
Paper proposes FinAR-Bench to evaluate LLMs in financial analysis tasks.
problem Inaccurate financial analysis by LLMs leading to investment and regulatory issues.
method Proposes FinAR-Bench, a benchmark dataset with three steps: key info extraction, financial indicator calculation, and logical reasoning.
result LLMs perform better in key info extraction and indicator calculation but struggle with logical reasoning.