CARE improves LLM aggregation by accounting for shared confounders.
problem LLM judges' correlated errors due to shared latent confounders.
method CARE explicitly models judges' scores as true quality and confounders.
result CARE reduces aggregation error by up to 26.8% across various benchmarks.
Aggregates diverse zero-shot LLM outputs for better corporate disclosure classification.
problem Combining varied zero-shot LLM predictions for improved stock return prediction.
method Multi-prompt framework with three fixed zero-shot LLM classifiers, logistic meta-classifier aggregation.
result Aggregated model outperforms single classifiers and baseline models, increasing balanced accuracy from 0.566 to 0.606.
Study reveals LLM personas have two distinct components: frame-robust aggregated traits and frame-dependent geometric features.
problem Evaluation of LLM personas via psychometric questionnaires discards within-instance correlation structure.
method Constructed within-instance correlation matrices from IPIP-50 responses and analyzed geometry on SPD manifolds under manipulated question orderings.
result Persona expression comprises two dissociable components: aggregated features (Big Five scores) and geometric features (SPD manifold).
New method improves LLM judge accuracy by accounting for dependencies in aggregated binary labels.
problem Classical label aggregation methods fail to account for dependencies among LLM judges, leading to miscalibrated predictions.
method Dependence-aware models based on Ising graphical models and latent factors.
result The proposed method outperforms classical methods on real-world datasets, reducing excess risk.
RESTA defends LLMs against jailbreaking attacks by adding random noise to embeddings.
problem Vulnerability of LLMs to jailbreaking attacks that generate harmful outputs.
method Adds random noise to embedding vectors and aggregates during token generation.
result RESTA achieves superior robustness versus utility tradeoffs compared to baseline defenses.
SmoothLLM defends LLMs from jailbreaking attacks by randomly perturbing inputs.
problem Adversaries can fool large language models into generating objectionable content.
method SmoothLLM randomly perturbs multiple copies of a prompt and aggregates predictions to detect adversarial inputs.
result SmoothLLM sets the state-of-the-art for robustness against various jailbreak attacks.
UQE uses LLMs to analyze unstructured data efficiently.
problem Efficient analytics on unstructured data.
method Proposes UQE, a query engine that uses LLMs to interpret UQL queries.
result Demonstrates efficient analytics on various unstructured data types.
The paper optimizes LLM accuracy by stopping early based on consistent answers.
problem Improving LLM accuracy in math and reasoning problems.
method Bayesian stopping policy to save on sampling costs, tracking only the L-1 most frequent answer counts.
result The L=3 stopping policy is sufficient for asymptotic optimality and significantly reduces inference costs.
MoE-F combines LLMs online for better time-series prediction.
problem Combining multiple LLMs for online time-series prediction.
method Time-adaptive stochastic filtering techniques to combine experts.
result MoE-F achieves 17% absolute and 48.5% relative F1 measure improvement.
LLMs can predict CFO responses to economic surveys
problem Measuring business sentiment
method Prompting an LLM to role-play as a CFO
result LLM reproduces individual human responses
Study uses LLMs for financial sentiment analysis without fine-tuning.
problem Lack of prescriptive knowledge to leverage generative models in FSA.
method Proposes a design framework with heterogeneous LLM agents based on Minsky's theory.
result Framework yields better accuracies, especially with substantial discussions.
This paper fine-tunes LLMs for stock return prediction using financial news.
problem Improving stock return forecasting accuracy using LLMs.
method Fine-tuning LLMs with text and forecasting modules, comparing encoder-only and decoder-only models, and integrating token-level representations.
result LLMs' aggregated token-level embeddings enhance return predictions for long-only and long-short portfolios.
AI agents in experimental markets exhibit behavioral patterns that aggregate into market dynamics.
problem Understanding AI trading behavior and its impact on market dynamics.
method Experimental asset markets populated by AI agents trained on Large Language Models (LLMs).
result AI agents' behavior leads to market dynamics similar to human traders, including bubbles.
Study assesses consistency and reproducibility of LLMs in finance and accounting tasks.
problem Consistency and reproducibility of LLM outputs in finance and accounting research.
method Extensive experimentation with 50 independent runs across 5 tasks using 3 OpenAI models.
result Task-specific patterns of consistency and reproducibility, with binary classification and sentiment analysis achieving near-perfect reproducibility.
Bayesian approach quantifies uncertainty in LLM evaluations.
problem Statistical uncertainty in evaluating LLM behavior.
method Bayesian evaluation of LLM behavior using probabilistic text generation strategies.
result Bayesian approach provides useful uncertainty quantification about LLM behavior.
A new method reduces preference distortion in LLM alignment.
problem Vulnerability of traditional LLM alignment methods to human preference heterogeneity.
method Sign Estimator: A simple, provably consistent, and efficient estimator using binary classification loss.
result Substantially reduces preference distortion over a panel of simulated personas.
Framework controls uncertainty in LLMs without labels or probabilities.
problem Managing uncertainty in black-box LLMs without token-level probability or true labels.
method Integrates generative models, UCP, and conformal alignment to control uncertainty.
result Achieves close-to-nominal coverage and tighter thresholds than split UCP.
Improved financial network predictability using LLM for edge filtering.
problem Spurious edges in financial networks from textual similarity.
method Two-stage framework: sparse candidate graph + LLM edge classification.
result LLM-based edge filtering improves Sharpe ratio and reduces drawdown.
CoT-UQ improves LLM uncertainty quantification by integrating reasoning steps.
problem LLMs' overconfidence and lack of response-wise uncertainty quantification.
method Integrates LLMs' reasoning steps into uncertainty estimation.
result Significantly improves uncertainty quantification accuracy (5.9% AUROC improvement).
A new framework evaluates LLMs by considering judge reliability.
problem Evaluating LLMs without ground truth labels can lead to biased results.
method Introduces judge-specific discrimination parameters and estimates model quality and judge reliability.
result Improves agreement with human preferences and produces calibrated uncertainty quantification.
Proposes a framework to quantify uncertainty in multi-step decision-making by LLMs.
problem Uncertainty quantification in multi-step decision-making scenarios of LLMs.
method A principled, information-theoretic framework decomposing uncertainty into internal and extrinsic components, and proposing UProp for efficient extrinsic uncertainty estimation.
result UProp significantly outperforms existing single-turn UQ baselines in multi-step decision-making benchmarks.
LLMs outperform strong tabular baselines on industrial car retrofit prediction.
problem Industrial retrofit planning on structured operational data
method Embedding features, direct prompted classification, and ML+LLM stacking
result LLMs outperform strong tabular baselines on industrial car retrofit prediction
The paper proposes a new method to evaluate LLM agent responses using ECDF clustering.
problem The standard evaluation of LLM agent responses via majority voting obscures response quality and distribution.
method The paper introduces a novel evaluation framework based on ECDF of cosine similarities and clustering of ECDFs using distances and k-medoids algorithm. result ECDF clustering reveals interpretable group structures in LLM responses, offering insights into agent settings.
This work explains the structural origins of attention sinks in LLMs.
problem Initial tokens disproportionately monopolize attention scores in LLMs.
method Traced to self-attention's value aggregation process and FFN layer activations.
result Attention sinks form due to variance discrepancy and dimension disparity.
A novel framework combines LLMs and RL for financial portfolio optimization.
problem Optimizing financial portfolios using sentiment analysis and market indicators.
method Hierarchical RL structure with base, meta, and super-agents.
result Achieved a 26% annualized return and Sharpe ratio of 1.2.
Study examines if LLMs' trading styles match real market behavior.
problem Lack of behavioral consistency in LLMs' trading strategies.
method Year-long simulations with LLMs, operationalizing behavioral finance drivers, and comparing with financial theory.
result LLMs' strategy switching is only partially consistent with behavioral finance theories.
PolySwarm uses a swarm of LLMs to predict and arbitrage prediction markets.
problem Real-time prediction market trading and latency arbitrage inefficiencies.
method PolySwarm employs a swarm of 50 diverse LLMs, Bayesian combination, and risk-controlled execution.
result Swarm aggregation outperforms single-model baselines in prediction tasks.
Improved crude oil price forecasting using multi-dimensional LLM sentiment signals.
problem Challenges in predicting crude oil prices due to unstructured news.
method Extracted five sentiment dimensions from GPT-4o, Llama 3.2-3b, and FinBERT models on energy-sector news articles.
result Combining GPT-4o and FinBERT yields the best predictive performance for weekly WTI crude oil futures returns.
Crowdsourced wisdom improves causal learning.
problem Improving causal learning through collective intelligence.
method Crowdsourcing, expert knowledge elicitation, aggregation techniques, and LLMs.
result Collective contributions enhance global causal structure.
Unified framework for certifying LLM reliability without extra supervision.
problem Improving reliability of large language models without additional supervision.
method Unified framework using majority voting and Martingale Majority Certificate (MMC).
result Certifiable inference in LLMs with statistical guarantees and adaptive stopping rules.
Duel-Evolve uses LLM self-preferences for test-time optimization of discrete outputs.
problem Optimizing LLM outputs at test time with limited or unreliable scalar rewards.
method Duel-Evolve uses pairwise comparisons from the LLM to guide optimization, aggregating them via a Bayesian Bradley-Terry model.
result Achieves significant improvement over existing methods in accuracy.
ValueBlindBench tests LLM-generated investment rationales for validity before returns are known.
problem Delayed-ground-truth evaluation of LLM-generated investment rationales.
method Agreement-gated stress testing protocol to validate LLM-judged rationales.
result ValueBlindBench prevents overclaims and identifies flawed financial constructs.
New framework selects high-quality pretraining data without training LLMs.
problem Slow progress in understanding pretraining data due to costly experiments.
method Statistical framework based on perplexity-benchmark correlations.
result Approach outperforms existing methods on multiple benchmarks.
CITE algorithm provides anytime-valid certification of model outputs.
problem Challenges in controlling error levels in LLM self-consistency.
method Certification by Intersection-union Testing with E-processes (CITE) algorithm.
result Provable control of false certification at any prescribed level under arbitrary stopping rules.
Service-induced congestion in memory-constrained LLM serving
problem Service-induced congestion in memory-constrained large language model (LLM) serving
method Developing a discrete-time dynamical model of memory-constrained LLM inference
result The system converges to a unique worst-case limit cycle that is asymptotically stable outside a Lebesgue-measure-zero exact-capture set, with throughput losses as large as 50%.
Study proposes a multi-agent framework to mitigate bias in sentiment analysis.
problem Bias in sentiment analysis models.
method Integrates multiple LLMs, incorporates dialogue sessions, and uses probabilistic prediction.
result KCS+IBC reduces entropy and increases variance, suggesting improved balance between aggregation and diversity.
ChatGPT snapshots predict future stock returns.
problem Predicting future stock returns using pre-cutoff text.
method Extracted LLM outlook scores from OpenAI snapshots.
result Outlook scores positively correlate with future stock returns.
New approach treats coordination as an architectural layer to improve LLM-based multi-agent systems.
problem Coordination defects lead to high failure rates in LLM-based multi-agent systems.
method Treats coordination as a configurable architectural layer separable from agent logic and information access.
result Configurations leave distinguishable signatures, enabling architectural reasoning and Pareto frontiers.
Modern LLMs fail at authorship attribution without fine-tuning, but topic embeddings outperform them.
problem Authorship attribution of the Federalist Papers using modern LLMs.
method Examined popular LLMs, compared word/phrase embeddings, and used Bayesian analysis with topic embeddings.
result Topic embeddings trained on 'function words' outperform default LLM embeddings in authorship attribution.
Enhances price sentiment index using survey comments.
problem Improving accuracy in price sentiment analysis.
method Classified comments from Economy Watchers Survey using LLMs.
result Higher correlation with existing indices.
FinReflectKG - EvalBench benchmarks financial KG extraction from SEC 10-K filings.
problem Lack of universal benchmark and evaluation framework for financial KG construction.
method Agentic and holistic evaluation principles, deterministic commit-then-justify judging protocol, binary and ordinal evaluations.
result Reflection-based extraction outperforms single-pass extraction in comprehensiveness, precision, and relevance.
Traditional methods outperform LLMs in forecasting corporate credit ratings.
problem Forecasting corporate credit ratings using LLMs.
method Comparison of traditional methods (XGBoost) and LLMs (LLaMA) on credit rating forecasting.
result XGBoost outperforms LLMs in forecasting corporate credit ratings.
Unified framework to bridge human and LLM judgments.
problem Systematic discrepancies between human and LLM evaluations.
method Latent human preference score and linear transformations of covariates.
result Higher agreement with human ratings and exposure of systematic gaps.
LLM-Lasso uses LLMs to improve feature selection in Lasso regression.
problem Improving feature selection in Lasso regression with domain-specific knowledge.
method Combines LLMs with Lasso regularization to generate feature weights.
result Outperforms standard Lasso and feature selection baselines in biomedical studies.
THERMOMETER calibrates LLMs efficiently for diverse tasks.
problem Calibrating large language models is challenging due to computational and versatility issues.
method THERMOMETER learns an auxiliary model for calibrating a LLM using data from multiple tasks.
result THERMOMETER produces better-calibrated responses for new tasks.
GPA improves LLM training speed by 8.71% for Llama-160M models.
problem Training Large Language Models (LLMs) with high memory overhead and slow convergence.
method Generalized Primal Averaging (GPA) extends Nesterov's method to eliminate memory-intensive two-loop structure.
result GPA achieves up to 10.13% speedup over AdamW in training Llama-1B model.
Regurgitative training with synthetic data harms LLM performance.
problem The impact of training LLMs with synthetic data generated by other LLMs.
method Fine-tuning GPT-3.5 with synthetic and real data in a machine translation task.
result Regurgitative training significantly reduces LLM performance.
FLUID-LLM uses LLMs to predict fluid dynamics with improved accuracy.
problem Leveraging LLMs for CFD due to their pattern recognition abilities but struggles with fluid dynamics complexities.
method Combines pre-trained LLMs with spatiotemporal-aware encoding to predict unsteady fluid dynamics.
result Significant performance improvements in CFD predictions across various datasets.