New technique reduces language biases in large language models.
problem Gap between language and thought in large language models.
method Proposes Language-of-Thoughts (LoT) technique to adjust order and tokens.
result Significantly reduces language biases and improves reasoning tasks.
Researchers analyze the structure of token spaces in large language models.
problem Understanding the geometric and topological structure of token spaces in large language models.
method Developed estimators for the dimension and Ricci scalar curvature of the token subspace and applied them to three large language models.
result The token subspace is not a manifold but a stratified manifold with significantly negative Ricci curvature.
Large language models predict human sensory judgments across multiple modalities.
problem Determining the extent of perceptual information in language.
method State-of-the-art large language models were used to predict sensory judgments across six psychophysical datasets.
result Large language models can predict human sensory judgments across multiple modalities with significant correlation to human data.
AudioPaLM combines text and speech models to improve speech processing and translation.
problem Improving speech understanding and generation models.
method Unified multimodal architecture combining text and speech models.
result AudioPaLM significantly outperforms existing speech translation systems.
Recently, pre-trained language representation flourishes as the mainstay of the natural language understanding community, e.g., BERT. These pre-trained language representations can create state-of-the-art results on a wide range of downstream tasks. Along with continuous significant performance improvement, the size an…
Large language models have recently achieved state of the art performance across a wide variety of natural language tasks. Meanwhile, the size of these models and their latency have significantly increased, which makes their usage costly, and raises an interesting question: do language models need to be large? We study…
Chronologically consistent models maintain accuracy with time-restricted data.
problem Training data introduces lookahead bias and training leakage in large language models.
method ChronoBERT and ChronoGPT trained with only available data at each time point.
result Models achieve strong performance and competitive with larger models, mitigating lookahead bias.
DS-CP improves reliability of uncertainty quantification for large language models under domain shift.
problem Overconfident and factually incorrect outputs (hallucinations) from large language models.
method Adapts conformal prediction to large language models under domain shift by reweighting calibration samples.
result DS-CP delivers more reliable coverage than standard conformal prediction, especially under substantial distribution shifts.
LLMs translate natural language trading intents into correct option strategies using a domain-specific language.
problem Challenges in translating natural language trading intents into correct option strategies due to the complexity of option chain data.
method Introduce Option Query Language (OQL) as a domain-specific intermediate representation to abstract option markets into high-level primitives under grammatical rules. Use LLMs as semantic parsers and validate queries by an engine.
result Significantly improves execution accuracy and logical consistency over direct baselines.
This work uses conformal prediction to quantify uncertainty in large language models for multiple-choice questions.
problem Ensuring robustness and reliability of large language models in high-stakes applications.
method Conformal prediction applied to multi-choice question answering.
result Uncertainty estimates from conformal prediction are closely related to prediction accuracy.
Optimus pre-trains sentences in a latent space for various NLP tasks.
problem Training large-scale language models for diverse NLP tasks.
method Pre-trained Variational Autoencoder (VAE) on large text corpus, fine-tuned for various tasks.
result Optimus achieves state-of-the-art on VAE language modeling benchmarks.
Large language models can't efficiently reason conditionally in a distribution-free setting.
problem Impossibility of conditional PAC-efficient reasoning in large language models.
method Proof of impossibility in a distribution-free setting for non-atomic input spaces.
result Any algorithm achieving conditional PAC efficiency must defer to the expert model with high probability.
New bounds show large language models can generalize beyond training data.
problem Generalization of large language models beyond training data.
method Compression bound derivation using prediction smoothing and SubLoRA.
result Large language models can discover generalizable regularities.
Study constructs a Japanese financial LLM benchmark.
problem Need for domain-specific benchmarks for LLMs.
method Constructed a benchmark with multiple Japanese and financial domain tasks.
result GPT-4 outperforms other models in the benchmark.
LLMs are compared to Markov chains for natural language processing.
problem Theoretical analysis of LLMs' generalization capabilities.
method Equivalence between LLMs and Markov chains, studying multi-step inference.
result Derives generalization bounds for LLMs, capturing their behavior in practice.
This work explains how linear representations in large language models arise from training objectives and gradient descent.
problem Understanding the origins of linear representations in large language models.
method A latent variable model to abstract and formalize concept dynamics, combined with analysis of the softmax cross-entropy objective and gradient descent.
result Linear representations emerge when learning from data matching the latent variable model, and this simple structure suffices to yield linear representations.
POUF fine-tunes large models without labeled data.
problem Lack of labeled data for fine-tuning large pre-trained models.
method Prompt-oriented unsupervised fine-tuning.
result Consistent improvements across various tasks.
Graph signal processing detects hallucinations in large language models.
problem Detecting factual reasoning from hallucinations in large language models.
method Modeling transformer layers as dynamic graphs, using spectral analysis to define diagnostics.
result Spectral signatures can distinguish different types of hallucinations and achieve high accuracy.
Shai is a 10B model for asset management tasks, outperforming baselines.
problem Improving performance in asset management tasks.
method Continuous pre-training and fine-tuning on asset management-specific data.
result Shai outperforms baseline models in asset management tasks.
Large language models improve futures market factor models in China.
problem Designing effective factor models for Chinese futures markets.
method Used large language models (GPT) to generate 40 factors for single and multi-factor portfolios.
result GPT-generated factors outperform benchmarks with high Sharpe ratios and alphas.
This paper benchmarks FinGPT for financial datasets using open-source large language models.
problem Challenges in integrating GPT-based models with financial datasets.
method Instruction Tuning paradigm for open-source large language models adapted for financial contexts.
result Demonstrates the effectiveness and adaptability of FinGPT in financial tasks.
Proposes a robust algorithm for aligning large language models with human preferences.
problem Misspecification in preference models, reference policies, and reward functions.
method Doubly robust preference optimization algorithm.
result Superior and more robust performance compared to state-of-the-art algorithms.
In natural language processing, it has been observed recently that generalization could be greatly improved by finetuning a large-scale language model pretrained on a large unlabeled corpus. Despite its recent success and wide adoption, finetuning a large pretrained language model on a downstream task is prone to degen…
We introduce a new sampling method for large language models that balances diversity and parallelism.
problem Balancing diversity and parallelism in decoding for large language models.
method Arithmetic sampling framework compatible with various sampling variations.
result Improves estimation of expected BLEU score reward and reduces the gap with beam search.
This work investigates uncertainty quantification for black-box large language models in natural language generation.
problem Lack of trustworthiness in responses generated by black-box large language models.
method Differentiated uncertainty vs confidence, proposed and compared several confidence/uncertainty measures, applied to selective NLG.
result A simple measure for semantic dispersion can predict the quality of LLM responses.
Study compares LLMs vs classical models for financial sentiment analysis.
problem Improving sentiment analysis in financial market news.
method Comparative analysis of LLMs and classical models.
result LLMs outperform classical models in sentiment analysis of financial news.
Develops a statistical framework to measure uncertainty in model rankings based on human preferences.
problem Uncertainty in model rankings based on human preferences due to mismatch between human and model preferences.
method Statistical framework using pairwise comparisons by humans and models to provide rank-sets for each model.
result Rank-sets constructed using only pairwise comparisons by strong models often do not cover the true ranking of human preferences.
ScheduleFree+ improves large language model training without schedules or learning rates.
problem Scaling up Schedule-Free Learning to large language models.
method Learning-rate-free and schedule-free method for training large language models.
result ScheduleFree+ outperforms SOTA schedules by 31% at 1000 tokens per parameter.
L3Ms fine-tune LLMs with constraints for tailored applications.
problem Inadequate alignment of LLMs for diverse applications.
method Formulate SFT and alignment as constrained optimization, using logarithmic barriers.
result Versatile and effective in achieving tailored alignments for various applications.
Efficient exploration improves large language model performance with fewer queries.
problem Improving large language model performance with fewer human feedback queries.
method Sequentially generates queries, fits a reward model to feedback, uses double Thompson sampling with epistemic neural network uncertainty.
result Efficient exploration enables high performance with far fewer queries.
The study analyzes how large language models form and express investor risk profiles.
problem Understanding how large language models (LLMs) form and express investor risk profiles.
method Examined three LLMs (GPT, Gemini, and Llama) and assessed their responses to a standardized risk questionnaire under varying prompts.
result LLMs generally form long-term investment profiles, but they exhibit different risk tolerance levels.
Work shows hallucination detection by LLMs is impossible without expert feedback.
problem Detecting hallucinations in LLMs is theoretically impossible without expert-labeled feedback.
method Investigated hallucination detection using a theoretical framework inspired by language identification.
result Automated hallucination detection is impossible for most language collections without expert-labeled feedback.
Study evaluates LLMs for predicting Chinese stock movements using financial news sentiments.
problem Evaluating LLMs' ability to predict stock price movements using financial news sentiments.
method Standardized experimental procedure with three LLMs, each with unique performance enhancement methods.
result Developed quantitative trading strategies and conducted back-tests to assess LLMs' performance.
Survey on using large models to train smaller datasets in NLP.
problem Lack of large datasets and computing resources for NLP tasks.
method Analysis of recent transfer learning approaches in NLP.
result Increased demand for transfer learning in NLP due to large models.
ConMeZO speeds up zeroth-order optimization for large language models.
problem Slow convergence in high-dimensional parameter spaces of large language models.
method Adaptive directional sampling in a cone centered around a momentum estimate.
result Achieves the same convergence rate as MeZO but up to 2X faster.
Study identifies key parameters and input dimensions making LLMs and VLMs brittle.
problem Vulnerability of large language and vision-language models to perturbations.
method Proposed FI measure based on information geometry to quantify sensitivity.
result Small subset of high FI parameters significantly contribute to brittleness.
Two methods improve 10-K item segmentation using large language models.
problem Challenges in extracting specific items from 10-K reports due to variations in document formats and item presentation.
method Two advanced item segmentation methods: GPT4ItemSeg and BERT4ItemSeg.
result BERT4ItemSeg achieves a macro-F1 of 0.9825, surpassing other methods.
This paper uses Gaussian mixtures to mimic interactions in large language models.
problem Simulating interactions in large language models (LLMs) is computationally expensive.
method Developed an interacting Gaussian mixture model (GMM) system to mimic LLM interactions.
result The interacting Gaussian mixture model system can generate, exchange, and update data and parameters efficiently.
Active testing for large language models is made more efficient and accurate.
problem Efficient evaluation of large language models with limited labels.
method Cost-saving measures and in-context learning for constructing a surrogate model.
result Significantly more accurate evaluations of LLM performance compared to random data acquisition.
Proposes an iterative algorithm for optimizing attention mechanisms in large language models.
problem Optimizing attention mechanisms in large language models.
method Iterative algorithm for rescaled hyperbolic functions regression.
result Efficiency and generalizability of the rescaled softmax regression framework.
Method reveals hidden token embeddings of large language models.
problem Understanding hidden token embeddings in large language models.
method Structured prompts to expose token input embeddings up to homeomorphism.
result Mathematical proof for generic LLMs shows effectiveness of method.
BloombergGPT is a large language model trained on financial data, outperforming existing models on financial tasks.
problem Lack of specialized large language models for finance.
method Trained on a 363 billion token dataset augmented with 345 billion tokens from general datasets, using a 50 billion parameter model.
result BloombergGPT outperforms existing models on financial tasks without sacrificing performance on general LLM benchmarks.
This paper investigates how large language models achieve neural collapse, a phenomenon linked to generalization.
problem Neural collapse in large language models under imbalanced and token-rich conditions.
method Empirical investigation of scaling and regularization effects on CLMs' progression towards neural collapse.
result Neural collapse properties develop with scale and regularization, linked to generalization in language modeling.
A simple model explains phase transition in large language models.
problem Understanding the emergence of abilities in large language models.
method Modeling LLM as a sequence-to-sequence random function and using a list decoder.
result A critical threshold exists where the expected number of erroneous sequences grows exponentially.
Automates debiasing for large language model evaluations through Fisher random walk.
problem Rigorous and scalable evaluation of large language models.
method Semiparametric efficient estimator using Fisher random walk for weighted residual balancing.
result Efficient estimation of contextual preference scores for large language models.
New attack reveals memorization patterns in pre-trained LLMs.
problem Determining if a data point was part of a pre-trained LLM's training set.
method Adapts MIA statistical tests to LLM's perplexity dynamics of subsequences.
result Significantly outperforms prior approaches in membership inference attacks.
This work simplifies data valuation for LLMs using Shapley value computation.
problem How to fairly distribute benefits from training superior LLMs with multiple data owners' resources.
method We leverage the specific mathematical structure of DPO to enable scalable Shapley value computation for LLMs.
result We demonstrate that Shapley value computation for LLMs trained with DPO is significantly simplified.
New framework reveals thermodynamic principles for LLM training.
problem Understanding the training dynamics of large language models.
method Introducing Neural Thermodynamic Laws (NTL) under river-valley loss landscape assumptions.
result Key thermodynamic quantities and principles naturally emerge in LLM training.