Improved language models with talking-heads attention.
problem Language model perplexity and quality issues.
method Added linear projections in multi-head attention.
result Better perplexities and quality in language tasks.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Small Language Models”
Small LLMs outperform large ones on simple tasks without extra labelling costs.
problem Performance of large commercial models in simple classification tasks.
method Logistic Regression on small LLM embeddings.
result Small LLMs equal or outperform large LLMs in 'tens-of-shot' classification tasks.
Study investigates predictive coding models for phonemic learning.
problem Understanding how predictive coding models generalize to different languages and dataset sizes.
method Investigated Autoregressive Predictive Coding and Contrastive Predictive Coding models in phoneme discrimination tasks for two languages with varying dataset sizes.
result Contrastive Predictive Coding model converges rapidly and outperforms Autoregressive Predictive Coding on both languages.
Generative Adversarial Networks (GANs) enjoy great success at image generation, but have proven difficult to train in the domain of natural language. Challenges with gradient estimation, optimization instability, and mode collapse have lead practitioners to resort to maximum likelihood pre-training, followed by small a…
Study improves estimation of rare language model outputs.
problem Estimating probabilities of rare outputs in language models.
method Importance sampling vs. activation extrapolation for low probability estimation.
result Importance sampling outperforms activation extrapolation.
Paper reduces vocabulary losslessly for language model cooperation.
problem Language models struggle to cooperate with different tokenizations.
method Established a theoretical framework for lossless vocabulary reduction.
result Efficiently converts models with different tokenizations to cooperate with maximal common vocabulary.
Modern LLMs fail at authorship attribution without fine-tuning, but topic embeddings outperform them.
problem Authorship attribution of the Federalist Papers using modern LLMs.
method Examined popular LLMs, compared word/phrase embeddings, and used Bayesian analysis with topic embeddings.
result Topic embeddings trained on 'function words' outperform default LLM embeddings in authorship attribution.
Vanishing gradients hinder reinforcement finetuning of language models.
problem Vanishing gradients impede the optimization of language models using reinforcement finetuning.
method The study identifies vanishing gradients as a fundamental optimization obstacle in reinforcement finetuning and proposes an initial supervised finetuning phase to mitigate this issue.
result An initial supervised finetuning phase is crucial for successful reinforcement finetuning of language models, as it helps prevent vanishing gradients and maximizes rewards.
In many scenarios of a language identification task, the user will specify a small set of languages which he/she can speak instead of a large set of all possible languages. We want to model such prior knowledge into the way we train our neural networks, by replacing the commonly used softmax loss function with a novel …
In drug-discovery-related tasks such as virtual screening, machine learning is emerging as a promising way to predict molecular properties. Conventionally, molecular fingerprints (numerical representations of molecules) are calculated through rule-based algorithms that map molecules to a sparse discrete space. However,…
ALIEN improves uncertainty estimation of language models by refining entropy-based methods.
problem Overconfidence in uncertainty estimation for language models, especially for difficult inputs.
method ALIEN refines entropy-based uncertainty by aligning it with prediction reliability, using a lightweight uncertainty head.
result ALIEN consistently outperforms strong baselines in detecting incorrect predictions and achieving the lowest calibration error.
The Partial Testimony of Logs: Evaluation of Language Model Generation under Confounded Model Choicecs.LG
Study uses three sources to evaluate language models fairly.
problem Bias in offline model evaluation due to confounded model choice.
method Combines observational logs, randomized experiments, and simulators.
result Randomized experiment and simulator together recover causal model values.
Self-taught optimizer improves code generation using language models.
problem Improving code generation using language models.
method Recursive self-improvement of a scaffolding program that generates code.
result Improved scaffolding program generates programs with significantly better performance.
Study identifies key parameters and input dimensions making LLMs and VLMs brittle.
problem Vulnerability of large language and vision-language models to perturbations.
method Proposed FI measure based on information geometry to quantify sensitivity.
result Small subset of high FI parameters significantly contribute to brittleness.
ZeroSCROLLS benchmarks zero-shot natural language understanding over long texts.
problem Evaluate natural language understanding models over long texts without training data.
method Adapt six tasks from SCROLLS benchmark and add four new datasets, including novel aggregation tasks.
result Claude outperforms ChatGPT, and GPT-4 achieves highest average score.
Widespread adoption of electronic health records (EHRs) has fueled the development of using machine learning to build prediction models for various clinical outcomes. This process is often constrained by having a relatively small number of patient records for training the model. We demonstrate that using patient repres…
We provide a comprehensive investigation of different custom and off-the-shelf architectures as well as different approaches to generating feature vectors for offensive language detection. We also show that these approaches work well on small and noisy datasets such as on the Offensive Language Identification Dataset (…
Word translation is a problem in machine translation that seeks to build models that recover word level correspondence between languages. Recent approaches to this problem have shown that word translation models can learned with very small seeding dictionaries, and even without any starting supervision. In this paper w…
In natural language processing, it has been observed recently that generalization could be greatly improved by finetuning a large-scale language model pretrained on a large unlabeled corpus. Despite its recent success and wide adoption, finetuning a large pretrained language model on a downstream task is prone to degen…
The CLT fails for LLM evaluations with small data, leading to underestimation of uncertainty.
problem Inaccurate uncertainty estimates in LLM evaluations with small datasets.
method Alternative frequentist and Bayesian methods for uncertainty quantification.
result CLT-based methods underestimate uncertainty in small data settings.
In this paper, we present Russian language datasets in the digital humanities domain for the evaluation of word embedding techniques or similar language modeling and feature learning algorithms. The datasets are split into two task types, word intrusion and word analogy, and contain 31362 task units in total. The chara…
ScheduleFree+ improves large language model training without schedules or learning rates.
problem Scaling up Schedule-Free Learning to large language models.
method Learning-rate-free and schedule-free method for training large language models.
result ScheduleFree+ outperforms SOTA schedules by 31% at 1000 tokens per parameter.
Charmer improves character-level adversarial attacks for language models.
problem Efficiency and effectiveness of character-level adversarial attacks for language models.
method Query-based adversarial attack method that maintains semantic similarity.
result Charmer achieves high attack success rate and similar adversarial examples.
Overparameterized transformer networks have obtained state of the art results in various natural language processing tasks, such as machine translation, language modeling, and question answering. These models contain hundreds of millions of parameters, necessitating a large amount of computation and making them prone t…
Language recognition system is typically trained directly to optimize classification error on the target language labels, without using the external, or meta-information in the estimation of the model parameters. However labels are not independent of each other, there is a dependency enforced by, for example, the langu…
Predicts language model performance from public models without training.
problem Understanding how language model performance varies with scale.
method Observational approach building scaling laws from publicly available models.
result Smooth, sigmoidal behavior of emergent phenomena is predictable from small models.
This work shows fine-tuning improves LLM uncertainty estimates efficiently.
problem Ensuring high-stakes predictions from large language models.
method Fine-tuning on a small dataset of correct and incorrect answers.
result Fine-tuned LLMs produce good generalization and tractable uncertainty estimates.
MoA framework improves financial LLM responses at low cost.
problem Complexity and cost in LLM research for financial domains.
method Mixture of Agents (MoA) framework using customized small models.
result MoA produces higher quality responses at lower costs.
SGPT-PINNs solve PDEs with sparse, small models.
problem Efficiently solving parametric PDEs with minimal resources.
method Sparse and small architecture, mathematically rigorous greedy algorithm, knowledge distillation, down-sampling.
result Achieves high efficiency with significantly fewer parameters.
A simple model explains phase transition in large language models.
problem Understanding the emergence of abilities in large language models.
method Modeling LLM as a sequence-to-sequence random function and using a list decoder.
result A critical threshold exists where the expected number of erroneous sequences grows exponentially.
Improved financial sentiment analysis using simple instruction tuning of LLMs.
problem Lack of accurate financial sentiment analysis by large language models.
method Instruction tuning of general-purpose LLMs with a small portion of financial sentiment data.
result Significant improvement in financial sentiment analysis, especially in complex scenarios.
GNMR controls runtime stability in low-precision language model training.
problem Efficient low-precision training faces numerical risks at specific operators.
method GNMR compares gradient norms to historical means, applying bounded recovery actions.
result GNMR preserves high-fidelity quality with sparse, budgeted recovery.
Neural network language models (NNLMs) have achieved ever-improving accuracy due to more sophisticated architectures and increasing amounts of training data. However, the inductive bias of these models (formed by the distributional hypothesis of language), while ideally suited to modeling most running text, results in …
ChatGPT predicts stock market reactions from news headlines without financial training.
problem Predicting stock price movements using non-financial data.
method Used post-knowledge-cutoff headlines to train ChatGPT-4, which forecasts stock market reactions.
result ChatGPT-4 can predict stock market reactions with high accuracy, especially for small stocks and negative news.
Over the past year, the emergence of transfer learning with large-scale language models (LM) has led to dramatic performance improvements across a broad range of natural language understanding tasks. However, the size and memory footprint of these large LMs makes them difficult to deploy in many scenarios (e.g. on mobi…
A quantum model classifies financial sentiment by mapping text chunks to quantum circuits.
problem Classifying financial texts with high accuracy and preserving semantic information.
method Chunked diagrams are mapped to quantum circuits, with a Transformer encoder and type embeddings added for context.
result The hybrid model improves sentiment classification over a simple averaging baseline.
Synthetic continued pretraining enhances model performance with synthetic data.
problem Data inefficiency in pretrained models when adapting to domain-specific documents.
method Synthetic data augmentation using EntiGraph to create a large synthetic corpus.
result Language models can answer questions and follow instructions without access to domain-specific documents.
Proposes an iterative algorithm for optimizing attention mechanisms in large language models.
problem Optimizing attention mechanisms in large language models.
method Iterative algorithm for rescaled hyperbolic functions regression.
result Efficiency and generalizability of the rescaled softmax regression framework.
Fine-tuning large language models requires minimal data, making them efficient.
problem Achieving state-of-the-art performance with large language models.
method Using BERT as an example, fine-tuning only the most critical layers of the pre-trained model.
result Fine-tuned models are close in parameter space to the pre-trained model, with many good solutions found in sparsified versions.
This paper analyzes speculative decoding, a method to speed up large language model inferences.
problem Theoretical understanding of speculative decoding is lacking.
method Conceptualizes speculative decoding as a markov chain problem and studies its key properties.
result Reveals fundamental connections between LLM components and their impact on decoding efficiency.
Study proposes a multi-agent system using LLMs for REIT trading, outperforming benchmarks.
problem Low-volatility Chinese REIT market, low risk-adjusted returns.
method Multi-agent framework with four types of agents, prediction model pathways, fine-tuning.
result Multi-agent strategies outperform buy-and-hold in terms of return, Sharpe ratio, and drawdown.
End-to-end training of automated speech recognition (ASR) systems requires massive data and compute resources. We explore transfer learning based on model adaptation as an approach for training ASR models under constrained GPU memory, throughput and training data. We conduct several systematic experiments adapting a Wa…
The paper proves impossibilities and positive results for universal machine translation.
problem Learning shared sentence representations across multiple language pairs.
method Formal proofs and analysis of natural generative processes.
result Lower bound on translation error and positive results under natural structure.
A new deep model generates molecules by fragments, improving validity and uniqueness.
problem Generating valid and unique molecules using deep learning.
method Develops a language model for molecular fragments, using frequency-based masking.
result Significantly outperforms other language model-based competitors in molecule generation.
Paper fine-tunes a language model to predict long-term stock buy signals.
problem Predicting long-term stock price movements with narrative text.
method Fine-tuning a small language model on 10-K reports for buy/sell decisions.
result Buy signals generated from 10-K text are most precise at 6 and 9 months, providing 4.8-9% improvement over random selection.
Multi-language speech datasets are scarce and often have small sample sizes in the medical domain. Robust transfer of linguistic features across languages could improve rates of early diagnosis and therapy for speakers of low-resource languages when detecting health conditions from speech. We utilize out-of-domain, unp…
Study uses LLMs to automate data insights discovery.
problem Extracting relevant insights from large data sets.
method Capture the Flag principle, LLMs, reasoning, code generation.
result LLMs can recognize meaningful data insights.
Data selection boosts fact memorization in language models.
problem Language models struggle to accurately memorize factual knowledge.
method Formalizes fact memorization, proposes data selection schemes based on training loss.
result Data selection boosts fact accuracy to model capacity and improves performance.