A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
This paper finds a linear relationship between t-SNE perplexity and data set size.
Perplexity fails to distinguish correct predictions from incorrect ones in some cases.
Proposes a parametric t-SNE without perplexity tuning.
t-Distributed Stochastic Neighbor Embedding (t-SNE) is one of the most widely used dimensionality reduction methods for data visualization, but it has a perplexity hyperparameter that requires manual selection. In practice, proper tuning of t-SNE perplexity requires users to understand the inner working of the method a…
New method reduces discrete flow transitions, improving perplexity estimation.
Transformers improve Finnish language modeling, achieving lower perplexity scores.
We present Meena, a multi-turn open-domain chatbot trained end-to-end on data mined and filtered from public domain social media conversations. This 2.6B parameter neural network is simply trained to minimize perplexity of the next token. We also propose a human evaluation metric called Sensibleness and Specificity Ave…
Improved language models using ratio-matching and KL divergence.
Unified framework for complex, split-complex, and dual numbers.
In order to find hyperparameters for a machine learning model, algorithms such as grid search or random search are used over the space of possible values of the models hyperparameters. These search algorithms opt the solution that minimizes a specific cost function. In language models, perplexity is one of the most pop…
Telescope detects LLM generated text by measuring token repetition probability.
We study large-scale kernel methods for acoustic modeling and compare to DNNs on performance metrics related to both acoustic modeling and recognition. Measuring perplexity and frame-level classification accuracy, kernel-based acoustic models are as effective as their DNN counterparts. However, on token-error-rates DNN…
Enhances LLM quantization with MDBF, improving perplexity and accuracy.
Streaming variational Bayes (SVB) is successful in learning LDA models in an online manner. However previous attempts toward developing online Monte-Carlo methods for LDA have little success, often by having much worse perplexity than their batch counterparts. We present a streaming Gibbs sampling (SGS) method, an onli…
Improved GEC models use scored data from large pretraining to outperform.
Kronecker Products (KP) have been used to compress IoT RNN Applications by 15-38x compression factors, achieving better results than traditional compression methods. However when KP is applied to large Natural Language Processing tasks, it leads to significant accuracy loss (approx 26%). This paper proposes a way to re…
Aioli unifies language model data mixing methods and improves performance.
Many challenges in natural language processing require generating text, including language translation, dialogue generation, and speech recognition. For all of these problems, text generation becomes more difficult as the text becomes longer. Current language models often struggle to keep track of coherence for long pi…
New attack reveals memorization patterns in pre-trained LLMs.
Recurrent Neural Networks (RNN), Long Short-Term Memory Networks (LSTM), and Memory Networks which contain memory are popularly used to learn patterns in sequential data. Sequential data has long sequences that hold relationships. RNN can handle long sequences but suffers from the vanishing and exploding gradient probl…
Theoretical analysis shows MDMs can be efficient but not for all metrics.
Neural text generation models are often autoregressive language models or seq2seq models. These models generate text by sampling words sequentially, with each word conditioned on the previous word, and are state-of-the-art for several machine translation and summarization benchmarks. These benchmarks are often defined …
ECLIPSE detects AI hallucinations in finance with high accuracy.
Paper proposes a new neural machine translation method for wave data.
New framework selects high-quality pretraining data without training LLMs.
Numeracy is the ability to understand and work with numbers. It is a necessary skill for composing and understanding documents in clinical, scientific, and other technical domains. In this paper, we explore different strategies for modelling numerals with language models, such as memorisation and digit-by-digit composi…
Unified SVD compression fails in practical tasks, highlighting the importance of per layer activation reconstruction.
QuantEase optimizes LLMs with CD-based quantization, achieving state-of-the-art performance.
DatedGPT prevents lookahead bias in financial forecasting models.
Neural language models (LMs) based on recurrent neural networks (RNN) are some of the most successful word and character-level LMs. Why do they work so well, in particular better than linear neural LMs? Possible explanations are that RNNs have an implicitly better regularization or that RNNs have a higher capacity for …
Proof of existence of a complex structure on the six-sphere, followed by an explicit computation of its underlying integrable almost complex tensor by the aid of inner automorphisms of the octonions, is exhibited. Both are elementary and self-contained however the size and complexity of the emerging almost complex tens…
We introduce "talking-heads attention" - a variation on multi-head attention which includes linearprojections across the attention-heads dimension, immediately before and after the softmax operation.While inserting only a small number of additional parameters and a moderate amount of additionalcomputation, talking-head…
A neural network method for topic modeling from few documents.
Simplified masked diffusion models improve discrete data generation.
This work compares and evaluates various sampling methods for neural language models.
New method boosts performance of diffusion models on discrete data like natural language.
We present two simple ways of reducing the number of parameters and accelerating the training of large Long Short-Term Memory (LSTM) networks: the first one is "matrix factorization by design" of LSTM matrix into the product of two smaller matrices, and the second one is partitioning of LSTM matrix, its inputs and stat…
AlphaPruning optimizes LLM pruning using HT-SR theory for better performance.
One-Shot Neural architecture search (NAS) attracts broad attention recently due to its capacity to reduce the computational hours through weight sharing. However, extensive experiments on several recent works show that there is no positive correlation between the validation accuracy with inherited weights from the supe…
We propose a new generative model of sentences that first samples a prototype sentence from the training corpus and then edits it into a new sentence. Compared to traditional models that generate from scratch either left-to-right or by first sampling a latent sentence vector, our prototype-then-edit model improves perp…
AutoScale improves LLM pre-training by adjusting data mixtures at different scales.
We develop dependent hierarchical normalized random measures and apply them to dynamic topic modeling. The dependency arises via superposition, subsampling and point transition on the underlying Poisson processes of these measures. The measures used include normalised generalised Gamma processes that demonstrate power …
We characterize language generation with stability and breadth, proving impossibility results.
We present power low rank ensembles (PLRE), a flexible framework for n-gram language modeling where ensembles of low rank matrices and tensors are used to obtain smoothed probability estimates of words in context. Our method can be understood as a generalization of n-gram modeling to non-integer n, and includes standar…
Stories generated with neural language models have shown promise in grammatical and stylistic consistency. However, the generated stories are still lacking in common sense reasoning, e.g., they often contain sentences deprived of world knowledge. We propose a simple multi-task learning scheme to achieve quantitatively …
Efficient sparse attention reduces self-attention complexity and improves model performance.
3BASiL-TM decomposes LLMs into sparse and low-rank matrices for efficient compression.
This research note combines two methods that have recently improved the state of the art in language modeling: Transformers and dynamic evaluation. Transformers use stacked layers of self-attention that allow them to capture long range dependencies in sequential data. Dynamic evaluation fits models to the recent sequen…