A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
New method reduces discrete flow transitions, improving perplexity estimation.
This paper finds a linear relationship between t-SNE perplexity and data set size.
Perplexity fails to distinguish correct predictions from incorrect ones in some cases.
Proposes a parametric t-SNE without perplexity tuning.
t-Distributed Stochastic Neighbor Embedding (t-SNE) is one of the most widely used dimensionality reduction methods for data visualization, but it has a perplexity hyperparameter that requires manual selection. In practice, proper tuning of t-SNE perplexity requires users to understand the inner working of the method a…
Transformers improve Finnish language modeling, achieving lower perplexity scores.
We present Meena, a multi-turn open-domain chatbot trained end-to-end on data mined and filtered from public domain social media conversations. This 2.6B parameter neural network is simply trained to minimize perplexity of the next token. We also propose a human evaluation metric called Sensibleness and Specificity Ave…
ECLIPSE detects AI hallucinations in finance with high accuracy.
Improved language models using ratio-matching and KL divergence.
Unified framework for complex, split-complex, and dual numbers.
Aioli unifies language model data mixing methods and improves performance.
In order to find hyperparameters for a machine learning model, algorithms such as grid search or random search are used over the space of possible values of the models hyperparameters. These search algorithms opt the solution that minimizes a specific cost function. In language models, perplexity is one of the most pop…
Telescope detects LLM generated text by measuring token repetition probability.
We study large-scale kernel methods for acoustic modeling and compare to DNNs on performance metrics related to both acoustic modeling and recognition. Measuring perplexity and frame-level classification accuracy, kernel-based acoustic models are as effective as their DNN counterparts. However, on token-error-rates DNN…
New framework selects high-quality pretraining data without training LLMs.
Improved language models with talking-heads attention.
Enhances LLM quantization with MDBF, improving perplexity and accuracy.
Streaming variational Bayes (SVB) is successful in learning LDA models in an online manner. However previous attempts toward developing online Monte-Carlo methods for LDA have little success, often by having much worse perplexity than their batch counterparts. We present a streaming Gibbs sampling (SGS) method, an onli…
Improved GEC models use scored data from large pretraining to outperform.
Kronecker Products (KP) have been used to compress IoT RNN Applications by 15-38x compression factors, achieving better results than traditional compression methods. However when KP is applied to large Natural Language Processing tasks, it leads to significant accuracy loss (approx 26%). This paper proposes a way to re…
A neural network method for topic modeling from few documents.
Many challenges in natural language processing require generating text, including language translation, dialogue generation, and speech recognition. For all of these problems, text generation becomes more difficult as the text becomes longer. Current language models often struggle to keep track of coherence for long pi…
New attack reveals memorization patterns in pre-trained LLMs.
Recurrent Neural Networks (RNN), Long Short-Term Memory Networks (LSTM), and Memory Networks which contain memory are popularly used to learn patterns in sequential data. Sequential data has long sequences that hold relationships. RNN can handle long sequences but suffers from the vanishing and exploding gradient probl…
This work compares and evaluates various sampling methods for neural language models.
Theoretical analysis shows MDMs can be efficient but not for all metrics.
Neural text generation models are often autoregressive language models or seq2seq models. These models generate text by sampling words sequentially, with each word conditioned on the previous word, and are state-of-the-art for several machine translation and summarization benchmarks. These benchmarks are often defined …
New method boosts performance of diffusion models on discrete data like natural language.
We present power low rank ensembles (PLRE), a flexible framework for n-gram language modeling where ensembles of low rank matrices and tensors are used to obtain smoothed probability estimates of words in context. Our method can be understood as a generalization of n-gram modeling to non-integer n, and includes standar…
Paper proposes a new neural machine translation method for wave data.
Numeracy is the ability to understand and work with numbers. It is a necessary skill for composing and understanding documents in clinical, scientific, and other technical domains. In this paper, we explore different strategies for modelling numerals with language models, such as memorisation and digit-by-digit composi…
Unified SVD compression fails in practical tasks, highlighting the importance of per layer activation reconstruction.
QuantEase optimizes LLMs with CD-based quantization, achieving state-of-the-art performance.
Paper proposes a neural network method for fast, interpretable AR model estimation.
DatedGPT prevents lookahead bias in financial forecasting models.
Neural language models (LMs) based on recurrent neural networks (RNN) are some of the most successful word and character-level LMs. Why do they work so well, in particular better than linear neural LMs? Possible explanations are that RNNs have an implicitly better regularization or that RNNs have a higher capacity for …
Proof of existence of a complex structure on the six-sphere, followed by an explicit computation of its underlying integrable almost complex tensor by the aid of inner automorphisms of the octonions, is exhibited. Both are elementary and self-contained however the size and complexity of the emerging almost complex tens…
We introduce multiplicative LSTM (mLSTM), a recurrent neural network architecture for sequence modelling that combines the long short-term memory (LSTM) and multiplicative recurrent neural network architectures. mLSTM is characterised by its ability to have different recurrent transition functions for each possible inp…
Simplified masked diffusion models improve discrete data generation.
How can we measure whether a natural language generation system produces both high quality and diverse outputs? Human evaluation captures quality but not diversity, as it does not catch models that simply plagiarize from the training set. On the other hand, statistical evaluation (i.e., perplexity) captures diversity b…
We present two simple ways of reducing the number of parameters and accelerating the training of large Long Short-Term Memory (LSTM) networks: the first one is "matrix factorization by design" of LSTM matrix into the product of two smaller matrices, and the second one is partitioning of LSTM matrix, its inputs and stat…
AlphaPruning optimizes LLM pruning using HT-SR theory for better performance.
One-Shot Neural architecture search (NAS) attracts broad attention recently due to its capacity to reduce the computational hours through weight sharing. However, extensive experiments on several recent works show that there is no positive correlation between the validation accuracy with inherited weights from the supe…
We propose a new generative model of sentences that first samples a prototype sentence from the training corpus and then edits it into a new sentence. Compared to traditional models that generate from scratch either left-to-right or by first sampling a latent sentence vector, our prototype-then-edit model improves perp…
AutoScale improves LLM pre-training by adjusting data mixtures at different scales.
Estimating the entropy based on data is one of the prototypical problems in distribution property testing and estimation. For estimating the Shannon entropy of a distribution on elements with independent samples, [Paninski2004] showed that the sample complexity is sublinear in , and [Valiant--Valiant2011] showed…
We develop dependent hierarchical normalized random measures and apply them to dynamic topic modeling. The dependency arises via superposition, subsampling and point transition on the underlying Poisson processes of these measures. The measures used include normalised generalised Gamma processes that demonstrate power …
We characterize language generation with stability and breadth, proving impossibility results.