New method reduces discrete flow transitions, improving perplexity estimation.
problem Stochasticity in discrete paths makes rectification strategies ineffective.
method Dynamic-optimal-transport-like minimization objective with minibatch strategies.
result 32 times reduction in transitions for same perplexity.
Aioli unifies language model data mixing methods and improves performance.
problem Optimizing the mixture of training data groups for language models.
method Unified optimization framework for dynamically adjusting mixture proportions.
result Aioli outperforms existing methods by up to 12.012 test perplexity points.
New attack reveals memorization patterns in pre-trained LLMs.
problem Determining if a data point was part of a pre-trained LLM's training set.
method Adapts MIA statistical tests to LLM's perplexity dynamics of subsequences.
result Significantly outperforms prior approaches in membership inference attacks.
This paper finds a linear relationship between t-SNE perplexity and data set size.
problem Choosing the right perplexity for t-SNE embeddings.
method Analyzed the relationship between perplexity and data set size.
result Embeddings remain structurally consistent when perplexity is adjusted accordingly.
Perplexity fails to distinguish correct predictions from incorrect ones in some cases.
problem Perplexity's limitations in model selection.
method Analyzing Transformer continuity and iso-perplexity plots.
result Perplexity does not always select the correct model.
Proposes a parametric t-SNE without perplexity tuning.
problem Non-parametric t-SNE's perplexity parameter limits DR quality.
method Multi-scale parametric t-SNE with deep neural network.
result Produces reliable embeddings with competitive neighborhood preservation.
t-Distributed Stochastic Neighbor Embedding (t-SNE) is one of the most widely used dimensionality reduction methods for data visualization, but it has a perplexity hyperparameter that requires manual selection. In practice, proper tuning of t-SNE perplexity requires users to understand the inner working of the method a…
Meena is a chatbot trained on social media data, achieving human-like conversation quality.
problem Creating a chatbot that can have human-like conversations in an open-domain setting.
method End-to-end training of a 2.6B parameter neural network on social media data, using perplexity and a human evaluation metric (SSA) to assess quality.
result Meena achieves a high human-like conversation quality (79% SSA) when compared to existing chatbots.
Transformers improve Finnish language modeling, achieving lower perplexity scores.
problem Improving language modeling for Finnish using deep learning models.
method Used BERT and Transformer-XL models in a sub-word setting, compared to LSTM.
result Transformer-XL outperforms LSTM, achieving a 27% better perplexity score.
We develop dependent hierarchical normalized random measures and apply them to dynamic topic modeling. The dependency arises via superposition, subsampling and point transition on the underlying Poisson processes of these measures. The measures used include normalised generalised Gamma processes that demonstrate power …
This research note combines two methods that have recently improved the state of the art in language modeling: Transformers and dynamic evaluation. Transformers use stacked layers of self-attention that allow them to capture long range dependencies in sequential data. Dynamic evaluation fits models to the recent sequen…
Improved language models using ratio-matching and KL divergence.
problem Efficiently modeling discrete data with diffusion models.
method Introduced new theorems and a novel CTMC transition-rate matrix for ratio-matching and KL divergence.
result 10-15% improvement in perplexity and faster training steps.
Unified framework for complex, split-complex, and dual numbers.
problem Analytic and geometric scope of real-analytic functions.
method Generalized Cauchy-Riemann structure and unified real algebra family.
result Milnor-Le type fibration theorem for nondegenerate algebras.
In order to find hyperparameters for a machine learning model, algorithms such as grid search or random search are used over the space of possible values of the models hyperparameters. These search algorithms opt the solution that minimizes a specific cost function. In language models, perplexity is one of the most pop…
Telescope detects LLM generated text by measuring token repetition probability.
problem Distinguishing LLM generated text from human writing.
method Telescope Perplexity, evaluating token repetition probability.
result Telescope Perplexity enables effective zero-shot LLM detection.
We study large-scale kernel methods for acoustic modeling and compare to DNNs on performance metrics related to both acoustic modeling and recognition. Measuring perplexity and frame-level classification accuracy, kernel-based acoustic models are as effective as their DNN counterparts. However, on token-error-rates DNN…
In this paper, we develop the continuous time dynamic topic model (cDTM). The cDTM is a dynamic topic model that uses Brownian motion to model the latent topics through a sequential collection of documents, where a "topic" is a pattern of word use that we expect to evolve over the course of the collection. We derive an…
Improved language models with talking-heads attention.
problem Language model perplexity and quality issues.
method Added linear projections in multi-head attention.
result Better perplexities and quality in language tasks.
Enhances LLM quantization with MDBF, improving perplexity and accuracy.
problem Limited performance of Double Binary Factorization in extreme quantization.
method Introduces Multi-envelope DBF, retaining sign matrices and replacing single envelope with rank-l envelope. result Improves perplexity and zero-shot accuracy over previous binary formats.
Streaming variational Bayes (SVB) is successful in learning LDA models in an online manner. However previous attempts toward developing online Monte-Carlo methods for LDA have little success, often by having much worse perplexity than their batch counterparts. We present a streaming Gibbs sampling (SGS) method, an onli…
Improved GEC models use scored data from large pretraining to outperform.
problem Addressing data sparsity in Grammatical Error Correction.
method Derive example-level scores from a smaller, higher-quality dataset and incorporate delta-log-perplexity into training schedules.
result Models trained on scored data achieve state-of-the-art results.
Dynamic topic models (DTMs) are very effective in discovering topics and capturing their evolution trends in time series data. To do posterior inference of DTMs, existing methods are all batch algorithms that scan the full dataset before each update of the model and make inexact variational approximations with mean-fie…
Many challenges in natural language processing require generating text, including language translation, dialogue generation, and speech recognition. For all of these problems, text generation becomes more difficult as the text becomes longer. Current language models often struggle to keep track of coherence for long pi…
Dynamic sparseness reduces neural network computation by selectively omitting parts of computations.
problem Reducing the computational and memory footprint of neural networks.
method Combining dynamic sparseness with block-wise matrix-vector multiplications to selectively omit parts of computations.
result The proposed method outperforms static sparseness and achieves similar perplexities at half the computational cost.
Recurrent Neural Networks (RNN), Long Short-Term Memory Networks (LSTM), and Memory Networks which contain memory are popularly used to learn patterns in sequential data. Sequential data has long sequences that hold relationships. RNN can handle long sequences but suffers from the vanishing and exploding gradient probl…
This paper improves compression of large NLP models using doped Kronecker Products.
problem Accuracy loss when compressing large NLP tasks with Kronecker Products.
method Doping Kronecker Products with an overlay matrix to recover accuracy, and a new regularization scheme called co matrix dropout regularization (CMR).
result Compression of a large language model with LSTM layers of size 25 MB by 25x with 1.4% loss in perplexity score.
Theoretical analysis shows MDMs can be efficient but not for all metrics.
problem Understanding the efficiency-accuracy trade-off of diffusion language models.
method Theoretical analysis of Masked Diffusion Model (MDM) using perplexity and sequence error rate as metrics.
result MDM achieves near-optimal perplexity but requires linear scaling for sequence error rate, highlighting efficiency-accuracy trade-offs.
Neural text generation models are often autoregressive language models or seq2seq models. These models generate text by sampling words sequentially, with each word conditioned on the previous word, and are state-of-the-art for several machine translation and summarization benchmarks. These benchmarks are often defined …
ECLIPSE detects AI hallucinations in finance with high accuracy.
problem Hallucinations in AI-generated answers limit safe deployment in finance.
method Combines entropy estimation and perplexity decomposition to measure model evidence use.
result ECLIPSE achieves ROC AUC of 0.89 and average precision of 0.90 on financial QA dataset.
Paper proposes a new neural machine translation method for wave data.
problem Limited real-world sensor data for continuous signal waves.
method Introduces window-based representation and iterative back-translation for wave data.
result Significant performance improvements in wave translation tasks.
New framework selects high-quality pretraining data without training LLMs.
problem Slow progress in understanding pretraining data due to costly experiments.
method Statistical framework based on perplexity-benchmark correlations.
result Approach outperforms existing methods on multiple benchmarks.
Efficient sparse attention reduces self-attention complexity and improves model performance.
problem Quadratic compute and memory requirements of self-attention for long sequences.
method Content-based sparse attention with dynamic routing module.
result Routing Transformer achieves state-of-the-art performance on various benchmarks.
Numeracy is the ability to understand and work with numbers. It is a necessary skill for composing and understanding documents in clinical, scientific, and other technical domains. In this paper, we explore different strategies for modelling numerals with language models, such as memorisation and digit-by-digit composi…
Low dimensional representations of words allow accurate NLP models to be trained on limited annotated data. While most representations ignore words' local context, a natural way to induce context-dependent representations is to perform inference in a probabilistic latent-variable sequence model. Given the recent succes…
We reparametrize ReLU NNs as splines to understand their learning dynamics.
problem Understanding the learning dynamics and inductive bias of neural networks.
method Reparametrize ReLU NNs as continuous piecewise linear splines to study learning dynamics.
result Standard weight initializations yield very flat functions, leading to strength and type of implicit regularization.
CATVI improves variational inference for Bayesian nonparametric models by reducing divergence and improving prediction accuracy.
problem Limitations of current variational inference methods in characterizing latent correlations and inferring true posterior dimensions.
method CATVI integrates conditional and adaptive truncation into variational inference, maximizing nonparametric evidence lower bound and using Monte Carlo integration.
result CATVI outperforms traditional methods in Bayesian nonparametric topic models, reducing perplexity and improving topic-word clustering.
Unified SVD compression fails in practical tasks, highlighting the importance of per layer activation reconstruction.
problem The failure of a unified SVD compression method in practical tasks like perplexity and accuracy.
method Unified optimization problem for SVD based compression methods, focusing on cross-layer coupling.
result Downstream metrics like perplexity and accuracy degrade severely compared to standard per layer SVD LLM.
QuantEase optimizes LLMs with CD-based quantization, achieving state-of-the-art performance.
problem Efficiently quantize large language models for deployment.
method Layer-wise quantization using CD-based algorithms with matrix and vector operations.
result State-of-the-art performance in perplexity and zero-shot accuracy.
DatedGPT prevents lookahead bias in financial forecasting models.
problem Lookahead bias in large language models trained on internet-scale data.
method Time-aware pretraining with annual data cutoffs and instruction fine-tuning.
result Models' knowledge is effectively bounded by their data cutoff year, improving forecasting validity.
Neural language models (LMs) based on recurrent neural networks (RNN) are some of the most successful word and character-level LMs. Why do they work so well, in particular better than linear neural LMs? Possible explanations are that RNNs have an implicitly better regularization or that RNNs have a higher capacity for …
Proof of existence of a complex structure on the six-sphere, followed by an explicit computation of its underlying integrable almost complex tensor by the aid of inner automorphisms of the octonions, is exhibited. Both are elementary and self-contained however the size and complexity of the emerging almost complex tens…
A neural network method for topic modeling from few documents.
problem Training topic models requires many documents, limiting their applicability.
method Neural networks learn topic model priors from a few documents, optimizing likelihood using EM algorithm.
result The method achieves better perplexity than existing methods on real-world text document sets.
Neural topic models improve topic discovery in large corpora.
problem Topic discovery in large text corpora.
method Neural auto-encoder based document embedding with PLSA assumptions.
result Neural topic models outperform LDA in capturing relevant topics and perplexity.
Simplified masked diffusion models improve discrete data generation.
problem Complex model formulations and unclear relationships hinder discrete data generative modeling.
method Developed a simple and general framework for masked diffusion models.
result Models trained on OpenWebText surpass prior diffusion language models and outperform autoregressive models.
This work compares and evaluates various sampling methods for neural language models.
problem Lack of systematic comparison and myths about sampling methods.
method Monte Carlo sampling, importance sampling, compensated partial summation, noise contrastive estimation.
result All sampling methods can perform equally well if posterior probabilities are corrected.
New method boosts performance of diffusion models on discrete data like natural language.
problem Performance of diffusion models on discrete data like natural language is poor.
method Proposes score entropy, a novel loss that extends score matching to discrete spaces.
result Significantly boosts performance on language modeling tasks.
Graph signal processing detects hallucinations in large language models.
problem Detecting factual reasoning from hallucinations in large language models.
method Modeling transformer layers as dynamic graphs, using spectral analysis to define diagnostics.
result Spectral signatures can distinguish different types of hallucinations and achieve high accuracy.
We present two simple ways of reducing the number of parameters and accelerating the training of large Long Short-Term Memory (LSTM) networks: the first one is "matrix factorization by design" of LSTM matrix into the product of two smaller matrices, and the second one is partitioning of LSTM matrix, its inputs and stat…