Transformers improve Finnish language modeling, achieving lower perplexity scores.
problem Improving language modeling for Finnish using deep learning models.
method Used BERT and Transformer-XL models in a sub-word setting, compared to LSTM.
result Transformer-XL outperforms LSTM, achieving a 27% better perplexity score.
Ouroboros accelerates training of large Transformer models.
problem Training large Transformer-based language models is slow and resource-intensive.
method Proposes a novel model-parallel algorithm to speed up training.
result Demonstrates significantly faster training speed compared to data parallelism.
Improved Transformer language models using dynamic evaluation.
problem Language model perplexity and accuracy improvements.
method Combining Transformers with dynamic evaluation techniques.
result Significant improvement in language model performance (e.g., 0.99 to 0.94 bits/char).
A framework uses free probability to analyze Transformer models.
problem Understanding the dynamics and complexity of Transformer-based language models.
method Formal operator-theoretic analysis using free probability theory.
result Entropy-based generalization bounds derived under freeness assumptions.
Enhances Transformer for hierarchical language understanding.
problem Lack of explicit hierarchical structure in Transformer models.
method Inspired by U-Net, integrates hierarchical processing into Transformers.
result Improved performance in chit-chat dialogue tasks.
This paper enhances language models with knowledge awareness.
problem Understanding how much knowledge pretrained language models grasp.
method Inserting explicit knowledge layers into pretraining without changing transformer architecture.
result Significantly more knowledge packed into transformer parameters.
Transformer models can solve complex math problems with less data.
problem Solving complex symbolic mathematics problems with limited data.
method Pretrain transformer models on language translation tasks and fine-tune for symbolic math.
result Pretrained transformer models achieve comparable accuracy to state-of-the-art models with less data.
Compressive Transformer learns long-range sequences by compressing past memories.
problem Learning long-range sequences in language and speech models.
method Compressive Transformer compresses past memories for efficient long-range sequence learning.
result State-of-the-art performance on language and speech benchmarks.
Structured dropout reduces transformer depth for efficient training and inference.
problem Overparameterized transformer networks overfit and require large computation.
method LayerDrop structured dropout for efficient pruning and regularization.
result Sub-networks of any depth can be selected from a large network without performance loss.
SMILES Transformer learns molecular fingerprints for drug discovery.
problem Poor performance of rule-based molecular fingerprints in shallow prediction models or small datasets.
method Unsupervised pre-training of a sequence-to-sequence language model on a corpus of SMILES.
result SMILES Transformer outperformed existing methods in small-data settings.
Effective theory for Transformer initialization improves model performance.
problem Improving performance of Transformers at initialization.
method Effective-theory analysis of signal propagation in wide and deep Transformers.
result Particular width scalings of initialization and training hyperparameters.
Analyzes how BPE tokenisation affects corpus statistics and model entropy in transformer models.
problem Understanding how natural language properties relate to tokenisation schemes in transformer models.
method Analyzes Shannon entropy of corpora under Zipfian distribution, investigates BPE transformations, trains language models, and uses attention diagnostics.
result Transformer models trained on BPE-tokenised corpora increasingly agree with Zipfian predictions as BPE depth increases, indicating reduced local token dependencies.
Transformer-XL extends language models' context length without disrupting temporal coherence.
problem Fixed-length context limitation in Transformers.
method Segment-level recurrence and novel positional encoding.
result 80% longer dependency, 450% longer than vanilla Transformers.
StonkBERT predicts stock price movements using company text data.
problem Can language models predict medium-run stock price movements?
method Fine-tuning transformer-based language models (BERT) on company text data (news articles, blogs, annual reports) for stock price performance classification.
result StonkBERT shows substantial improvement in predictive accuracy compared to traditional models, with news articles providing the best results.
ETC improves Transformer models for long and structured inputs.
problem Scaling input length and encoding structured inputs in Transformers.
method Introduces global-local attention, relative position encodings, and CPC pre-training.
result Achieves state-of-the-art results on four natural language datasets.
Transformer models improve arithmetic accuracy with number decomposition.
problem Transformer models struggle with arithmetic operations without decomposition.
method Fine-tuning models with a pipeline that decomposes numbers into units, tens, etc.
result Accuracy increased by 63% in five-digit addition tasks.
Transformer models perform slower than convolutional networks in learning hierarchical language structures.
problem Understanding how neural networks learn hierarchical language structures.
method Theoretical scaling laws and empirical validation of neural network performance.
result Convolutional networks outperform transformers in learning hierarchical language structures.
I-BERT extends Transformer's self-attention to arbitrary input lengths.
problem Transformer models struggle with inductive generalization to unseen input lengths.
method Replaces positional encodings with a recurrent layer.
result I-BERT achieves state-of-the-art results on algorithmic tasks.
Funnel-Transformer reduces computation by compressing sequence data.
problem Redundant token-level representations in language processing.
method Gradually compresses sequence of hidden states to a shorter one.
result Funnel-Transformer outperforms standard Transformer with fewer FLOPs.
Adaptive attention span extends Transformer's context size.
problem Limited context size in Transformers limits model performance.
method A self-attention mechanism that learns optimal attention span.
result State-of-the-art performance on character-level language modeling.
This study analyzes how one-layer transformers learn regular language recognition tasks.
problem Understanding how one-layer transformers solve regular language recognition tasks like even pairs and parity check.
method Theoretical analysis of training dynamics and gradient descent for a one-layer transformer.
result A one-layer transformer can solve even pairs directly but needs CoT for parity check. Training phases show rapid growth in attention layer followed by logarithmic growth in linear layer.
Evolved Transformer improves on Transformer architecture for language tasks.
problem Improving Transformer architecture for sequence tasks.
method Evolutionary architecture search with warm starting and dynamic resource allocation.
result Evolved Transformer achieves state-of-the-art BLEU scores and reduces parameter count.
New model adds persistent memory to self-attention layers for improved performance.
problem Improving transformer performance by removing feed-forward layers.
method Augmenting self-attention layers with persistent memory vectors.
result The model outperforms standard transformers on language modeling benchmarks.
LLMs are compared to Markov chains for natural language processing.
problem Theoretical analysis of LLMs' generalization capabilities.
method Equivalence between LLMs and Markov chains, studying multi-step inference.
result Derives generalization bounds for LLMs, capturing their behavior in practice.
Study reveals neural scaling laws in random graphs and natural language models.
problem Understanding the origin of neural scaling laws in complex systems.
method Examined scaling laws in transformers trained on random walks and simplified natural language models.
result Neural scaling laws emerge in the absence of power law structure in data correlations.
A new transformer model corrects diacritics and typos in multiple languages.
problem Restoring diacritics and correcting typos in online communications.
method Employing a universal ByT5 transformer model trained on 12 languages, including Lithuanian.
result Achieves > 98% accuracy in diacritics restoration and typos correction.
Tool visualizes Transformer model attention for better understanding.
problem Understanding complex attention mechanisms in deep learning models.
method Developed an open-source tool to visualize attention at three levels.
result Visualization helps interpret and analyze Transformer models.
Pre-training on different modalities improves Transformer performance in offline reinforcement learning.
problem Improving Transformer-based models for offline reinforcement learning tasks.
method Empirical investigation of pre-training on language and vision data, analysis of internal representations, and fine-tuning with no context.
result Pre-trained Transformers with language data can utilize context-like information to solve reinforcement learning tasks efficiently.
BreakGPT predicts asset price surges using LLMs.
problem Predicting sharp upward movements in volatile financial markets.
method Adapts LLMs for time series forecasting, combining LLM capabilities with Transformer models.
result BreakGPT effectively captures local and global temporal dependencies.
Linear attention model outperforms full attention in language tasks.
problem Quadratic scaling of full attention limits model size.
method Introduced a linear attention mechanism.
result Linear attention models achieve comparable performance to full attention models.
Improved BERT model with latent persona and topic variables.
problem Improving BERT's domain-specific utility while maintaining generalization.
method Combining BERT with Universal Transformer, adding latent persona and topic variables.
result Pre-trained model for social texts outperforms baseline.
New language models improve essay scoring accuracy.
problem Improving essay scoring accuracy using AI.
method Used BERT and XLNet language models to compare with traditional methods.
result Achieved above human-level accuracy on AES dataset.
A new method uses a frozen language model to improve sample efficiency in reinforcement learning.
problem Improving sample efficiency in reinforcement learning with partially observable environments.
method FROZEN Hopfield network and HELM (History Embedding Language Model) method.
result HELM achieves new state-of-the-art results on Minigrid and Procgen environments.
Transformer models improve financial sentiment measurement.
problem Capturing nuanced sentiment from financial news articles.
method Transformer-based language models for sentiment classification and aggregation.
result Transformer models outperform traditional dictionary-based methods in sentiment classification.
ViLT is a faster vision-and-language model without convolution or region supervision.
problem Efficiency and expressive power limitations in current VLP models.
method A convolution-free Vision-and-Language Transformer (ViLT) that processes visual inputs similarly to textual inputs.
result ViLT is up to tens of times faster with competitive or better performance.
Paper trains models to resist string transformations.
problem Vulnerability of NLP models to adversarial string transformations.
method Combines search and abstraction techniques for robust training.
result Trained models resist combinations of user-defined transformations.
Transformers can perform well with less long-range memory.
problem The need for deep long-range memory in Transformers for language modeling.
method Performed interventions to show performance can be achieved with fewer long-range memories and by limiting attention range.
result Comparable performance can be achieved with 6X fewer long-range memories and better performance with limited attention range.
The study analyzes numerical stability in large language models using mixed-precision arithmetic.
problem Numerical stability of large language models using low-precision arithmetic.
method Developed a mixed-precision analysis of transformer inference, deriving bounds for condition numbers and forward error.
result Established that numerical stability is determined by the interplay between weight magnitude and the growth of the residual stream.
Generates biomedical abstracts from titles, years, and keywords.
problem Difficulties in understanding biomedical research papers due to specialized language.
method Conditional transformer-based language model with metadata conditioning.
result Generated abstracts are more relevant and non-trivial than GPT-2.
The study analyzes the convergence rate of a large transformer model.
problem Understanding the convergence rate of over-parameterized transformer models.
method Theoretical analysis focusing on gradient descent optimization.
result Theoretical upper bound on missclassification probability.
Develops a cross-lingual hate speech detection model using pre-trained Transformers.
problem Detecting hate speech in low-resource languages.
method Utilizes frozen Transformer language models and AXEL attention-based classification block for zero-shot and few-shot learning.
result Demonstrates highly competitive results on English and Spanish subsets of the HatEval challenge.
Transformer model for probabilistic dynamical systems.
problem Modeling high-dimensional dynamical systems from noisy observations.
method Parallel between dynamical systems and language modeling; transformer-based model with geometrical properties; iterative training algorithm.
result Fine-grid approximation of conditional probabilities for high-dimensional systems.
This paper investigates efficient Transformers and finds they scale with problem size.
problem Finding suitable replacements for standard Transformers in large-scale tasks.
method Modeling efficient Transformers (Sparse and Linear) as Dynamic Programming problems and analyzing their reasoning capabilities.
result Efficient Transformers scale with problem size, but can be more efficient for certain DP problems.
Linformer reduces transformer complexity to linear, improving efficiency.
problem High cost of training and deploying large transformer models for long sequences.
method Approximates self-attention with low-rank matrix, proposing Linformer with O(n) complexity. result Linformer performs similarly to standard transformers but is more memory- and time-efficient.
This paper addresses anisotropy in Transformer models, providing geometric insights and empirical support.
problem Anisotropy phenomenon in Transformer models, challenging their geometric interpretation.
method Derive geometric arguments and use concept-based mechanistic interpretability during training.
result Activation-derived directions capture large gradient energy and a larger share of gradient anisotropy than normal controls.
Downscaled models outperform larger ones on GLUE tasks.
problem Difficulty in attributing performance changes to specific factors in large language models.
method Pre-trained down-scaled versions of Transformer-based architectures on a common corpus, benchmarked on GLUE tasks.
result MLM + NSP (BERT-style) consistently outperforms other objectives.
Transformers' self-attention mechanism is mapped to a generalized Potts model.
problem Uncertainty in what type of data distribution self-attention can efficiently learn.
method Decouple word positions and embeddings, then show self-attention learns a generalized Potts model.
result Training self-attention is equivalent to solving the inverse Potts problem.
Tree-Transformer improves grammar correction in code and natural language.
problem Grammar correction in tree-structured data.
method Tree-Transformer neural network architecture for tree-structured data.
result Significant improvement in grammar correction accuracy.