Tensor Train layer improves BLEU scores in NMT models.
problem Improving Neural Machine Translation (NMT) models' performance.
method Implemented Tensor Train layer in TensorFlow for NMT training.
result Higher learning rates and more 'rectangular' core dimensions improve BLEU scores.
EBR improves NMT by re-ranking samples drawn from MLE-trained models.
problem Discrepancy between MLE and BLEU score in neural machine translation.
method Train an energy-based model to mimic BLEU score, then use it for re-ranking.
result EBR consistently improves NMT performance by +4 BLEU points on IWSLT'14 German-English.
Improved speech translation model using cleaned data and ensemble decoding.
problem End-to-end speech translation from English to German.
method Fine-tuning on cleaned data, weight normalization, label smoothing, checkpoint averaging, ensemble decoding.
result Ensemble model achieved a BLEU score of 10.24 on test data.
Transformer++ improves neural machine translation BLEU scores.
problem Handling long-range dependencies in sentences.
method Proposes a new multi-head attention mechanism with context learning.
result Achieves new state-of-the-art BLEU scores on English-to-German and English-to-French translation tasks.
An NMT system for Indic languages outperforms Google Translate.
problem Challenges in translating Indic languages efficiently.
method Encoder-decoder with attention mechanism for neural machine translation.
result Outperforms Google Translate with a 6 BLEU score margin on English-Gujarati translation.
The paper presents a neural machine translation system for Wolof.
problem Low-resource conditions for translating French to Wolof.
method Developed a bidirectional LSTM-based encoder-decoder architecture with attention mechanisms.
result Best model achieved a 47% BLEU score for French-Wolof translation.
Proposes a faster Transformer decoding method by truncating target-side self-attention windows.
problem Efficiency in Transformer decoding with minimal BLEU score loss.
method N-gram assumption to truncate target-side self-attention windows.
result N-gram masked self-attention model maintains BLEU score for N values from 4 to 8. Paper proposes a neural network for generating better questions from text.
problem Automatic generation of relevant questions from sentences and paragraphs.
method Adaptive copying recurrent neural network model with a copying mechanism added to a bidirectional LSTM architecture.
result The model outperforms state-of-the-art methods in question generation metrics.
Parallel decoding improves machine translation efficiency and accuracy.
problem Efficiently generating translations from left to right.
method Conditional masked language modeling for parallel decoding.
result Improves translation performance by over 4 BLEU points.
Paper proposes metrics to evaluate both quality and diversity in text generation models.
problem Existing metrics only evaluate quality or diversity, not both.
method Approximate the distance between generative model and real data distribution using n-gram and BERT features.
result Proposed metrics better evaluate both quality and diversity of text generation models.
Evolved Transformer improves on Transformer architecture for language tasks.
problem Improving Transformer architecture for sequence tasks.
method Evolutionary architecture search with warm starting and dynamic resource allocation.
result Evolved Transformer achieves state-of-the-art BLEU scores and reduces parameter count.
Improved machine translation with INT8 hardware using a novel training method.
problem Training accurate machine translation models with limited hardware precision.
method Convert all Transformer matrix multiplications to 8-bit integer (INT8) without sacrificing accuracy.
result INT8 Transformer models achieve BLEU scores 99.3% to 100% relative to FP32 models.
Paper identifies and solves a 'scrambled translation' issue in UNMT models.
problem Scrambled translation in UNMT models using word shuffle noise.
method Retraining UNMT models without noise after a pre-defined number of iterations.
result Retraining strategy improves BLEU scores by 10-20% on various language pairs.
FullyQT quantizes Transformer models to 8-bit precision without sacrificing translation quality.
problem Reducing computational costs of neural machine translation methods without performance loss.
method Proposes FullyQT, an all-inclusive quantization strategy for the Transformer.
result Achieves state-of-the-art quantization results, maintaining or improving translation quality.
This paper compares log-likelihood and BLEU scores for sequence generation tasks.
problem The discrepancy between density estimation and sequence generation performance.
method Comparing several density estimators on five machine translation tasks.
result The correlation between log-likelihood and BLEU varies depending on model families.
Memory-augmented neural networks improve machine translation performance.
problem Improving machine translation accuracy and flexibility.
method Evaluation of Neural Turing Machines and Differentiable Neural Computers for machine translation tasks.
result Memory-augmented neural networks perform similarly to attentional encoders on Vietnamese to English tasks but have lower BLEU scores on Romanian to English tasks.
We introduce a new sampling method for large language models that balances diversity and parallelism.
problem Balancing diversity and parallelism in decoding for large language models.
method Arithmetic sampling framework compatible with various sampling variations.
result Improves estimation of expected BLEU score reward and reduces the gap with beam search.
Auto-sizing improves machine translation efficiency by 3.9 BLEU points with fewer parameters.
problem Optimizing neural network architecture and hyperparameters for low-resource machine translation is computationally expensive.
method Auto-sizing uses regularization to dynamically adjust network size during training.
result Auto-sizing improves BLEU scores by up to 3.9 points with one-third fewer parameters.
Differentiable sampling corrects alignment issues in neural machine translation.
problem Incorrect alignment of reference words and sampled output in scheduled sampling.
method Optimizes alignment probability based on model's soft alignment prediction.
result Improves BLEU score compared to maximum likelihood and scheduled sampling.
Generative model generates realistic text without reinforcement learning.
problem Training GANs for natural language processing is challenging due to discrete sequences.
method Autoencoder learns low-dimensional sentence representation, GAN generates vectors in this space.
result Model generates realistic text with competitive BLEU scores and human ratings.
Enhances persona-based conversation model for multi-turn dialogue.
problem Improving persona-based conversation models for multi-turn dialogue.
method Introduced additional input modality into hredGAN to capture external attributes.
result Persona hredGAN (phredGAN) outperforms existing models in multi-turn dialogue corpora. A new method samples sequences without replacement using Gumbel-Top-k trick.
problem Sampling sequences without replacement from large domains efficiently.
method Gumbel-Top-k trick applied to Stochastic Beam Search.
result Efficient sampling with linear model evaluations in k. Paper tackles zero-shot translation by encouraging consistent agreement in models.
problem Challenges of generalizing multilingual translation without parallel data.
method Reformulated as probabilistic inference, introduced consistent agreement-based training.
result Agreement-based learning improves zero-shot translation by 2-3 BLEU points.
MLMs score better than autoregressive models without finetuning.
problem Improving language model scores without fine-tuning.
method Compute pseudo-log-likelihood scores (PLLs) by masking tokens one by one.
result PLLs outperform GPT-2 scores in various NLP tasks.
This paper tackles rare word problem in low-resource language pairs using NMT.
problem Rare word problem in neural machine translation, especially for low-resource languages.
method Three solutions: enhanced source context, morphology learning, and wordnet synonyms.
result Significant improvements in BLEU scores (+1.0 points) on English-Vietnamese and Japanese-Vietnamese.
We present power low rank ensembles (PLRE), a flexible framework for n-gram language modeling where ensembles of low rank matrices and tensors are used to obtain smoothed probability estimates of words in context. Our method can be understood as a generalization of n-gram modeling to non-integer n, and includes standar…
Transforms web content for better visibility in AI-driven search engines.
problem Disruption of traditional SEO by generative AI search engines.
method Fine-tunes a BART-base transformer on synthetically generated training data.
result Significant improvements in ROUGE-L and BLEU scores, and substantial visibility gains in generative search responses.
Sequence to sequence learning has recently emerged as a new paradigm in supervised learning. To date, most of its applications focused on only one task and not much work explored this framework for multiple tasks. This paper examines three multi-task learning (MTL) settings for sequence to sequence models: (a) the onet…
We introduce Generative Neural Machine Translation (GNMT), a latent variable architecture which is designed to model the semantics of the source and target sentences. We modify an encoder-decoder translation model by adding a latent variable as a language agnostic representation which is encouraged to learn the meaning…
Efficiently improves non-autoregressive sequence models for better translation performance.
problem Heavy inference latency and inconsistent output sentences in non-autoregressive models.
method Incorporates a structured inference module with an efficient CRF approximation and dynamic transition technique.
result Significantly better translation performance (BLEU score 26.80) compared to previous non-autoregressive models.
Paper explores how text generation quality and diversity metrics relate to distribution fitting.
problem Unclear relation between text generation quality and diversity metrics and distribution fitting.
method Theoretical approach to prove a linear combination of quality and diversity metrics can be a divergence metric.
result CR/NRR proposed as a better substitute for BLEU/Self-BLEU metrics.
Transformer improves sequence generation with insertion and deletion phases.
problem Sequence generation challenges in machine translation.
method Insertion-Deletion Transformer with iterative insertion and deletion phases.
result Significant BLEU score improvement over insertion-only models.
We present an approach to interactive-predictive neural machine translation that attempts to reduce human effort from three directions: Firstly, instead of requiring humans to select, correct, or delete segments, we employ the idea of learning from human reinforcements in form of judgments on the quality of partial tra…
Transformers trained with three normalization changes outperform state-of-the-art on low-resource translation tasks.
problem Improving the training of Transformers, especially on low-resource datasets.
method Three normalization changes: PreNorm, ScaleNorm, and FixNorm.
result Significant improvements in BLEU scores on low-resource translation tasks.
We describe Sockeye (version 1.12), an open-source sequence-to-sequence toolkit for Neural Machine Translation (NMT). Sockeye is a production-ready framework for training and applying models as well as an experimental platform for researchers. Written in Python and built on MXNet, the toolkit offers scalable training a…
Our research extends the Bilingual Evaluation Understudy (BLEU) evaluation technique for statistical machine translation to make it more adjustable and robust. We intend to adapt it to resemble human evaluation more. We perform experiments to evaluate the performance of our technique against the primary existing evalua…
Improved neural language models using adversarial training.
problem Overfitting in large-scale neural language models.
method Adversarial training mechanism to introduce adversarial noise to output embedding layers.
result Improved perplexity scores and BLEU scores on various language tasks.
New model improves diversity in question generation over text.
problem Inadequate evaluation metrics for diverse questions.
method Proposed a variational encoder-decoder model and evaluation scheme.
result Improved diversity in generated questions without loss of quality.
Improved text generation using transferable rewards from related tasks.
problem Non-differentiable task-specific scores limit the use of policy gradient methods in text generation.
method Transferable Reward Learner that uses model-based rewards for sentence-level and phrase-level similarity.
result Improved performance on semantic evaluation measures in image captioning tasks.
Syntax-enhanced models boost machine translation and NLP performance.
problem Limited training data and complex models struggle in NLP tasks.
method Syntax information was explicitly fed into Transformer and BERT models.
result Syntax-infused models achieved significant BLEU improvements.
Transformer models outperform recurrent ones in modeling hierarchical data.
problem Modeling hierarchical structure in data.
method Introducing Multiresolution Transformer Networks leveraging self-attention.
result Multiresolution Transformer Networks significantly outperform state-of-the-art models on query suggestion datasets.
The variational autoencoder (VAE) imposes a probabilistic distribution (typically Gaussian) on the latent space and penalizes the Kullback--Leibler (KL) divergence between the posterior and prior. In NLP, VAEs are extremely difficult to train due to the problem of KL collapsing to zero. One has to implement various heu…
Paper aims to improve education online in South Africa using NMT for Setswana.
problem Inclusion of machine translation for low-resourced local languages in online education.
method Demonstrates state-of-the-art performance on English-to-Setswana translation using the Transformer architecture.
result Current NMT techniques show promise for African languages, beating previous methods by 5.33 BLEU points.
This study examines how the size and alignment of pretraining data affect the performance of large language models on downstream tasks.
problem Understanding how the size and alignment of pretraining data impact the performance of large language models on downstream tasks.
method Investigated the scaling behavior of large language models in a transfer learning setting, focusing on machine translation tasks.
result The size of the finetuning dataset and the distribution alignment between pretraining and downstream data significantly influence the scaling behavior of downstream performance.
K-TanH is a fast, efficient deep learning activation function.
problem Improving the speed and efficiency of deep learning models.
method K-TanH is a novel, highly accurate, low-precision integer approximation of TanH.
result K-TanH achieves up to 5x speedup over Intel SVML and is more efficient.
MARGE learns to reconstruct text by paraphrasing, achieving strong performance across multiple tasks.
problem Training sequence-to-sequence models with limited supervision.
method Unsupervised multi-lingual multi-document paraphrasing objective.
result Strong zero-shot performance on document translation and various tasks in multiple languages.
This paper evaluates LLMs for technical market analysis, finding GPT-4 Turbo and FinGPT outperform passive benchmarks.
problem Evaluating LLMs for technical market analysis in financial markets.
method Structured evaluation of five LLMs (GPT-4 Turbo, Claude 3 Opus, Gemini 1.5 Pro, Llama 3 70B, FinGPT) on four tasks: candlestick pattern recognition, directional signal generation, backtesting, and financial report comprehension.
result GPT-4 Turbo and FinGPT outperform passive benchmarks in simulated backtesting, with GPT-4 Turbo achieving the highest annualized return and Sharpe ratio.
We present a recurrent encoder-decoder deep neural network architecture that directly translates speech in one language into text in another. The model does not explicitly transcribe the speech into text in the source language, nor does it require supervision from the ground truth source language transcription during t…