Paper defines and evaluates attention mechanisms for sequence to sequence tasks.
problem Improving sequence to sequence mapping tasks.
method Mathematical definition and evaluation of attention mechanisms in sequence to sequence models.
result Models using greater attention perform better and converge faster.
Paper tackles embedding attributed sequences in unsupervised learning.
problem Mining tasks over attributed sequences with dependencies between sequences and attributes.
method Proposes a deep multimodal learning framework, NAS, for unsupervised learning of attributed sequences.
result NAS produces task-independent embeddings for various mining tasks on real-world datasets.
Study on how task sequence properties affect continual learning algorithms.
problem Understanding how task sequence properties influence continual learning algorithms.
method Proposes a new procedure using task space modeling and correlation analysis.
result Error rates are correlated to a task sequence's total complexity but not to sequential heterogeneity.
A new method optimizes neural sequence models for better task performance.
problem Training neural sequence models with maximum likelihood estimation ignores task losses.
method Maximum likelihood guided parameter search (MGS) in the parameter space.
result MGS optimizes sequence-level losses, reducing repetition and non-termination.
Study improves financial chatbot command understanding.
problem Improving chatbot command understanding for financial contexts.
method Sequence to sequence learning and Multi-Task Learning techniques.
result Enhanced performance in intent and content extraction.
New model controls memory in seq2seq tasks, revealing learning regimes.
problem Understanding memory in seq2seq tasks using neural networks.
method Introducing a stochastic switching-Ornstein-Uhlenbeck (SSOU) model to control memory and a measure of non-Markovianity.
result Two learning regimes emerge from the interplay of time scales in the SSOU process.
Optimizes task sequencing in reinforcement learning to improve exploration and policy discovery.
problem Improving exploration and policy discovery in reinforcement learning.
method General optimization framework for task sequencing using different objective functions.
result Curriculum learning can improve initial performance, reduce suboptimal actions, and discover better policies.
GTI network learns linguistic features for multi-task sequence tagging.
problem Improving neural model performance on multi-task sequence tagging without explicit features.
method GTI network with neural gate modules to learn relations between tasks.
result GTI network outperforms baselines on chunking and NER tasks.
New method for learning evolving tasks with performance guarantees.
problem Learning tasks in a sequence with evolving similarity.
method Adaptable learning methodology with performance guarantees.
result Improved performance in multiple scenarios with reliable guarantees.
Hybrid model combines RNNs, encoders-decoders, and Transformers for sequence tasks.
problem Sequence labelling tasks
method Combination of bidirectional RNNs, encoder-decoder, and Transformer models
result Results are close to state-of-the-art and better for some tasks
Focused RNNs improve sequence processing tasks with attention mechanisms.
problem Sequence processing tasks benefit from more focused attention mechanisms.
method Focused hierarchical RNNs with discrete gating mechanisms that selectively attend to relevant parts of the input.
result Focused RNNs outperform existing methods on various sequence processing tasks, including MS MARCO and SearchQA.
GIRNet tackles multi-tasking with mixed domain sequences, improving sentiment and tagging tasks.
problem Labeling sequences with mixed domain data and position-specific inference.
method Unified position-sensitive multi-task RNN architecture with gated state sequences from auxiliary data.
result GIRNet achieves new state-of-the-art performance in sentiment classification, POS tagging, and target position-sensitive annotation.
Paper presents IMRCs for evolving tasks with forward and backward learning.
problem Incremental learning of evolving tasks with few samples per task.
method Incremental minimax risk classifiers (IMRCs) that exploit forward and backward learning.
result IMRCs provide significant performance improvement, especially with reduced sample sizes.
Sequences have become first class citizens in supervised learning thanks to the resurgence of recurrent neural networks. Many complex tasks that require mapping from or to a sequence of observations can now be formulated with the sequence-to-sequence (seq2seq) framework which employs the chain rule to efficiently repre…
Bidirectional sequence generation improves performance in conversational tasks.
problem Neural sequence generation typically considers only past tokens, limiting performance.
method Introduced placeholder tokens that can consider both past and future tokens in sequence generation.
result Bidirectional model outperforms competitive baselines on conversational tasks.
New insights into how encoder-decoder networks generate attention matrices.
problem Understanding how encoder-decoder networks use attention matrices.
method Decomposing hidden states into temporal and input-driven components.
result Attention matrices are formed based on task requirements, not architecture type.
Sequence to sequence learning has recently emerged as a new paradigm in supervised learning. To date, most of its applications focused on only one task and not much work explored this framework for multiple tasks. This paper examines three multi-task learning (MTL) settings for sequence to sequence models: (a) the onet…
A new deep learning system improves tagging accuracy on various tasks.
problem Improving tagging accuracy on diverse sequence tagging tasks.
method Combines DNN and DRL for augmented tagging, evaluated on ATIS and CoNLL-2003 datasets.
result Outperforms state-of-the-art models by 1.9% on ATIS and 1.4% on CoNLL-2003 datasets.
MTDS improves sequence generation adaptability via latent code control.
problem Lack of adaptability in sequence generation models like RNNs.
method Hierarchical multi-task dynamical systems (MTDS) with latent code control.
result MTDS enables style transfer, interpolation, and morphing in generated sequences.
Paper uses PPO and PPO-dynamic for sequence generation tasks, improving stability and performance.
problem Intractable backpropagation issue in sequence generation tasks.
method Replaces policy gradient with PPO and proposes a dynamic approach for PPO (PPO-dynamic).
result PPO and PPO-dynamic outperform policy gradient in sequence generation tasks.
We present a recurrent encoder-decoder deep neural network architecture that directly translates speech in one language into text in another. The model does not explicitly transcribe the speech into text in the source language, nor does it require supervision from the ground truth source language transcription during t…
Randomized positional encodings boost transformer performance on longer sequences.
problem Transformers struggle with generalizing to sequences of arbitrary length.
method Introduced randomized positional encodings that simulate longer sequences and randomly select positions.
result Randomized positional encodings increase test accuracy by 12.0% on average for sequences of unseen length.
Universal Transformer improves sequence modeling accuracy.
problem Feed-forward sequence models fail on simple tasks.
method Parallel-in-time self-attentive recurrent model combining feed-forward and RNN properties.
result UTs outperform standard Transformers on various tasks.
OptiGAN uses GAN and RL to optimize sequence generation for specific goals.
problem Challenging in sequence generation tasks to generate sequences with specific desired goals.
method Integrates GAN and RL to optimize desired goal scores using policy gradients.
result Achieves higher desired scores in text and real-valued sequence generation.
Sequence-to-Sequence (seq2seq) modeling has rapidly become an important general-purpose NLP tool that has proven effective for many text-generation and sequence-labeling tasks. Seq2seq builds on deep neural language modeling and inherits its remarkable accuracy in estimating local, next-word distributions. In this work…
Paper tackles counterfactual sentence detection and evaluation.
problem Detect and evaluate counterfactual sentences in natural language.
method Used a BERT base model for classification and a hybrid BERT Multi-Layer Perceptron for sequence identification. Introduced cascaded linear inputs to improve performance.
result Achieved an F1 score of 85.00% in Task 1 and 83.90% in Task 2.
The celebrated Sequence to Sequence learning (Seq2Seq) technique and its numerous variants achieve excellent performance on many tasks. However, many machine learning tasks have inputs naturally represented as graphs; existing Seq2Seq models face a significant challenge in achieving accurate conversion from graph form …
Unified model for sequence labeling and classification.
problem Efficiently perform multiple sequence labeling tasks.
method Generative framework with shared natural language output space.
result Significant improvements in few-shot and low-resource slot labeling.
This paper compares log-likelihood and BLEU scores for sequence generation tasks.
problem The discrepancy between density estimation and sequence generation performance.
method Comparing several density estimators on five machine translation tasks.
result The correlation between log-likelihood and BLEU varies depending on model families.
We combine multi-task learning and semi-supervised learning by inducing a joint embedding space between disparate label spaces and learning transfer functions between label embeddings, enabling us to jointly leverage unlabelled data and auxiliary, annotated datasets. We evaluate our approach on a variety of sequence cl…
Novel complex RNN improves stability and performance in sequence tasks.
problem Lack of complex representations in deep learning for sequence tasks.
method Developed a complex gated recurrent cell combining complex-valued and norm-preserving state transitions with a gating mechanism.
result Improves stability and convergence properties, performs competitively on various tasks.
Meta-learning improves event prediction from short sequences.
problem Predicting events from short sequences is challenging.
method Meta-learning approach using recurrent neural networks and monotonic neural networks.
result Meta-learning enhances long-term prediction performance.
The paper extends influence functions to sequence tagging tasks for better model interpretability.
problem Lack of interpretability methods for sequence tagging models.
method Define and compute influence of training instance segments on test segment predictions.
result The segment influence method tracks with true influence and identifies annotation errors.
Joint training model for TTS and VC tasks using Tacotron and WaveNet.
problem Training a shared model for text-to-speech and voice conversion.
method Extended Tacotron model with dual attention mechanism for shared tasks, WaveNet for waveform generation.
result Joint training of a shared model achieves both TTS and VC tasks efficiently.
Locally-contextual CRFs improve sequence labeling performance.
problem Improving sequence labeling with contextual embeddings.
method Locally-contextual nonlinear CRFs using deep neural networks.
result Consistently outperforms linear chain CRF and previous state of the art.
BigBird improves transformer performance on NLP tasks with longer sequences.
problem Quadratic dependency on sequence length in transformer models.
method Sparse attention mechanism reducing quadratic dependency to linear.
result Significant improvement in performance on NLP tasks like QA and summarization.
This research tackles uncertainty estimation in autoregressive structured prediction tasks.
problem Ensuring safety and robustness of AI systems through accurate uncertainty estimation.
method Develops a unified probabilistic ensemble-based framework for token-level and sequence-level uncertainty estimation.
result Provides baselines for error and out-of-domain detection on translation and speech recognition datasets.
IGLOO slices features space to process long sequences efficiently.
problem Efficiency in processing long sequences with neural networks.
method Slicing features space into non-local patches and using their relationships to build sequence representations.
result IGLOO can handle dependencies up to 20,000 steps efficiently.
Neural model improves text normalization for non-English languages.
problem Improving text normalization in non-English languages with limited data.
method Sequence-to-sequence model with character and word embeddings, using pre-trained word embeddings with subword information.
result Achieved state-of-the-art F1 score on Arabic language correction dataset.
Algorithm finds novel policies for tasks using novelty reward.
problem Finding distinct policies for a given task.
method Creates a second reward function for novelty, updates policies to balance task and novelty rewards.
result Collection of distinct policies for a task.
Self-attentional models improve chatbot efficiency and performance.
problem Training efficient task-oriented dialogue generation systems.
method Applied self-attentional models to three datasets for chatbot training.
result Self-attentional models outperform recurrence-based models in efficiency and performance.
The paper improves part-of-speech tagging with multi-task learning and character-level word representations.
problem Improving part-of-speech tagging accuracy.
method Developed a new character-level word representation using feedforward neural network, pretraining with existing word vectors, and an additional prediction of neighbour labels as an auxiliary loss.
result The methods significantly improved POS tagging performance on English and Russian languages.
Improved autoencoder boosts sequence learning with less memory.
problem Training recurrent networks with high memory costs.
method Sparse Predictive Autoencoder (bRSM) with recurrent connections and boosting rule.
result Near optimal performance on stochastic sequence learning task.
Hybrid framework prevents forgetting in continual learning.
problem Avoiding forgetting in learning new tasks without forgetting old ones.
method Hybrid continual learning framework combining architecture growth and experience replay.
result Hybrid approach effectively avoids forgetting across multiple tasks.
PRAGMA models financial event sequences for various banking tasks.
problem Handling diverse financial data for multiple applications.
method Pre-training a Transformer model on a large banking event corpus with a self-supervised objective.
result PRAGMA achieves superior performance across multiple financial domains from raw event sequences.
Traditional pairwise sequence alignment is based on matching individual samples from two sequences, under time monotonicity constraints. However, in many application settings matching subsequences (segments) instead of individual samples may bring in additional robustness to noise or local non-causal perturbations. Thi…
Probabilistic models use quantum circuits for sequence tasks.
problem Sequence modeling with classical datasets.
method Gradient-free algorithm based on matrix product states.
result Circuit-based models provide a useful inductive bias for classical datasets.
A new model improves recurrent neural networks' ability to memorize long sequences.
problem Improving recurrent neural networks' ability to memorize long sequences and extract task-relevant features.
method Proposes a Linear Memory Network with an encoding-based memorization component and a specialized training algorithm.
result Improves the final performance of recurrent neural networks when memorizing long sequences is necessary.