Paper defines and evaluates attention mechanisms for sequence to sequence tasks.
problem Improving sequence to sequence mapping tasks.
method Mathematical definition and evaluation of attention mechanisms in sequence to sequence models.
result Models using greater attention perform better and converge faster.
Paper tackles embedding attributed sequences in unsupervised learning.
problem Mining tasks over attributed sequences with dependencies between sequences and attributes.
method Proposes a deep multimodal learning framework, NAS, for unsupervised learning of attributed sequences.
result NAS produces task-independent embeddings for various mining tasks on real-world datasets.
Study on how task sequence properties affect continual learning algorithms.
problem Understanding how task sequence properties influence continual learning algorithms.
method Proposes a new procedure using task space modeling and correlation analysis.
result Error rates are correlated to a task sequence's total complexity but not to sequential heterogeneity.
A new method optimizes neural sequence models for better task performance.
problem Training neural sequence models with maximum likelihood estimation ignores task losses.
method Maximum likelihood guided parameter search (MGS) in the parameter space.
result MGS optimizes sequence-level losses, reducing repetition and non-termination.
New model controls memory in seq2seq tasks, revealing learning regimes.
problem Understanding memory in seq2seq tasks using neural networks.
method Introducing a stochastic switching-Ornstein-Uhlenbeck (SSOU) model to control memory and a measure of non-Markovianity.
result Two learning regimes emerge from the interplay of time scales in the SSOU process.
Optimizes task sequencing in reinforcement learning to improve exploration and policy discovery.
problem Improving exploration and policy discovery in reinforcement learning.
method General optimization framework for task sequencing using different objective functions.
result Curriculum learning can improve initial performance, reduce suboptimal actions, and discover better policies.
GTI network learns linguistic features for multi-task sequence tagging.
problem Improving neural model performance on multi-task sequence tagging without explicit features.
method GTI network with neural gate modules to learn relations between tasks.
result GTI network outperforms baselines on chunking and NER tasks.
New method for learning evolving tasks with performance guarantees.
problem Learning tasks in a sequence with evolving similarity.
method Adaptable learning methodology with performance guarantees.
result Improved performance in multiple scenarios with reliable guarantees.
Hybrid model combines RNNs, encoders-decoders, and Transformers for sequence tasks.
problem Sequence labelling tasks
method Combination of bidirectional RNNs, encoder-decoder, and Transformer models
result Results are close to state-of-the-art and better for some tasks
Paper presents IMRCs for evolving tasks with forward and backward learning.
problem Incremental learning of evolving tasks with few samples per task.
method Incremental minimax risk classifiers (IMRCs) that exploit forward and backward learning.
result IMRCs provide significant performance improvement, especially with reduced sample sizes.
Sequences have become first class citizens in supervised learning thanks to the resurgence of recurrent neural networks. Many complex tasks that require mapping from or to a sequence of observations can now be formulated with the sequence-to-sequence (seq2seq) framework which employs the chain rule to efficiently repre…
Bidirectional sequence generation improves performance in conversational tasks.
problem Neural sequence generation typically considers only past tokens, limiting performance.
method Introduced placeholder tokens that can consider both past and future tokens in sequence generation.
result Bidirectional model outperforms competitive baselines on conversational tasks.
New insights into how encoder-decoder networks generate attention matrices.
problem Understanding how encoder-decoder networks use attention matrices.
method Decomposing hidden states into temporal and input-driven components.
result Attention matrices are formed based on task requirements, not architecture type.
Sequence to sequence learning has recently emerged as a new paradigm in supervised learning. To date, most of its applications focused on only one task and not much work explored this framework for multiple tasks. This paper examines three multi-task learning (MTL) settings for sequence to sequence models: (a) the onet…
A new deep learning system improves tagging accuracy on various tasks.
problem Improving tagging accuracy on diverse sequence tagging tasks.
method Combines DNN and DRL for augmented tagging, evaluated on ATIS and CoNLL-2003 datasets.
result Outperforms state-of-the-art models by 1.9% on ATIS and 1.4% on CoNLL-2003 datasets.
MTDS improves sequence generation adaptability via latent code control.
problem Lack of adaptability in sequence generation models like RNNs.
method Hierarchical multi-task dynamical systems (MTDS) with latent code control.
result MTDS enables style transfer, interpolation, and morphing in generated sequences.
We present a recurrent encoder-decoder deep neural network architecture that directly translates speech in one language into text in another. The model does not explicitly transcribe the speech into text in the source language, nor does it require supervision from the ground truth source language transcription during t…
This study evaluates the performances of an LSTM network for detecting and extracting the intent and content of com- mands for a financial chatbot. It presents two techniques, sequence to sequence learning and Multi-Task Learning, which might improve on the previous task.
Randomized positional encodings boost transformer performance on longer sequences.
problem Transformers struggle with generalizing to sequences of arbitrary length.
method Introduced randomized positional encodings that simulate longer sequences and randomly select positions.
result Randomized positional encodings increase test accuracy by 12.0% on average for sequences of unseen length.
Recurrent Neural Networks (RNNs) with attention mechanisms have obtained state-of-the-art results for many sequence processing tasks. Most of these models use a simple form of encoder with attention that looks over the entire sequence and assigns a weight to each token independently. We present a mechanism for focusing…
OptiGAN uses GAN and RL to optimize sequence generation for specific goals.
problem Challenging in sequence generation tasks to generate sequences with specific desired goals.
method Integrates GAN and RL to optimize desired goal scores using policy gradients.
result Achieves higher desired scores in text and real-valued sequence generation.
Sequence-to-Sequence (seq2seq) modeling has rapidly become an important general-purpose NLP tool that has proven effective for many text-generation and sequence-labeling tasks. Seq2seq builds on deep neural language modeling and inherits its remarkable accuracy in estimating local, next-word distributions. In this work…
Paper tackles counterfactual sentence detection and evaluation.
problem Detect and evaluate counterfactual sentences in natural language.
method Used a BERT base model for classification and a hybrid BERT Multi-Layer Perceptron for sequence identification. Introduced cascaded linear inputs to improve performance.
result Achieved an F1 score of 85.00% in Task 1 and 83.90% in Task 2.
The celebrated Sequence to Sequence learning (Seq2Seq) technique and its numerous variants achieve excellent performance on many tasks. However, many machine learning tasks have inputs naturally represented as graphs; existing Seq2Seq models face a significant challenge in achieving accurate conversion from graph form …
Unified model for sequence labeling and classification.
problem Efficiently perform multiple sequence labeling tasks.
method Generative framework with shared natural language output space.
result Significant improvements in few-shot and low-resource slot labeling.
We combine multi-task learning and semi-supervised learning by inducing a joint embedding space between disparate label spaces and learning transfer functions between label embeddings, enabling us to jointly leverage unlabelled data and auxiliary, annotated datasets. We evaluate our approach on a variety of sequence cl…
This paper compares log-likelihood and BLEU scores for sequence generation tasks.
problem The discrepancy between density estimation and sequence generation performance.
method Comparing several density estimators on five machine translation tasks.
result The correlation between log-likelihood and BLEU varies depending on model families.
The paper extends influence functions to sequence tagging tasks for better model interpretability.
problem Lack of interpretability methods for sequence tagging models.
method Define and compute influence of training instance segments on test segment predictions.
result The segment influence method tracks with true influence and identifies annotation errors.
Meta-learning improves event prediction from short sequences.
problem Predicting events from short sequences is challenging.
method Meta-learning approach using recurrent neural networks and monotonic neural networks.
result Meta-learning enhances long-term prediction performance.
Joint training model for TTS and VC tasks using Tacotron and WaveNet.
problem Training a shared model for text-to-speech and voice conversion.
method Extended Tacotron model with dual attention mechanism for shared tasks, WaveNet for waveform generation.
result Joint training of a shared model achieves both TTS and VC tasks efficiently.
Locally-contextual CRFs improve sequence labeling performance.
problem Improving sequence labeling with contextual embeddings.
method Locally-contextual nonlinear CRFs using deep neural networks.
result Consistently outperforms linear chain CRF and previous state of the art.
BigBird improves transformer performance on NLP tasks with longer sequences.
problem Quadratic dependency on sequence length in transformer models.
method Sparse attention mechanism reducing quadratic dependency to linear.
result Significant improvement in performance on NLP tasks like QA and summarization.
In several natural language tasks, labeled sequences are available in separate domains (say, languages), but the goal is to label sequences with mixed domain (such as code-switched text). Or, we may have available models for labeling whole passages (say, with sentiments), which we would like to exploit toward better po…
This research tackles uncertainty estimation in autoregressive structured prediction tasks.
problem Ensuring safety and robustness of AI systems through accurate uncertainty estimation.
method Develops a unified probabilistic ensemble-based framework for token-level and sequence-level uncertainty estimation.
result Provides baselines for error and out-of-domain detection on translation and speech recognition datasets.
Recurrent neural networks (RNNs) sequentially process data by updating their state with each new data point, and have long been the de facto choice for sequence modeling tasks. However, their inherently sequential computation makes them slow to train. Feed-forward and convolutional architectures have recently been show…
Algorithm finds novel policies for tasks using novelty reward.
problem Finding distinct policies for a given task.
method Creates a second reward function for novelty, updates policies to balance task and novelty rewards.
result Collection of distinct policies for a task.
Self-attentional models improve chatbot efficiency and performance.
problem Training efficient task-oriented dialogue generation systems.
method Applied self-attentional models to three datasets for chatbot training.
result Self-attentional models outperform recurrence-based models in efficiency and performance.
Hybrid framework prevents forgetting in continual learning.
problem Avoiding forgetting in learning new tasks without forgetting old ones.
method Hybrid continual learning framework combining architecture growth and experience replay.
result Hybrid approach effectively avoids forgetting across multiple tasks.
Improved autoencoder boosts sequence learning with less memory.
problem Training recurrent networks with high memory costs.
method Sparse Predictive Autoencoder (bRSM) with recurrent connections and boosting rule.
result Near optimal performance on stochastic sequence learning task.
PRAGMA models financial event sequences for various banking tasks.
problem Handling diverse financial data for multiple applications.
method Pre-training a Transformer model on a large banking event corpus with a self-supervised objective.
result PRAGMA achieves superior performance across multiple financial domains from raw event sequences.
Traditional pairwise sequence alignment is based on matching individual samples from two sequences, under time monotonicity constraints. However, in many application settings matching subsequences (segments) instead of individual samples may bring in additional robustness to noise or local non-causal perturbations. Thi…
Probabilistic models use quantum circuits for sequence tasks.
problem Sequence modeling with classical datasets.
method Gradient-free algorithm based on matrix product states.
result Circuit-based models provide a useful inductive bias for classical datasets.
A new model improves recurrent neural networks' ability to memorize long sequences.
problem Improving recurrent neural networks' ability to memorize long sequences and extract task-relevant features.
method Proposes a Linear Memory Network with an encoding-based memorization component and a specialized training algorithm.
result Improves the final performance of recurrent neural networks when memorizing long sequences is necessary.
State-of-the-art sequence labeling systems traditionally require large amounts of task-specific knowledge in the form of hand-crafted features and data pre-processing. In this paper, we introduce a novel neutral network architecture that benefits from both word- and character-level representations automatically, by usi…
Bandit structured prediction describes a stochastic optimization framework where learning is performed from partial feedback. This feedback is received in the form of a task loss evaluation to a predicted output structure, without having access to gold standard structures. We advance this framework by lifting linear ba…
Learning to remember long sequences remains a challenging task for recurrent neural networks. Register memory and attention mechanisms were both proposed to resolve the issue with either high computational cost to retain memory differentiability, or by discounting the RNN representation learning towards encoding shorte…
Deep network predicts action sequences for complex tasks from a scene image.
problem Scalable task and motion planning from initial scene images.
method Deep convolutional recurrent neural network that predicts action sequences.
result Predicts promising action sequences, reducing motion planning problems.
We describe a fully data driven model that learns to perform a retrosynthetic reaction prediction task, which is treated as a sequence-to-sequence mapping problem. The end-to-end trained model has an encoder-decoder architecture that consists of two recurrent neural networks, which has previously shown great success in…