A novel sequence-to-sequence model predicts missing sensor data.
problem Missing sensor data in sequences.
method Formulated a novel sequence-to-sequence model using forward and backward RNNs.
result The model produces the lowest errors in 12% more cases than the current state-of-the-art.
The paper develops a convex parameterization for robust RNNs ensuring stability and robustness.
problem Lack of stability and robustness guarantees in RNNs for sequence-to-sequence mapping applications.
method Formulated convex sets of RNNs with stability and robustness guarantees using incremental quadratic constraints.
result The proposed model structure ensures global exponential stability and bounds on incremental ℓ2 gain. Recurrent Neural Networks (RNNs) achieve state-of-the-art results in many sequence-to-sequence modeling tasks. However, RNNs are difficult to train and tend to suffer from overfitting. Motivated by the Data Processing Inequality (DPI), we formulate the multi-layered network as a Markov chain, introducing a training met…
Model learns accent patterns from small datasets.
problem Automatic speech recognition struggles with rare accents.
method Automatically retrieves phonological generalizations from a small dataset.
result Model generates a million phonological variations of words.
STConvS2S improves weather forecasting using only convolutional layers.
problem Predicting future weather conditions more accurately.
method Proposes a deep learning architecture combining spatiotemporal convolutional layers.
result Outperforms state-of-the-art architectures for forecasting tasks.
Learning latent representations from long text sequences is an important first step in many natural language processing applications. Recurrent Neural Networks (RNNs) have become a cornerstone for this challenging task. However, the quality of sentences during RNN-based decoding (reconstruction) decreases with the leng…
Combines CNN and RNN for hierarchical image classification.
problem Hierarchical relations between image categories are not captured by flat classifiers.
method Uses a CNN for feature extraction and an RNN for capturing hierarchical class relations. Incorporates residual learning.
result Hierarchical networks outperform state-of-the-art CNNs on a real-world dataset.
Reversible RNNs reduce memory usage in training without sacrificing performance.
problem Memory-intensive training of RNNs limits model flexibility.
method Developed a scheme for perfect reversible RNNs with forgetting, reducing memory by 10-15x.
result Achieved comparable performance to traditional models with reduced activation memory cost.
Paper explores EEG-based speech recognition using transformers, showing faster training and better performance for smaller vocabularies.
problem Continuous speech recognition using EEG features.
method Transformer-based ASR model compared to RNN-based models.
result Transformer models perform better for smaller vocabularies but RNN models outperform them for larger vocabularies.
SeqSleepNet tackles automatic sleep staging as a sequence-to-sequence problem.
problem Automatic sleep staging as a sequence-to-sequence classification problem.
method End-to-end hierarchical recurrent neural network (SeqSleepNet) with filterbank and attention-based recurrent layers.
result SeqSleepNet achieves high accuracy (87.1% overall accuracy, 83.3% macro F1-score, 0.815 Cohen's kappa) on a publicly available dataset.
Much combinatorial optimisation problems constitute a non-polynomial (NP) hard optimisation problem, i.e., they can not be solved in polynomial time. One such problem is finding the shortest route between two nodes on a graph. Meta-heuristic algorithms such as A∗ along with mixed-integer programming (MIP) methods …
New method embeds RNN Seq2Seq models to visualize spatiotemporal data.
problem Visualizing and interpreting spatiotemporal data in sequence prediction tasks.
method Embedding approach to visualize and interpret RNN Seq2Seq model representations.
result Embedding space projections of RNN Seq2Seq models capture spatiotemporal dynamics.
In this work, we first analyze the memory behavior in three recurrent neural networks (RNN) cells; namely, the simple RNN (SRN), the long short-term memory (LSTM) and the gated recurrent unit (GRU), where the memory is defined as a function that maps previous elements in a sequence to the current output. Our study show…
Transformer networks approximate Hölder and Sobolev functions with fixed-depth networks.
problem Nonparametric regression with dependent observations.
method Established novel upper bounds for Transformer networks approximating Hölder and Sobolev functions under various β-mixing data assumptions. result Explicit convergence rates for nonparametric regression problems under β-mixing data assumptions. Paper defines and evaluates attention mechanisms for sequence to sequence tasks.
problem Improving sequence to sequence mapping tasks.
method Mathematical definition and evaluation of attention mechanisms in sequence to sequence models.
result Models using greater attention perform better and converge faster.
Sequence learning improves query expansion in information retrieval.
problem Improving query expansion in information retrieval systems.
method Used sequence to sequence algorithms to extract keywords from sentence embeddings and trained a neural network on open datasets.
result Sequence to sequence models can capture complex query expansion relations in word embeddings.
Transformers can approximate any sequence-to-sequence function, surprising given their complexity.
problem Understanding the expressive power of Transformer models for sequence-to-sequence functions.
method Established that Transformers are universal approximators of continuous permutation equivariant sequence-to-sequence functions with compact support, and extended this to arbitrary functions using positional encodings.
result Transformers are universal approximators of arbitrary continuous sequence-to-sequence functions on a compact domain.
This research explores various sampling methods and probability distributions for hard alignment in sequence-to-sequence TTS synthesis.
problem Improving alignment accuracy in sequence-to-sequence text-to-speech synthesis.
method Investigated various sampling methods (greedy, beam, random) and probability distributions (Bernoulli, Concrete) for hard alignment.
result Deterministic search is more preferable than stochastic search for natural alignment transition.
The availability of large amounts of time series data, paired with the performance of deep-learning algorithms on a broad class of problems, has recently led to significant interest in the use of sequence-to-sequence models for time series forecasting. We provide the first theoretical analysis of this time series forec…
Self-regulation improves sequence-to-sequence learning by choosing feedback types.
problem Different types of feedback have varying costs and effects on learning.
method Self-regulation strategies decide when to ask for different types of feedback.
result Self-regulator discovers optimal cost-quality trade-off by mixing feedback types.
SequenceR uses seq-to-seq learning to fix bugs in code.
problem Fixing bugs in code using machine learning.
method Sequence-to-sequence learning with copy mechanism, trained on curated code samples.
result SequenceR can perfectly predict and find correct patches for bugs.
A new loss function speeds up sequence-to-sequence models for continuous outputs.
problem Slow and memory-intensive softmax layer limits vocabulary size and translation quality.
method Proposes a probabilistic loss and continuous embedding layer training/inference procedure.
result Models achieve up to 2.5x speed-up in training time with similar translation quality.
Investigates neural TTS systems for Japanese and English.
problem Improving neural TTS systems for high-quality speech synthesis.
method Comparative study of neural sequence-to-sequence TTS vs. DNN pipeline TTS, varying model architecture, parameter size, and language.
result A neural sequence-to-sequence TTS system requires sufficient model parameters and a powerful encoder for high-quality speech synthesis.
Seq2Slate models for ranking by predicting appealing item slates.
problem Ranking items in a slate to present a user appealing set of items.
method Sequence-to-sequence model that predicts the next best item to place on the slate.
result The model captures complex dependencies between items and learns from weak supervision.
We present a recurrent encoder-decoder deep neural network architecture that directly translates speech in one language into text in another. The model does not explicitly transcribe the speech into text in the source language, nor does it require supervision from the ground truth source language transcription during t…
Seq2seq models predict complex multi-physics systems' time evolution.
problem Predicting the time-evolution of complex multi-physics systems.
method Sequence-to-sequence models applied to multi-physics simulations.
result Seq2seq models accurately emulate complex systems and predict their evolution.
Streaming ASR with transformer achieves low WER.
problem Real-time ASR with speech recognition.
method Time-restricted self-attention and triggered attention mechanisms.
result 2.8% and 7.2% WER for LibriSpeech test data.
Generative models have long been the dominant approach for speech recognition. The success of these models however relies on the use of sophisticated recipes and complicated machinery that is not easily accessible to non-practitioners. Recent innovations in Deep Learning have given rise to an alternative - discriminati…
Convolutional model disaggregates electricity consumption data.
problem Disaggregating aggregate electricity consumption data into individual appliance usage.
method Gated linear unit convolutional layers and residual blocks refine neural network output. Partially overlapped sequences are averaged for final output.
result The proposed model outperforms existing models in disaggregating various appliance usage.
Model predicts variable length time series outputs.
problem Fixed-length prediction in recurrent networks.
method Sequence to sequence model with dynamic output length.
result Model balances prediction accuracy and length.
Study improves financial chatbot command understanding.
problem Improving chatbot command understanding for financial contexts.
method Sequence to sequence learning and Multi-Task Learning techniques.
result Enhanced performance in intent and content extraction.
Study improves voice conversion model with Mel-spectrogram augmentation.
problem Insufficient speech pairs data for training sequence-to-sequence voice conversion models.
method Experimented with Mel-spectrogram augmentation using SpecAugment policies and proposed new augmentation policies.
result Time axis warping policies showed better performance in training the voice conversion model.
CLVSA predicts financial market trends using LSTM and attention mechanisms.
problem Predicting trends in financial markets due to complex interactions.
method Hybrid model combining LSTM, sequence-to-sequence, attention, and convolutional LSTM.
result CLVSA outperforms basic models in predicting financial market trends.
Sequence-to-Sequence (seq2seq) modeling has rapidly become an important general-purpose NLP tool that has proven effective for many text-generation and sequence-labeling tasks. Seq2seq builds on deep neural language modeling and inherits its remarkable accuracy in estimating local, next-word distributions. In this work…
Delayed-RNN approximates stacked and bidirectional RNNs.
problem Improving RNN expressiveness and representational capacity.
method Weight-constrained delayed-RNN, equivalent to stacked-RNNs, with partial acausality.
result Delayed-RNN can approximate stacked and bidirectional RNNs, outperforming them in some tasks.
The study forecasts, reconstructs, and selects features of ocean waves using neural networks.
problem Forecasting, reconstructing, and feature selection of ocean waves.
method Recurrent and sequence-to-sequence neural networks, Bayesian hyperparameter optimization, Elastic Net method.
result Proposed methods outperform alternatives in significant wave height reconstruction.
Study reveals how machine translation models can leak data privacy.
problem Detecting if a specific data sample was used to train machine translation models.
method Defined membership inference problem for sequence-to-sequence models, tested on machine translation models.
result Machine translation models can leak private information about training data.
We describe a fully data driven model that learns to perform a retrosynthetic reaction prediction task, which is treated as a sequence-to-sequence mapping problem. The end-to-end trained model has an encoder-decoder architecture that consists of two recurrent neural networks, which has previously shown great success in…
Deep learning models predict solar irradiance for short-term forecasts.
problem Accurate prediction of solar irradiance for renewable energy integration.
method Sequence-to-sequence LSTM models for GHI forecasting, incorporating spatial-temporal features.
result LSTM models outperform traditional techniques in short-term GHI forecasting.
Lingvo is a flexible framework for deep learning sequence models.
problem Sequence-to-sequence modeling challenges in deep learning.
method Modular, scalable, and customizable Tensorflow framework.
result Supports distributed training and quantized inference.
Sequence-to-sequence models predict resource usage for co-scheduled jobs in data centers.
problem Challenges in co-scheduling jobs due to resource interference and inefficiencies.
method Sequence-to-sequence models based on recurrent neural networks for workload interference prediction.
result Models accurately forecast resource usage trends from job profiles, improving scheduling decisions.
Improved speech recognition with language model integration in sequence-to-sequence models.
problem Improving word error rate in speech recognition models.
method Log-linear combination of acoustic and language models with per-token renormalization.
result The proposed method shows good improvements over standard model combination on Librispeech system.
GAN-AE generates synthetic data for imbalanced sequence classification.
problem Imbalanced sequence classification in medical devices.
method Autoencoder and GAN architecture for generating synthetic data.
result GAN-AE outperforms other synthetic data generation methods.
Softmax attention approximates complex functions and subsumes many known universal approximators.
problem Universal approximation of continuous sequence-to-sequence functions.
method Interpolation-based analysis of attention's internal mechanism, showing its ability to approximate ReLU functions.
result Softmax attention is a universal approximator for continuous sequence-to-sequence functions.
Attention forcing improves sequence-to-sequence model training stability.
problem Training auto-regressive sequence-to-sequence models with attention mechanism is challenging.
method Attention forcing guides the model with generated output history and reference attention.
result Attention forcing trains models to recover from mistakes without requiring a schedule or classifier.
Recurrent Neural Networks (RNNs) are powerful sequence modeling tools. However, when dealing with high dimensional inputs, the training of RNNs becomes computational expensive due to the large number of model parameters. This hinders RNNs from solving many important computer vision tasks, such as Action Recognition in …
PF-RNNs use particle filtering to model uncertainty in RNNs for better sequential data prediction.
problem Highly variable and noisy sequential data.
method PF-RNNs maintain a latent state distribution as a set of particles, updating with Bayes rule.
result PF-RNNs outperform standard RNNs on various sequence prediction tasks.
BART pretrains sequence-to-sequence models by corrupting text and reconstructing it.
problem Improving natural language generation, translation, and comprehension.
method BART uses a denoising autoencoder trained on a Transformer architecture with various noising techniques.
result BART achieves state-of-the-art performance on various NLP tasks with minimal training resources.