Using sequence to sequence algorithms for query expansion has not been explored yet in Information Retrieval literature nor in Question-Answering's. We tried to fill this gap in the literature with a custom Query Expansion engine trained and tested on open datasets. Starting from open datasets, we built a Query Expansi…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,786 papers · 148 categories
Trend · papers per month
48 results for “sequence-to-sequence networks”
We present a recurrent encoder-decoder deep neural network architecture that directly translates speech in one language into text in another. The model does not explicitly transcribe the speech into text in the source language, nor does it require supervision from the ground truth source language transcription during t…
CLVSA predicts financial market trends using LSTM and attention mechanisms.
problem Predicting trends in financial markets due to complex interactions.
method Hybrid model combining LSTM, sequence-to-sequence, attention, and convolutional LSTM.
result CLVSA outperforms basic models in predicting financial market trends.
New insights into how encoder-decoder networks generate attention matrices.
problem Understanding how encoder-decoder networks use attention matrices.
method Decomposing hidden states into temporal and input-driven components.
result Attention matrices are formed based on task requirements, not architecture type.
k-Nearest Neighbors is one of the most fundamental but effective classification models. In this paper, we propose two families of models built on a sequence to sequence model and a memory network model to mimic the k-Nearest Neighbors model, which generate a sequence of labels, a sequence of out-of-sample feature vecto…
The paper develops a convex parameterization for robust RNNs ensuring stability and robustness.
problem Lack of stability and robustness guarantees in RNNs for sequence-to-sequence mapping applications.
method Formulated convex sets of RNNs with stability and robustness guarantees using incremental quadratic constraints.
result The proposed model structure ensures global exponential stability and bounds on incremental gain.
Recurrent neural networks and sequence to sequence models require a predetermined length for prediction output length. Our model addresses this by allowing the network to predict a variable length output in inference. A new loss function with a tailored gradient computation is developed that trades off prediction accur…
This article explores the concepts of ocean wave multivariate multistep forecasting, reconstruction and feature selection. We introduce recurrent neural network frameworks, integrated with Bayesian hyperparameter optimization and Elastic Net methods. We consider both short- and long-term forecasts and reconstruction, f…
A convolutional sequence to sequence non-intrusive load monitoring model is proposed in this paper. Gated linear unit convolutional layers are used to extract information from the sequences of aggregate electricity consumption. Residual blocks are also introduced to refine the output of the neural network. The partiall…
Sequence-to-sequence models predict resource usage for co-scheduled jobs in data centers.
problem Challenges in co-scheduling jobs due to resource interference and inefficiencies.
method Sequence-to-sequence models based on recurrent neural networks for workload interference prediction.
result Models accurately forecast resource usage trends from job profiles, improving scheduling decisions.
Investigates neural TTS systems for Japanese and English.
problem Improving neural TTS systems for high-quality speech synthesis.
method Comparative study of neural sequence-to-sequence TTS vs. DNN pipeline TTS, varying model architecture, parameter size, and language.
result A neural sequence-to-sequence TTS system requires sufficient model parameters and a powerful encoder for high-quality speech synthesis.
Variational Bayesian Sequence-to-Sequence Networks for Memory-Efficient Sign Language Translationstat.ML
A new memory-efficient sign language translation model reduces weight usage.
problem Memory constraints in real-time sign language translation.
method Variational Bayesian sequence-to-sequence network with Gaussian posterior and Indian Buffet Process prior.
result The proposed model achieves substantial weight compression without compromising performance.
New model controls memory in seq2seq tasks, revealing learning regimes.
problem Understanding memory in seq2seq tasks using neural networks.
method Introducing a stochastic switching-Ornstein-Uhlenbeck (SSOU) model to control memory and a measure of non-Markovianity.
result Two learning regimes emerge from the interplay of time scales in the SSOU process.
We describe a fully data driven model that learns to perform a retrosynthetic reaction prediction task, which is treated as a sequence-to-sequence mapping problem. The end-to-end trained model has an encoder-decoder architecture that consists of two recurrent neural networks, which has previously shown great success in…
Framework for renewable energy forecasting and feature engineering.
problem Forecasting and feature extraction for multivariate processes in renewable energy.
method Derivative-free optimization, ensemble of sequence-to-sequence networks, additive resampling, Bootstrap aggregating.
result The proposed method outperforms other machine learning techniques in long-term forecasts and feature selection.
A novel sequence-to-sequence model predicts missing sensor data.
problem Missing sensor data in sequences.
method Formulated a novel sequence-to-sequence model using forward and backward RNNs.
result The model produces the lowest errors in 12% more cases than the current state-of-the-art.
The Fisher information metric is an important foundation of information geometry, wherein it allows us to approximate the local geometry of a probability distribution. Recurrent neural networks such as the Sequence-to-Sequence (Seq2Seq) networks that have lately been used to yield state-of-the-art performance on speech…
Softmax attention approximates complex functions and subsumes many known universal approximators.
problem Universal approximation of continuous sequence-to-sequence functions.
method Interpolation-based analysis of attention's internal mechanism, showing its ability to approximate ReLU functions.
result Softmax attention is a universal approximator for continuous sequence-to-sequence functions.
In recent times, sequence-to-sequence (seq2seq) models have gained a lot of popularity and provide state-of-the-art performance in a wide variety of tasks such as machine translation, headline generation, text summarization, speech to text conversion, and image caption generation. The underlying framework for all these…
Bandit structured prediction describes a stochastic optimization framework where learning is performed from partial feedback. This feedback is received in the form of a task loss evaluation to a predicted output structure, without having access to gold standard structures. We advance this framework by lifting linear ba…
Transformers can approximate any sequence-to-sequence function, surprising given their complexity.
problem Understanding the expressive power of Transformer models for sequence-to-sequence functions.
method Established that Transformers are universal approximators of continuous permutation equivariant sequence-to-sequence functions with compact support, and extended this to arbitrary functions using positional encodings.
result Transformers are universal approximators of arbitrary continuous sequence-to-sequence functions on a compact domain.
This research explores various sampling methods and probability distributions for hard alignment in sequence-to-sequence TTS synthesis.
problem Improving alignment accuracy in sequence-to-sequence text-to-speech synthesis.
method Investigated various sampling methods (greedy, beam, random) and probability distributions (Bernoulli, Concrete) for hard alignment.
result Deterministic search is more preferable than stochastic search for natural alignment transition.
The recently proposed Sequence-to-Sequence (seq2seq) framework advocates replacing complex data processing pipelines, such as an entire automatic speech recognition system, with a single neural network trained in an end-to-end fashion. In this contribution, we analyse an attention-based seq2seq speech recognition syste…
The availability of large amounts of time series data, paired with the performance of deep-learning algorithms on a broad class of problems, has recently led to significant interest in the use of sequence-to-sequence models for time series forecasting. We provide the first theoretical analysis of this time series forec…
Neural attention (NA) has become a key component of sequence-to-sequence models that yield state-of-the-art performance in as hard tasks as abstractive document summarization (ADS) and video captioning (VC). NA mechanisms perform inference of context vectors; these constitute weighted sums of deterministic input sequen…
Multistep traffic forecasting on road networks is a crucial task in successful intelligent transportation system applications. To capture the complex non-stationary temporal dynamics and spatial dependency in multistep traffic-condition prediction, we propose a novel deep learning framework named attention graph convol…
Utilizing Character and Word Embeddings for Text Normalization with Sequence-to-Sequence Modelscs.CL
Text normalization is an important enabling technology for several NLP tasks. Recently, neural-network-based approaches have outperformed well-established models in this task. However, in languages other than English, there has been little exploration in this direction. Both the scarcity of annotated data and the compl…
The energy output a photo voltaic(PV) panel is a function of solar irradiation and weather parameters like temperature and wind speed etc. A general measure for solar irradiation called Global Horizontal Irradiance (GHI), customarily reported in Watt/meter, is a generic indicator for this intermittent energy resour…
Sparse Transformers can approximate dense Transformers with only O(n) connections.
problem Can sparse Transformers approximate arbitrary sequence-to-sequence functions?
method Proposed sufficient conditions for universal approximation and proved that sparse Transformers with O(n) connections can approximate dense models.
result Sparse Transformers with O(n) connections can approximate the same function class as dense models with n^2 connections.
This tutorial introduces a new and powerful set of techniques variously called "neural machine translation" or "neural sequence-to-sequence models". These techniques have been used in a number of tasks regarding the handling of human language, and can be a powerful tool in the toolbox of anyone who wants to model seque…
This study evaluates the performances of an LSTM network for detecting and extracting the intent and content of com- mands for a financial chatbot. It presents two techniques, sequence to sequence learning and Multi-Task Learning, which might improve on the previous task.
When Do Transformers Outperform Feedforward and Recurrent Networks? A Statistical Perspectivestat.ML
Transformers can outperform feedforward and recurrent networks due to dynamic sparsity.
problem Understanding when and why Transformers outperform other neural network architectures.
method Analyzing a sequence-to-sequence data generating model with dynamic sparsity, proving sample complexity differences between feedforward, recurrent, and Transformers.
result Transformers can learn dynamic sparsity models with lower sample complexity than feedforward and recurrent networks.
SNNs can represent complex functions efficiently.
problem Understanding the representational power of SNNs.
method Viewed as sequence-to-sequence processors, analyzed using spike train functions.
result SNNs have the universal representation property for certain functions.
Copy this Sentencecs.LG
Attention is an operation that selects some largest element from some set, where the notion of largest is defined elsewhere. Applying this operation to sequence to sequence mapping results in significant improvements to the task at hand. In this paper we provide the mathematical definition of attention and examine its …
STConvS2S improves weather forecasting using only convolutional layers.
problem Predicting future weather conditions more accurately.
method Proposes a deep learning architecture combining spatiotemporal convolutional layers.
result Outperforms state-of-the-art architectures for forecasting tasks.
Generative models have long been the dominant approach for speech recognition. The success of these models however relies on the use of sophisticated recipes and complicated machinery that is not easily accessible to non-practitioners. Recent innovations in Deep Learning have given rise to an alternative - discriminati…
Deep learning model predicts traffic flows across entire network for multiple steps ahead.
problem Accurately forecasting future traffic flows across all network links.
method Spatial-Temporal Sequence to Sequence (STSeq2Seq) model combining seq2seq and graph convolution.
result STSeq2Seq achieves state-of-the-art performance in traffic forecasting.
Automatic sleep staging has been often treated as a simple classification problem that aims at determining the label of individual target polysomnography (PSG) epochs one at a time. In this work, we tackle the task as a sequence-to-sequence classification problem that receives a sequence of multiple epochs as input and…
This paper applies deep learning to ordinal regression, modeling it as a binary search.
problem Ordinal regression with deep learning models.
method Formulated ordinal regression as a binary search problem, using recurrent neural networks.
result Deep learning model shows comparable or better predictive power compared to traditional methods.
Recurrent Neural Networks (RNNs) achieve state-of-the-art results in many sequence-to-sequence modeling tasks. However, RNNs are difficult to train and tend to suffer from overfitting. Motivated by the Data Processing Inequality (DPI), we formulate the multi-layered network as a Markov chain, introducing a training met…
New analysis shows RPE-based Transformers can't approximate all functions.
problem Understanding the limitations of RPE-based Transformers in approximating continuous functions.
method Mathematical analysis and development of a novel attention module (URPE) to overcome limitations.
result RPE-based Transformers can't approximate all continuous sequence-to-sequence functions, even with depth and width.
Deep learning model forecasts PV power production with high accuracy.
problem Accurate forecasting of photovoltaic power production is needed for electricity infrastructure.
method Sequence to sequence model with attention mechanism using weather predictions and historical data.
result The model outperforms existing methods in forecast skill score.
Memory-Augmented Recurrent Networks improve dialogue coherence by expanding conversation history storage.
problem Fixed-size vectors limit dialogue coherence; attention mechanisms are computationally expensive.
method Introduce Neural Turing Machines (NTMs) to provide flexible and permanent storage for dialogue history.
result Improved perplexity performance compared to existing baselines.
Transformer networks approximate Hölder and Sobolev functions with fixed-depth networks.
problem Nonparametric regression with dependent observations.
method Established novel upper bounds for Transformer networks approximating Hölder and Sobolev functions under various -mixing data assumptions.
result Explicit convergence rates for nonparametric regression problems under -mixing data assumptions.
Study improves voice conversion model with Mel-spectrogram augmentation.
problem Insufficient speech pairs data for training sequence-to-sequence voice conversion models.
method Experimented with Mel-spectrogram augmentation using SpecAugment policies and proposed new augmentation policies.
result Time axis warping policies showed better performance in training the voice conversion model.
Sequence-to-Sequence (seq2seq) modeling has rapidly become an important general-purpose NLP tool that has proven effective for many text-generation and sequence-labeling tasks. Seq2seq builds on deep neural language modeling and inherits its remarkable accuracy in estimating local, next-word distributions. In this work…
This paper presents a novel end-to-end approach to program repair based on sequence-to-sequence learning. We devise, implement, and evaluate a system, called SequenceR, for fixing bugs based on sequence-to-sequence learning on source code. This approach uses the copy mechanism to overcome the unlimited vocabulary probl…
In the past few years, neural abstractive text summarization with sequence-to-sequence (seq2seq) models have gained a lot of popularity. Many interesting techniques have been proposed to improve seq2seq models, making them capable of handling different challenges, such as saliency, fluency and human readability, and ge…