Develops a new point process model for detecting neural spike sequences.
problem Detecting sparse sequences of neural spikes in high-dimensional spike trains.
method A point process model that represents sequence occurrences as marked events in continuous time, with learnable time warping parameters.
result Demonstrates improved detection and modeling of neural spike sequences.
Investigates neural TTS systems for Japanese and English.
problem Improving neural TTS systems for high-quality speech synthesis.
method Comparative study of neural sequence-to-sequence TTS vs. DNN pipeline TTS, varying model architecture, parameter size, and language.
result A neural sequence-to-sequence TTS system requires sufficient model parameters and a powerful encoder for high-quality speech synthesis.
Neural SDEs model continuous sequences using neural networks.
problem Modeling continuous-time dynamics in sequence data.
method Interprets time-series as samples from a continuous dynamical system, parameterized by Neural SDE.
result Demonstrates superior performance in diverse sequence modeling tasks.
New model controls memory in seq2seq tasks, revealing learning regimes.
problem Understanding memory in seq2seq tasks using neural networks.
method Introducing a stochastic switching-Ornstein-Uhlenbeck (SSOU) model to control memory and a measure of non-Markovianity.
result Two learning regimes emerge from the interplay of time scales in the SSOU process.
Enhances sequence memory capacity in neural networks.
problem Limited sequence capacity in Hopfield-like neural networks.
method Introducing a nonlinear interaction term and a generalized pseudoinverse rule.
result Significantly increased sequence capacity with novel scaling laws.
The celebrated Sequence to Sequence learning (Seq2Seq) technique and its numerous variants achieve excellent performance on many tasks. However, many machine learning tasks have inputs naturally represented as graphs; existing Seq2Seq models face a significant challenge in achieving accurate conversion from graph form …
A new method optimizes neural sequence models for better task performance.
problem Training neural sequence models with maximum likelihood estimation ignores task losses.
method Maximum likelihood guided parameter search (MGS) in the parameter space.
result MGS optimizes sequence-level losses, reducing repetition and non-termination.
This tutorial introduces a new and powerful set of techniques variously called "neural machine translation" or "neural sequence-to-sequence models". These techniques have been used in a number of tasks regarding the handling of human language, and can be a powerful tool in the toolbox of anyone who wants to model seque…
Neural model outperforms ETAS in forecasting Central Apennines earthquakes.
problem Short-term seismicity forecasting with incomplete data.
method Extended a neural network model to the magnitude domain, using it to forecast earthquakes above a target magnitude threshold.
result Neural model outperforms ETAS at lower magnitude thresholds, due to its robustness to missing data.
Improved neural models for diverse user event sequences.
problem Challenges in modeling diverse user event sequences.
method Mixtures of latent embeddings with amortized variational inference.
result Systematic improvements over existing work for various predictive metrics.
New KNN test improves association analysis of high-dimensional sequencing data.
problem Challenges in using neural networks for high-dimensional sequencing data analysis.
method Kernel-based neural network (KNN) test for complex association analysis.
result KNN test outperforms SKAT in detecting non-linear and interaction effects.
Summarization of long sequences into a concise statement is a core problem in natural language processing, requiring non-trivial understanding of the input. Based on the promising results of graph neural networks on highly structured data, we develop a framework to extend existing sequence encoders with a graph compone…
Undirected neural sequence models such as BERT (Devlin et al., 2019) have received renewed interest due to their success on discriminative natural language understanding tasks such as question-answering and natural language inference. The problem of generating sequences directly from these models has received relativel…
We propose a conditional non-autoregressive neural sequence model based on iterative refinement. The proposed model is designed based on the principles of latent variable models and denoising autoencoders, and is generally applicable to any sequence generation task. We extensively evaluate the proposed model on machine…
This paper introduces a new neural ODE model for continuous-time sequence generation.
problem Representing and predicting continuous-time sequences with high accuracy.
method A neural emission model and neural ODE define the latent state evolution, with an Energy-based model for prior distribution.
result The model outperforms existing methods in various tasks, including long-horizon predictions.
Deep recurrent neural networks perform well on sequence data and are the model of choice. However, it is a daunting task to decide the structure of the networks, i.e. the number of layers, especially considering different computational needs of a sequence. We propose a layer flexible recurrent neural network with adapt…
A new model improves recurrent neural networks' ability to memorize long sequences.
problem Improving recurrent neural networks' ability to memorize long sequences and extract task-relevant features.
method Proposes a Linear Memory Network with an encoding-based memorization component and a specialized training algorithm.
result Improves the final performance of recurrent neural networks when memorizing long sequences is necessary.
In this paper we propose a deep neural network model with an encoder-decoder architecture that translates images of math formulas into their LaTeX markup sequences. The encoder is a convolutional neural network (CNN) that transforms images into a group of feature maps. To better capture the spatial relationships of mat…
Machine-assisted treatment recommendations hold a promise to reduce physician time and decision errors. We formulate the task as a sequence-to-sequence prediction model that takes the entire time-ordered medical history as input, and predicts a sequence of future clinical procedures and medications. It is built on the …
The paper develops a convex parameterization for robust RNNs ensuring stability and robustness.
problem Lack of stability and robustness guarantees in RNNs for sequence-to-sequence mapping applications.
method Formulated convex sets of RNNs with stability and robustness guarantees using incremental quadratic constraints.
result The proposed model structure ensures global exponential stability and bounds on incremental ℓ2 gain. Neural network framework for language recognition considers sequence information and improves accuracy.
problem Challenging task of automatic language identification in noisy conditions.
method Proposes a neural network framework with bidirectional LSTM and attention modeling for relevance weighting.
result Significant improvements over conventional methods in noisy conditions and multi-speaker speech.
Paper proposes using LSTM for LSH-based sequence alignment.
problem Sequence alignment using deep learning models.
method Deep bidirectional LSTM for feature learning and LSH-based sequence alignment.
result Higher accuracy achieved with LSTM-based model.
Learned factor graphs improve inference from time sequences using neural networks.
problem Inference from time sequences with limited labeled data.
method Combines model-based algorithms and data-driven ML tools for stationary time sequences.
result Learned factor graphs can accurately infer from small training sets.
Improves text-to-speech speed by interleaving character reading and audio synthesis.
problem Latency in text-to-speech models limits their use in time-sensitive tasks.
method Reinforcement learning to train an agent to choose the order of character reading and audio synthesis.
result The proposed method successfully balances latency and audio quality.
We propose a neural architecture with the main characteristics of the most successful neural models of the last years: bidirectional RNNs, encoder-decoder, and the Transformer model. Evaluation on three sequence labelling tasks yields results that are close to the state-of-the-art for all tasks and better than it for s…
We describe a fully data driven model that learns to perform a retrosynthetic reaction prediction task, which is treated as a sequence-to-sequence mapping problem. The end-to-end trained model has an encoder-decoder architecture that consists of two recurrent neural networks, which has previously shown great success in…
The paper explores how to make neural networks extrapolate longer sequences.
problem Neural networks struggle to extrapolate beyond seen data, especially for longer sequences.
method The authors propose a model with separate content- and location-based attention mechanisms.
result Models with the proposed attention mechanisms are better at extrapolating longer sequences.
k-Nearest Neighbors is one of the most fundamental but effective classification models. In this paper, we propose two families of models built on a sequence to sequence model and a memory network model to mimic the k-Nearest Neighbors model, which generate a sequence of labels, a sequence of out-of-sample feature vecto…
NeSS combines neural and symbolic approaches for better compositional generalization.
problem Lack of compositional generalization in deep learning models.
method NeSS uses a neural network to generate traces, executed by a symbolic stack machine with sequence manipulation.
result Achieves 100% generalization performance across multiple domains.
Study reveals issues with neural autoregressive models and proposes mode recovery cost.
problem Unreasonable affinity of neural autoregressive models to short and long sequences.
method Investigates modes of ground-truth, empirical, and decoding-induced distributions via mode recovery cost.
result Mode recovery cost varies depending on ground-truth distribution and impacts decoding-induced distribution.
Proposes a method to train neural networks directly on compressed text data.
problem Training neural networks on compressed text data without decompression.
method Introduces composer modules to encode symbols from grammar compression rules into vector representations.
result Demonstrates that the proposed method can achieve both memory and computational efficiency while maintaining moderate performance.
Planar neural networks learn image transformations from sequences.
problem Learning image transformations for mental simulation.
method Using planar neural networks, the study investigates various factors affecting the learning of image transformations.
result The approach can effectively learn and transfer image transformations, including translation, rotation, and scaling.
Time-aware neural models improve system identification from unevenly sampled data.
problem Improving system identification from continuous variables with unevenly sampled time data.
method Introduced a time-aware and stationary extension of recurrent neural networks.
result Demonstrated improved performance on industrial input/output processes.
Text normalization is an important enabling technology for several NLP tasks. Recently, neural-network-based approaches have outperformed well-established models in this task. However, in languages other than English, there has been little exploration in this direction. Both the scarcity of annotated data and the compl…
Recurrent Neural Networks (RNNs) have been proven to be effective in modeling sequential data and they have been applied to boost a variety of tasks such as document classification, speech recognition and machine translation. Most of existing RNN models have been designed for sequences assumed to be identically and ind…
We introduce a new neural architecture to learn the conditional probability of an output sequence with elements that are discrete tokens corresponding to positions in an input sequence. Such problems cannot be trivially addressed by existent approaches such as sequence-to-sequence and Neural Turing Machines, because th…
A convolutional sequence to sequence non-intrusive load monitoring model is proposed in this paper. Gated linear unit convolutional layers are used to extract information from the sequences of aggregate electricity consumption. Residual blocks are also introduced to refine the output of the neural network. The partiall…
Meta-learning improves event prediction from short sequences.
problem Predicting events from short sequences is challenging.
method Meta-learning approach using recurrent neural networks and monotonic neural networks.
result Meta-learning enhances long-term prediction performance.
Graph Hawkes Neural Network forecasts evolving graph sequences.
problem Modeling dynamic graph sequences with complex event types.
method Generalized Hawkes process to neural network, capturing complex event impacts.
result Effective at predicting future events in evolving graph sequences.
Tutorial on using neural networks for single cell data analysis.
problem Handling large sequencing datasets efficiently.
method Single cell variational inference using variational auto-encoder.
result Model learns data distribution for insights.
STConvS2S improves weather forecasting using only convolutional layers.
problem Predicting future weather conditions more accurately.
method Proposes a deep learning architecture combining spatiotemporal convolutional layers.
result Outperforms state-of-the-art architectures for forecasting tasks.
Models for sequential data such as the recurrent neural network (RNN) often implicitly model a sequence as having a fixed time interval between observations and do not account for group-level effects when multiple sequences are observed. We propose a model for grouped sequential data based on the RNN that accounts for …
Extracts main content from web pages using neural sequence labeling.
problem Lack of generalization in existing web page content extraction models.
method Neural sequence labeling model using HTML tags and words as input.
result Model outperforms state-of-the-art and adapts to changes in web page structure.
New method identifies latent variables in cognitive models using neural networks.
problem Inference of latent variables in complex cognitive models is limited.
method Recurrent neural networks and simulation-based inference for latent variable sequences.
result Extends neural Bayes estimation to broader classes of cognitive models.
System uses neural networks to prove program equivalence via rewrite rules.
problem Proving equivalence between two dataflow graphs.
method Developed a graph-to-sequence neural network trained on example generation to find semantics-preserving rewrite rules.
result System correctly outputs a rewrite sequence for 96% of program pairs, proving equivalence.
Benchmark for math reasoning models from human proofs.
problem Measuring and accelerating machine learning models in high-level mathematical reasoning.
method Built a non-synthetic dataset from theorem prover proofs, defined a task for model to fill in missing propositions, used hierarchical transformer to improve performance.
result Neural models can capture non-trivial mathematical reasoning, hierarchical transformer outperforms baseline.
QRNN uses quantum neurons to learn sequences efficiently.
problem Efficiently learning sequences with quantum computing.
method Parametrized quantum neurons and amplitude amplification.
result QRNN outperforms classical RNNs on sequence learning tasks.
In this paper, we propose a novel neural network model called RNN Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN encodes a sequence of symbols into a fixed-length vector representation, and the other decodes the representation into another sequence of symbols. The encoder and decoder of t…