Sequence-to-Sequence (seq2seq) modeling has rapidly become an important general-purpose NLP tool that has proven effective for many text-generation and sequence-labeling tasks. Seq2seq builds on deep neural language modeling and inherits its remarkable accuracy in estimating local, next-word distributions. In this work…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,051 papers · 148 categories
Trend · papers per month
48 results for “sequence training”
Study on continuous sequence classification with distribution uncertainty.
problem Classifying continuous sequences with varying distribution uncertainty.
method Proposes distribution-free tests for three test designs: fixed-length, sequential, and two-phase tests.
result Error probabilities decay exponentially fast for all test designs.
Improves NMT by sampling context from predicted sequence during training.
problem Error accumulation and overcorrection in NMT due to mismatched training and inference contexts.
method Samples context words from both ground truth and predicted sequences during training.
result Significant improvements on multiple datasets, including Chinese->English and WMT'14 English->German.
This research proves guarantees on sequence models' generalization to longer and novel sequences.
problem Generalization to longer sequences and novel token combinations in sequence models.
method Provable guarantees on length and compositional generalization for various sequence models.
result Limited capacity models achieve both length and compositional generalization with diverse training distributions.
Optimal control theory improves machine teaching efficiency.
problem Finding the shortest training sequence for a sequential learning algorithm to reach a target model.
method Formulated as a time-optimal control problem, leveraging optimal control theory and computational tools.
result Optimal training sequences can vastly outperform existing heuristics.
We present a recurrent encoder-decoder deep neural network architecture that directly translates speech in one language into text in another. The model does not explicitly transcribe the speech into text in the source language, nor does it require supervision from the ground truth source language transcription during t…
Deep neural network translates math formula images to LaTeX sequences.
problem Translating math formula images to LaTeX sequences accurately and efficiently.
method Encoder-decoder architecture with CNN and LSTM, sequence-level training with policy gradient.
result State-of-the-art performance on sequence-based and image-based evaluation metrics.
Attention forcing improves sequence-to-sequence model training stability.
problem Training auto-regressive sequence-to-sequence models with attention mechanism is challenging.
method Attention forcing guides the model with generated output history and reference attention.
result Attention forcing trains models to recover from mistakes without requiring a schedule or classifier.
Sequence learning improves query expansion in information retrieval.
problem Improving query expansion in information retrieval systems.
method Used sequence to sequence algorithms to extract keywords from sentence embeddings and trained a neural network on open datasets.
result Sequence to sequence models can capture complex query expansion relations in word embeddings.
Paper proposes a faster RAE with sequence-aware encoding.
problem Training recurrent autoencoders is challenging and time-consuming.
method Introduces a recurrent autoencoder with sequence-aware encoding using 1D convolutional layers.
result The proposed autoencoder trains faster than standard RAE.
Study improves voice conversion model with Mel-spectrogram augmentation.
problem Insufficient speech pairs data for training sequence-to-sequence voice conversion models.
method Experimented with Mel-spectrogram augmentation using SpecAugment policies and proposed new augmentation policies.
result Time axis warping policies showed better performance in training the voice conversion model.
Sequence-to-sequence models, such as attention-based models in automatic speech recognition (ASR), are typically trained to optimize the cross-entropy criterion which corresponds to improving the log-likelihood of the data. However, system performance is usually measured in terms of word error rate (WER), not log-likel…
CGAN fails to improve deterministic sequence predictions, revealing a theoretical limitation.
problem Improving deterministic sequence predictions with CGAN.
method Developed an adversarial content loss approach.
result CGAN does not improve deterministic sequence predictions.
Bootstrapped Training of Score-Conditioned Generator for Offline Design of Biological Sequencesq-bio.QM
Algorithm optimizes biological sequences using bootstrapped training with a score-conditioned generator.
problem Optimizing biological sequences for a black-box score function.
method Bootstrapped training of score-conditioned generator (BootGen) algorithm.
result Our method outperforms competitive baselines on biological sequential design tasks.
OrderNet learns to sort sequences from training data.
problem Sorting unordered sequences where correct order is not easily defined.
method OrderNet is a neural architecture designed to be permutation equivariant and capable of rich interactions.
result OrderNet outperforms previous techniques in generalizing to longer sequences.
We show that the subsurface projection of a train track splitting sequence is an unparameterized quasi-geodesic in the curve complex of the subsurface. For the proof we introduce induced tracks, efficient position, and wide curves. This result is an important step in the proof that the disk complex is Gromov hyperbolic…
We study a variant of the source identification game with training data in which part of the training data is corrupted by an attacker. In the addressed scenario, the defender aims at deciding whether a test sequence has been drawn according to a discrete memoryless source , whose statistics are known to hi…
A new loss function speeds up sequence-to-sequence models for continuous outputs.
problem Slow and memory-intensive softmax layer limits vocabulary size and translation quality.
method Proposes a probabilistic loss and continuous embedding layer training/inference procedure.
result Models achieve up to 2.5x speed-up in training time with similar translation quality.
Deep Neural Network (DNN) acoustic models often use discriminative sequence training that optimises an objective function that better approximates the word error rate (WER) than frame-based training. Sequence training is normally implemented using Stochastic Gradient Descent (SGD) or Hessian Free (HF) training. This pa…
Pre-Training of Deep Bidirectional Protein Sequence Representations with Structural Informationq-bio.BM
PLUS pre-trains protein sequences with structural info, improving performance.
problem Lack of labeled protein sequences for training models.
method PLUS combines masked language modeling with same-family prediction for pre-training.
result PLUS-RNN outperforms other models in protein biology tasks.
Develops a new point process model for detecting neural spike sequences.
problem Detecting sparse sequences of neural spikes in high-dimensional spike trains.
method A point process model that represents sequence occurrences as marked events in continuous time, with learnable time warping parameters.
result Demonstrates improved detection and modeling of neural spike sequences.
Optimal Change-Point Detection with Training Sequences in the Large and Moderate Deviations Regimescs.IT
Paper optimizes change-point detection using learned distributions from training sequences.
problem Optimal change-point detection with unknown pre- and post-change distributions.
method Designs a change-point estimator using training sequences and test sequences.
result Optimal confidence width characterized as a function of undetected error.
A new model improves recurrent neural networks' ability to memorize long sequences.
problem Improving recurrent neural networks' ability to memorize long sequences and extract task-relevant features.
method Proposes a Linear Memory Network with an encoding-based memorization component and a specialized training algorithm.
result Improves the final performance of recurrent neural networks when memorizing long sequences is necessary.
Differentiable losses for combinatorial optimization problems in sequence modeling.
problem Mismatch between training and inference objectives in sequence models.
method Gradient descent over linear programs representing combinatorial optimization problems.
result Gradient descent can be applied to combinatorial optimization problems efficiently.
We use Bayesian optimal experimental design to generate near-optimal attention sequences for faster hard attention training.
problem Training hard attention mechanisms is computationally expensive and high-variance.
method We frame hard attention as a BOED problem and use approximation methods from BOED to generate near-optimal attention sequences.
result Near-optimal attention sequences can speed up hard attention training and be reused by other networks.
New deep learning method outperforms random training data.
problem Improving accuracy of deep learning algorithms in high dimensions.
method Training with low-discrepancy sequences instead of random data.
result Significantly outperforms standard deep learning algorithms.
Generates new human genomic sequences for LAI training.
problem Lack of accessible reference data sets for LAI.
method Class-conditional VAE-GAN to generate realistic sequences.
result Generated sequences improve LAI method performance.
Self-training improves neural sequence generation by correcting incorrect predictions.
problem Improving neural sequence generation models using unlabeled data.
method Injecting pseudo-parallel data (model predictions) into the labeled dataset and using dropout as a regularizer.
result Noisy self-training significantly improves performance on machine translation and text summarization benchmarks.
Improved speech recognition with language model integration in sequence-to-sequence models.
problem Improving word error rate in speech recognition models.
method Log-linear combination of acoustic and language models with per-token renormalization.
result The proposed method shows good improvements over standard model combination on Librispeech system.
Recurrent Kernel Networksstat.ML
End-to-end training of recurrent neural networks for biological sequences.
problem Training recurrent neural networks for biological sequence classification.
method Generalized convolutional kernel networks with recurrent structures.
result Outperforms existing methods for protein classification tasks.
Generative models have long been the dominant approach for speech recognition. The success of these models however relies on the use of sophisticated recipes and complicated machinery that is not easily accessible to non-practitioners. Recent innovations in Deep Learning have given rise to an alternative - discriminati…
Paper tackles adaptive deletion of data points from trained models.
problem Adaptive deletion of data points from trained models.
method Reduction from adaptive to non-adaptive deletion guarantees using differential privacy and max information.
result Strong provable deletion guarantees for adaptive deletion sequences.
A new method optimizes neural sequence models for better task performance.
problem Training neural sequence models with maximum likelihood estimation ignores task losses.
method Maximum likelihood guided parameter search (MGS) in the parameter space.
result MGS optimizes sequence-level losses, reducing repetition and non-termination.
Neural networks struggle with long sequences, but a new method improves their performance.
problem Neural networks struggle to generalize to longer sequences and unseen data.
method Proposed a learned conditional masking mechanism and binary encoding for numbers.
result Models can now generalize far outside their training range with near-perfect accuracy.
Neural approaches to sequence labeling often use a Conditional Random Field (CRF) to model their output dependencies, while Recurrent Neural Networks (RNN) are used for the same purpose in other tasks. We set out to establish RNNs as an attractive alternative to CRFs for sequence labeling. To do so, we address one of t…
GAN-AE generates synthetic data for imbalanced sequence classification.
problem Imbalanced sequence classification in medical devices.
method Autoencoder and GAN architecture for generating synthetic data.
result GAN-AE outperforms other synthetic data generation methods.
Three LF training criteria improve neural network acoustic models without cross-entropy pre-training.
problem Improving purely sequence-trained neural network acoustic models.
method Comparison of three lattice-free discriminative training criteria (MMI, bMMI, sMBR) on LVCSR tasks.
result LF-bMMI models outperform plain LF-MMI models by 5% WER on Switchboard datasets.
Let denote the genus orientable surface with punctures. We show that nested train track sequences constitute -quasiconvex subsets of the curve graph, effectivizing a theorem of Masur and Minsky. As a consequence, the genus disk set is -quasiconvex. We also show that splitti…
SRS improves deep learning accuracy on small image datasets.
problem Training deep neural networks with limited images per class.
method Fixed sequence index sampling with batch replacement.
result Significantly improved classification accuracy on CIFAR-100.
FIGARO generates symbolic music with fine-grained control.
problem Minimal control over generated music sequences.
method Description-to-sequence task, learning conditional distribution of sequences given high-level descriptions.
result State-of-the-art controllable symbolic music generation.
Neural network optimizes learning sequence for reading words.
problem Children struggle with learning to read words due to inconsistent spelling-sound correspondences.
method Used a neural network to structure learning trials to optimize generalization accuracy.
result Significant improvement in generalization accuracy compared to random or frequency-based sequences.
Improves spatio-temporal forecasting by reducing errors between training and inference.
problem Accumulation of small errors in Seq2Seq models during inference due to different distributions of training and inference phases.
method Curriculum learning based on Temporal Progressive Growing Sampling to replace some ground-truth context with generated predictions.
result Better models long-term dependencies and outperforms baseline approaches on two datasets.
Paper proposes using LSTM for LSH-based sequence alignment.
problem Sequence alignment using deep learning models.
method Deep bidirectional LSTM for feature learning and LSH-based sequence alignment.
result Higher accuracy achieved with LSTM-based model.
Study SGD dynamics in sequence models, revealing training phases and influence of sequence length.
problem Understanding SGD in sequence models like attention networks.
method Derived closed-form population loss and analyzed SGD dynamics for SSI models.
result Two distinct training phases: escape from uninformative initialization and alignment with target subspace.
MARGE learns to reconstruct text by paraphrasing, achieving strong performance across multiple tasks.
problem Training sequence-to-sequence models with limited supervision.
method Unsupervised multi-lingual multi-document paraphrasing objective.
result Strong zero-shot performance on document translation and various tasks in multiple languages.
Improves E2E ASR performance on numeric sequences with additional training data and denormalization.
problem Challenges in recognizing numeric sequences out-of-vocabulary in ASR systems.
method Uses text-to-speech for additional numeric training data and a small-footprint neural network for denormalization.
result Reduction of WER by up to a factor of 8 in the longest numeric sequences.
This study investigates global normalization in neural models, showing its effectiveness in search-aware training.
problem Theoretical equivalence of global and local normalization in high-capacity models, practical advantage unclear.
method Continuous relaxation of beam search for training globally normalized recurrent sequence models.
result Globally normalized models are more effective than locally normalized ones in inexact search.
We describe a fully data driven model that learns to perform a retrosynthetic reaction prediction task, which is treated as a sequence-to-sequence mapping problem. The end-to-end trained model has an encoder-decoder architecture that consists of two recurrent neural networks, which has previously shown great success in…