Mining tasks over sequential data, such as clickstreams and gene sequences, require a careful design of embeddings usable by learning algorithms. Recent research in feature learning has been extended to sequential data, where each instance consists of a sequence of heterogeneous items with a variable length. However, m…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “Task Sequencing”
A new method optimizes neural sequence models for better task performance.
problem Training neural sequence models with maximum likelihood estimation ignores task losses.
method Maximum likelihood guided parameter search (MGS) in the parameter space.
result MGS optimizes sequence-level losses, reducing repetition and non-termination.
New model controls memory in seq2seq tasks, revealing learning regimes.
problem Understanding memory in seq2seq tasks using neural networks.
method Introducing a stochastic switching-Ornstein-Uhlenbeck (SSOU) model to control memory and a measure of non-Markovianity.
result Two learning regimes emerge from the interplay of time scales in the SSOU process.
New method for learning evolving tasks with performance guarantees.
problem Learning tasks in a sequence with evolving similarity.
method Adaptable learning methodology with performance guarantees.
result Improved performance in multiple scenarios with reliable guarantees.
We study the relationship between catastrophic forgetting and properties of task sequences. In particular, given a sequence of tasks, we would like to understand which properties of this sequence influence the error rates of continual learning algorithms trained on the sequence. To this end, we propose a new procedure …
Copy this Sentencecs.LG
Attention is an operation that selects some largest element from some set, where the notion of largest is defined elsewhere. Applying this operation to sequence to sequence mapping results in significant improvements to the task at hand. In this paper we provide the mathematical definition of attention and examine its …
Paper presents IMRCs for evolving tasks with forward and backward learning.
problem Incremental learning of evolving tasks with few samples per task.
method Incremental minimax risk classifiers (IMRCs) that exploit forward and backward learning.
result IMRCs provide significant performance improvement, especially with reduced sample sizes.
Sequences have become first class citizens in supervised learning thanks to the resurgence of recurrent neural networks. Many complex tasks that require mapping from or to a sequence of observations can now be formulated with the sequence-to-sequence (seq2seq) framework which employs the chain rule to efficiently repre…
New insights into how encoder-decoder networks generate attention matrices.
problem Understanding how encoder-decoder networks use attention matrices.
method Decomposing hidden states into temporal and input-driven components.
result Attention matrices are formed based on task requirements, not architecture type.
Sequence to sequence learning has recently emerged as a new paradigm in supervised learning. To date, most of its applications focused on only one task and not much work explored this framework for multiple tasks. This paper examines three multi-task learning (MTL) settings for sequence to sequence models: (a) the onet…
Recent studies have shown that neural models can achieve high performance on several sequence labelling/tagging problems without the explicit use of linguistic features such as part-of-speech (POS) tags. These models are trained only using the character-level and the word embedding vectors as inputs. Others have shown …
We present a recurrent encoder-decoder deep neural network architecture that directly translates speech in one language into text in another. The model does not explicitly transcribe the speech into text in the source language, nor does it require supervision from the ground truth source language transcription during t…
Dynamical system models (including RNNs) often lack the ability to adapt the sequence generation or prediction to a given context, limiting their real-world application. In this paper we show that hierarchical multi-task dynamical systems (MTDSs) provide direct user control over sequence generation, via use of a latent…
This study evaluates the performances of an LSTM network for detecting and extracting the intent and content of com- mands for a financial chatbot. It presents two techniques, sequence to sequence learning and Multi-Task Learning, which might improve on the previous task.
Randomized positional encodings boost transformer performance on longer sequences.
problem Transformers struggle with generalizing to sequences of arbitrary length.
method Introduced randomized positional encodings that simulate longer sequences and randomly select positions.
result Randomized positional encodings increase test accuracy by 12.0% on average for sequences of unseen length.
Recurrent Neural Networks (RNNs) with attention mechanisms have obtained state-of-the-art results for many sequence processing tasks. Most of these models use a simple form of encoder with attention that looks over the entire sequence and assigns a weight to each token independently. We present a mechanism for focusing…
OptiGAN uses GAN and RL to optimize sequence generation for specific goals.
problem Challenging in sequence generation tasks to generate sequences with specific desired goals.
method Integrates GAN and RL to optimize desired goal scores using policy gradients.
result Achieves higher desired scores in text and real-valued sequence generation.
Sequence-to-Sequence (seq2seq) modeling has rapidly become an important general-purpose NLP tool that has proven effective for many text-generation and sequence-labeling tasks. Seq2seq builds on deep neural language modeling and inherits its remarkable accuracy in estimating local, next-word distributions. In this work…
Curriculum learning in reinforcement learning is used to shape exploration by presenting the agent with increasingly complex tasks. The idea of curriculum learning has been largely applied in both animal training and pedagogy. In reinforcement learning, all previous task sequencing methods have shaped exploration with …
We investigated the training of a shared model for both text-to-speech (TTS) and voice conversion (VC) tasks. We propose using an extended model architecture of Tacotron, that is a multi-source sequence-to-sequence model with a dual attention mechanism as the shared model for both the TTS and VC tasks. This model can a…
Yseop at SemEval-2020 Task 5: Cascaded BERT Language Model for Counterfactual Statement Analysiscs.CL
Paper tackles counterfactual sentence detection and evaluation.
problem Detect and evaluate counterfactual sentences in natural language.
method Used a BERT base model for classification and a hybrid BERT Multi-Layer Perceptron for sequence identification. Introduced cascaded linear inputs to improve performance.
result Achieved an F1 score of 85.00% in Task 1 and 83.90% in Task 2.
Self-attentional models are a new paradigm for sequence modelling tasks which differ from common sequence modelling methods, such as recurrence-based and convolution-based sequence learning, in the way that their architecture is only based on the attention mechanism. Self-attentional models have been used in the creati…
The celebrated Sequence to Sequence learning (Seq2Seq) technique and its numerous variants achieve excellent performance on many tasks. However, many machine learning tasks have inputs naturally represented as graphs; existing Seq2Seq models face a significant challenge in achieving accurate conversion from graph form …
Unified model for sequence labeling and classification.
problem Efficiently perform multiple sequence labeling tasks.
method Generative framework with shared natural language output space.
result Significant improvements in few-shot and low-resource slot labeling.
We combine multi-task learning and semi-supervised learning by inducing a joint embedding space between disparate label spaces and learning transfer functions between label embeddings, enabling us to jointly leverage unlabelled data and auxiliary, annotated datasets. We evaluate our approach on a variety of sequence cl…
The paper extends influence functions to sequence tagging tasks for better model interpretability.
problem Lack of interpretability methods for sequence tagging models.
method Define and compute influence of training instance segments on test segment predictions.
result The segment influence method tracks with true influence and identifies annotation errors.
Meta-learning improves event prediction from short sequences.
problem Predicting events from short sequences is challenging.
method Meta-learning approach using recurrent neural networks and monotonic neural networks.
result Meta-learning enhances long-term prediction performance.
Locally-contextual CRFs improve sequence labeling performance.
problem Improving sequence labeling with contextual embeddings.
method Locally-contextual nonlinear CRFs using deep neural networks.
result Consistently outperforms linear chain CRF and previous state of the art.
In this work, we present a reinforcement learning algorithm that can find a variety of policies (novel policies) for a task that is given by a task reward function. Our method does this by creating a second reward function that recognizes previously seen state sequences and rewards those by novelty, which is measured u…
BigBird improves transformer performance on NLP tasks with longer sequences.
problem Quadratic dependency on sequence length in transformer models.
method Sparse attention mechanism reducing quadratic dependency to linear.
result Significant improvement in performance on NLP tasks like QA and summarization.
In several natural language tasks, labeled sequences are available in separate domains (say, languages), but the goal is to label sequences with mixed domain (such as code-switched text). Or, we may have available models for labeling whole passages (say, with sentiments), which we would like to exploit toward better po…
Recurrent neural networks (RNNs) sequentially process data by updating their state with each new data point, and have long been the de facto choice for sequence modeling tasks. However, their inherently sequential computation makes them slow to train. Feed-forward and convolutional architectures have recently been show…
Hybrid framework prevents forgetting in continual learning.
problem Avoiding forgetting in learning new tasks without forgetting old ones.
method Hybrid continual learning framework combining architecture growth and experience replay.
result Hybrid approach effectively avoids forgetting across multiple tasks.
PRAGMA models financial event sequences for various banking tasks.
problem Handling diverse financial data for multiple applications.
method Pre-training a Transformer model on a large banking event corpus with a self-supervised objective.
result PRAGMA achieves superior performance across multiple financial domains from raw event sequences.
Traditional pairwise sequence alignment is based on matching individual samples from two sequences, under time monotonicity constraints. However, in many application settings matching subsequences (segments) instead of individual samples may bring in additional robustness to noise or local non-causal perturbations. Thi…
We propose a neural architecture with the main characteristics of the most successful neural models of the last years: bidirectional RNNs, encoder-decoder, and the Transformer model. Evaluation on three sequence labelling tasks yields results that are close to the state-of-the-art for all tasks and better than it for s…
State-of-the-art sequence labeling systems traditionally require large amounts of task-specific knowledge in the form of hand-crafted features and data pre-processing. In this paper, we introduce a novel neutral network architecture that benefits from both word- and character-level representations automatically, by usi…
Bandit structured prediction describes a stochastic optimization framework where learning is performed from partial feedback. This feedback is received in the form of a task loss evaluation to a predicted output structure, without having access to gold standard structures. We advance this framework by lifting linear ba…
Learning to remember long sequences remains a challenging task for recurrent neural networks. Register memory and attention mechanisms were both proposed to resolve the issue with either high computational cost to retain memory differentiability, or by discounting the RNN representation learning towards encoding shorte…
Deep network predicts action sequences for complex tasks from a scene image.
problem Scalable task and motion planning from initial scene images.
method Deep convolutional recurrent neural network that predicts action sequences.
result Predicts promising action sequences, reducing motion planning problems.
We describe a fully data driven model that learns to perform a retrosynthetic reaction prediction task, which is treated as a sequence-to-sequence mapping problem. The end-to-end trained model has an encoder-decoder architecture that consists of two recurrent neural networks, which has previously shown great success in…
LaMBO optimizes biological sequences using autoencoders and Bayesian optimization.
problem Bayesian optimization for drug design is limited by discrete, high-dimensional decision variables.
method Jointly trains denoising autoencoder with a Gaussian process head for gradient-based optimization in latent space.
result LaMBO outperforms genetic optimizers and requires no large pretraining corpus.
MARGE learns to reconstruct text by paraphrasing, achieving strong performance across multiple tasks.
problem Training sequence-to-sequence models with limited supervision.
method Unsupervised multi-lingual multi-document paraphrasing objective.
result Strong zero-shot performance on document translation and various tasks in multiple languages.
P3BO optimizes biological sequence design by combining multiple methods.
problem Variability in performance of black-box optimization methods for biological sequence design.
method Population-Based Black-Box Optimization (P3BO) that samples sequences from an ensemble of methods, weighting by past performance.
result P3BO outperforms individual methods, proposing higher quality and more diverse sequences.
Understanding optimal prompts for binary sequence predictors is challenging.
problem Finding good prompts for binary sequence predictors is difficult.
method Viewing prompting as finding the best conditioning sequence on a near-optimal sequence predictor, using empirical and statistical analysis.
result Optimal prompts can be better understood given the pretraining distribution, which is not usually available.
Many sequence-to-sequence generation tasks, including machine translation and text-to-speech, can be posed as estimating the density of the output y given the input x: p(y|x). Given this interpretation, it is natural to evaluate sequence-to-sequence models using conditional log-likelihood on a test set. However, the go…
In sequence learning tasks such as language modelling, Recurrent Neural Networks must learn relationships between input features separated by time. State of the art models such as LSTM and Transformer are trained by backpropagation of losses into prior hidden states and inputs held in memory. This allows gradients to f…
Effectively capturing graph node sequences in the form of vector embeddings is critical to many applications. We achieve this by (i) first learning vector embeddings of single graph nodes and (ii) then composing them to compactly represent node sequences. Specifically, we propose SENSE-S (Semantically Enhanced Node Seq…