Improves sequential recommendation with relation-aware self-attention.
problem Improving accuracy in sequential recommendation.
method Integrates Transformer's self-attention mechanism with a probabilistic model of recommendation context.
result Significant improvements over recent baseline models.
R-SQAIR adds relational bias to sequential object attention models for better object interactions.
problem Traditional sequential multi-object attention models struggle with relational inferences.
method Proposes R-SQAIR, a relational extension of SQAIR with a parallel pairwise interaction module.
result Demonstrates gains in object relations and combinatorial generalization over sequential mechanisms.
MEANTIME improves sequential recommendation by using multi-temporal embeddings and attention mechanisms.
problem Limited use of timestamp information and information bottleneck in sequential recommendation models.
method MEANTIME employs multiple types of temporal embeddings and attention mechanisms to capture diverse patterns from user behavior sequences.
result MEANTIME outperforms state-of-the-art sequential recommendation methods.
RANP improves neural processes for sequential data.
problem Capturing temporal order and recurrent structure from sequential data.
method Incorporated ANP into a recurrent neural network.
result RANP outperforms NPs and LSTMs in 1D regression and autonomous-driving tasks.
New model mimics neural next item recommendation using Hankel matrices.
problem Next item recommendation efficiency and structural knowledge capture.
method Tensor factorization with Hankel matrix representation.
result Model performs competitively with neural networks but is simpler.
New attention mechanism improves meta-transfer learning in dynamic tasks.
problem Underfitting in meta-transfer learning with dynamic tasks.
method Proposed Recurrent Memory Reconstruction (RMR) attention mechanism.
result ASNP-RMR significantly outperforms baselines in various tasks.
This paper proposes an attention module augmented relational network called SARN(Sequential Attention Relational Network) that can carry out relational reasoning by extracting reference objects and making efficient pairing between objects. SARN greatly reduces the computational and memory requirements of the relational…
Introduces recency bias to improve time-series forecasting.
problem Lack of recency bias in standard Transformer attention for time-series data.
method Reweights attention scores with a smooth heavy-tailed decay to emphasize nearby observations.
result Recency-biased attention consistently improves sequential modeling and achieves competitive performance on time-series forecasting benchmarks.
GAT-AGNN learns stock trends using graph and attention mechanisms.
problem Predicting dynamic stock trends in a complex market.
method Sequential graph structure with attention mechanisms.
result GAT-AGNN outperforms state-of-the-art methods in stock trend prediction.
This study examines how sequential correlations affect in-context learning in sequence models.
problem Understanding how in-context learning works with sequentially correlated data.
method Extended linear regression model to sequentially correlated data, tested on transformer architectures.
result Sequential correlations alter the effective context length and attention architecture effectiveness.
Bi-LSTM with attention generates jazz music with rich nuances.
problem Challenges in producing realistic music with structure and rationality.
method Deep learning, Bi-directional Long Short Term Memory (Bi-LSTM) Neural Network with Attention.
result Bi-LSTMs with attention preserve the richness and technical nuances of old style music.
In (Yang et al. 2016), a hierarchical attention network (HAN) is created for document classification. The attention layer can be used to visualize text influential in classifying the document, thereby explaining the model's prediction. We successfully applied HAN to a sequential analysis task in the form of real-time m…
RAN model recognizes multiple activities from unlabeled sensor data.
problem Handling weakly labeled multi-activity data from wearable sensors.
method Recurrent Attention Networks (RAN) for sequential multi-activity recognition and localization.
result RAN model can infer multiple activities and determine activity locations from unlabeled data.
SANST uses self-attentive networks with spatial and temporal embeddings for better POI recommendations.
problem Next point-of-interest (POI) recommendation for users based on their history.
method SANST incorporates spatio-temporal patterns into self-attentive networks.
result SANST outperforms state-of-the-art models by up to 13.65% in nDCG@10.
A new feature selection method using attention for neural networks.
problem Feature selection for neural networks with a budget constraint.
method Sequential Attention: greedy forward selection with attention weights.
result Achieves state-of-the-art results for neural networks.
CoT improves transformer sample efficiency by reducing input token dependencies and attention sparsity.
problem Transformer sample inefficiency in simple tasks.
method Demonstrated through parity-learning setup, showing CoT reduces required samples from exponential to polynomial.
result Transformer learns function within polynomial samples with CoT, requiring exponential samples without CoT.
SeqAttnGAN generates interactive images based on multi-turn text descriptions.
problem Interactive image editing with multi-turn textual commands.
method SeqAttnGAN uses a neural state tracker and GAN framework for sequential image generation and refinement.
result SeqAttnGAN outperforms state-of-the-art models on interactive image editing tasks.
New method embeds time span into self-attention for better temporal pattern recognition.
problem Capturing temporal patterns in event sequences without recurrent networks.
method Functional time representation learning with Bochner's and Mercer's Theorems.
result Proposed methods outperform baseline models in various continuous-time event sequence prediction tasks.
A new transformer model uses Monte Carlo methods for sequence prediction.
problem Predicting sequences from observations with uncertainty.
method Integrates Monte Carlo methods into a transformer architecture to model stochastic sequences.
result Proposes a generative model with predictive distributions.
The paper proposes a new method to improve microcredit decisions by modeling sequential loan interactions.
problem Improving microcredit decision-making by addressing population bias and model generalization.
method The authors introduce a multi-stage interaction sequence (MSIS) method that models sequential loan interactions and uses a hierarchical attention module to leverage interaction information.
result The MSIS method effectively remedies population bias and improves model generalization on a real loan data set.
Music highlights are valuable contents for music services. Most methods focused on low-level signal features. We propose a method for extracting highlights using high-level features from convolutional recurrent attention networks (CRAN). CRAN utilizes convolution and recurrent layers for sequential learning with an att…
Diffusion Transformer captures spatial-temporal dependencies in sequential data.
problem Capturing rich spatial and temporal dependencies in sequential data.
method Established theoretical guarantees for diffusion transformers learning Gaussian process data.
result Spatial-temporal dependencies are captured within attention layers of diffusion transformers.
New interpretation of attention in Transformers and Graph Attention Networks.
problem Understanding and improving attention mechanisms in deep learning models.
method Decomposed attention into a kernel and a normalization term; generalized the kernel function and norm.
result Generalized attention leads to better performance on various tasks.
FMT model improves multimodal sequential learning across language, vision, and acoustic data.
problem Modeling spatio-temporal dynamics across multiple modalities.
method Factorized Multimodal Transformer (FMT) that models intramodal and intermodal dynamics in a factorized manner.
result FMT outperforms existing models on 3 datasets and 21 labels, setting new state of the art.
Study SGD dynamics in sequence models, revealing training phases and influence of sequence length.
problem Understanding SGD in sequence models like attention networks.
method Derived closed-form population loss and analyzed SGD dynamics for SSI models.
result Two distinct training phases: escape from uninformative initialization and alignment with target subspace.
Self-attention prefers sparse functions of input sequences, reducing sample complexity.
problem Understanding the inductive biases of self-attention in modeling long-range dependencies.
method Theoretical analysis and synthetic experiments to probe sample complexity of learning sparse functions with Transformers.
result Bounded-norm Transformer networks can represent sparse functions of the input sequence with logarithmic sample complexity.
CDSA uses self-attention to impute missing values in multivariate, geo-tagged time series data.
problem Missing values in multivariate, geo-tagged time series data.
method Cross-Dimensional Self-Attention (CDSA) for sequence modeling across time, location, and sensor measurements.
result CDSA outperforms state-of-the-art methods in imputation and forecasting on real-world datasets.
Improved Transformer language models using dynamic evaluation.
problem Language model perplexity and accuracy improvements.
method Combining Transformers with dynamic evaluation techniques.
result Significant improvement in language model performance (e.g., 0.99 to 0.94 bits/char).
HCRNN uses hierarchical contexts to improve recommendation models.
problem Challenges in modeling user interest transitions and drifts in recommendation systems.
method Introduces HCRNN with three hierarchical contexts (global, local, temporary) and a hierarchical context-based gate structure.
result HCRNN outperformed other models in sequential recommendation tasks.
Traffic flow prediction is crucial for urban traffic management and public safety. Its key challenges lie in how to adaptively integrate the various factors that affect the flow changes. In this paper, we propose a unified neural network module to address this problem, called Attentive Crowd Flow Machine~(ACFM), which …
Partially observable environments present an important open challenge in the domain of sequential control learning with delayed rewards. Despite numerous attempts during the two last decades, the majority of reinforcement learning algorithms and associated approximate models, applied to this context, still assume Marko…
New approach improves linear-time attention for language models.
problem Challenges of quadratic attention in long-sequence modelling, especially for discrete data.
method Reinterpreting linear attention through latent probabilistic graphical models, introducing asymmetric structure and recurrent parameterisation.
result Our model achieves competitive performance and outperforms existing linear attention variants on language modelling benchmarks.
A reinforcement learning model uses attention to make actions interpretable.
problem Creating interpretable reinforcement learning models.
method Soft attention mechanism to focus on task-relevant information.
result The model achieves competitive performance on Atari tasks and is interpretable.
A novel CAB-XDE framework predicts speculative stock prices with high accuracy.
problem Forecasting speculative stock prices in volatile markets.
method Customized attention BiLSTM with XGBoost, integrating attention mechanism and weight determination theory-error reciprocal method.
result Empirically validated with MAPE of 0.0037, MAE of 84.40, and RMSE of 106.14.
Analyzes self-attention in recurrent networks, proving it mitigates vanishing gradients.
problem Vanishing gradients in recurrent networks when capturing long-term dependencies.
method Formal analysis of self-attention's effect on gradient propagation, proposing a relevancy screening mechanism.
result Self-attention mitigates vanishing gradients in recurrent networks, providing guarantees.
Transformers adapted to spherical geometry using space-filling curves.
problem Generalizing transformers to geometric domains like spheres.
method Attention heads following a space-filling curve.
result Introduction of the Spiroformer on a 2-sphere.
Paper introduces regularization for multi-head attention to spot keywords.
problem Redundancy in multi-head attention leads to lack of rich information.
method Regularization technique to enforce orthogonality between attention heads.
result Significant improvement in keyword spotting performance.
TabNet learns tabular data with attention, improving performance and interpretability.
problem Efficient learning and interpretability in tabular data.
method Sequential attention to feature selection, self-supervised learning.
result Outperforms other models on tabular datasets, interpretable feature attributions.
There has recently been significant interest in hard attention models for tasks such as object recognition, visual captioning and speech recognition. Hard attention can offer benefits over soft attention such as decreased computational cost, but training hard attention models can be difficult because of the discrete la…
SigGPDE scales sparse Gaussian processes for sequential data.
problem Predicting and quantifying uncertainty in sequential data.
method Sparse variational inference framework for Gaussian Processes, leveraging GP signature kernel gradients as PDE solutions.
result Significant computational gains and state-of-the-art performance on large sequential datasets.
This paper reviews methods for interpreting deep learning models with sequential data.
problem Limited interpretability of deep learning models in sequential data domains.
method Reviews and compares techniques for sequential interpretability.
result Current techniques have limitations and future research is needed.
Fast Bayesian inference with adaptable priors for real-time applications.
problem Intractable exact posterior computation limits Bayesian inference's adoption.
method Distribution Transformer architecture that learns mappings between priors and posteriors.
result Significant reduction in computation time from minutes to milliseconds.
SeqFM models dynamic and sequential features for better predictive analytics.
problem Inadequate handling of sequential dependencies in existing FM-based models.
method Introduces SeqFM, a novel model that incorporates multi-view self-attention to model static, dynamic, and their interactions.
result SeqFM outperforms existing models in ranking, classification, and regression tasks on six large-scale datasets.
Recurrent Neural Network (RNN) has been successfully applied in many sequence learning problems. Such as handwriting recognition, image description, natural language processing and video motion analysis. After years of development, researchers have improved the internal structure of the RNN and introduced many variants…
CDM models counterfactual outcomes in longitudinal data with improved accuracy.
problem Predicting counterfactual outcomes in longitudinal data with complex time-dependent confounding.
method Causal Diffusion Model (CDM) using denoising diffusion architecture with relational self-attention.
result CDM outperforms state-of-the-art methods in generating full probabilistic distributions of counterfactual outcomes.
Self-attentive network improves emotion recognition in conversations.
problem Emotion recognition in dyadic conversations using deep learning.
method Introduces a novel self-attention mechanism for capturing temporal dynamics without a decoder.
result Outperforms state-of-the-art alternatives on the IEMOCAP benchmark.
The study reveals the spectral structure of attention layers and its implications for generalization.
problem Understanding the spectral structure and generalization of trained attention layers.
method Empirical risk minimization in a single-head tied-attention layer, using random matrix theory, spin-glass theory, and approximate message passing.
result Exact high-dimensional characterization of training and test error, interpolation and recovery thresholds, and spectrum of the key and query matrices.
Study integrates attentional and spacing factors to improve category learning models.
problem Understanding the impact of training sequences on category learning.
method Introduced a novel integration of attentional factors and spacing into logistic knowledge tracing models.
result Enhanced model predicts students' learning outcomes better than existing models.