Two attention models improve human activity recognition by focusing on important signals and sensor modalities.
problem Noise and unimportant signal components in recurrent networks for human activity recognition.
method Temporal and sensor attention mechanisms with continuity constraints.
result State-of-the-art results on three datasets, showing improved understandability and mean F1 score.
Analyzes self-attention in recurrent networks, proving it mitigates vanishing gradients.
problem Vanishing gradients in recurrent networks when capturing long-term dependencies.
method Formal analysis of self-attention's effect on gradient propagation, proposing a relevancy screening mechanism.
result Self-attention mitigates vanishing gradients in recurrent networks, providing guarantees.
CRAN extracts music highlights using attention and recurrent layers.
problem Extracting valuable music highlights from signals.
method Convolutional Recurrent Attention Networks (CRAN) with attention mechanism.
result CRAN outperforms three baseline methods in highlighting extraction.
RANP improves neural processes for sequential data.
problem Capturing temporal order and recurrent structure from sequential data.
method Incorporated ANP into a recurrent neural network.
result RANP outperforms NPs and LSTMs in 1D regression and autonomous-driving tasks.
A new neural network learns from acoustic scenes by suppressing irrelevant patterns.
problem Acoustic scenes are rich and redundant, making classification challenging.
method Spatio-temporal attention pooling layer coupled with a convolutional recurrent neural network.
result The method outperforms a strong convolutional neural network baseline and sets new state-of-the-art performance.
Improved electrical load forecasting model using Fourier-enhanced RNN.
problem Electrical load time series downscaling with high accuracy and low error.
method Combines recurrent neural network with Fourier seasonal embeddings and self-attention.
result Significantly reduces RMSE across different time horizons compared to existing methods.
RAU integrates attention into GRU for better sequence learning.
problem Lack of attention mechanism in GRU leads to information redundancy or loss.
method RAU adds an attention gate to GRU to adaptively focus on regions of interest.
result RAU consistently outperforms GRU and other methods in various tasks.
LARNN improves LSTM's performance with attention and multi-head mechanism.
problem Improving LSTM's performance in sequence prediction tasks.
method Derived from LSTM and RNN, LARNN uses attention and multi-head mechanism to query past states.
result LARNN achieves 91.92% test accuracy, significantly better than vanilla LSTM.
New insights into how encoder-decoder networks generate attention matrices.
problem Understanding how encoder-decoder networks use attention matrices.
method Decomposing hidden states into temporal and input-driven components.
result Attention matrices are formed based on task requirements, not architecture type.
Paper presents a neural network method for automated bug and ticket classification.
problem Automated classification of bug and ticket content in systems.
method Recurrent neural network with hierarchical attention mechanism.
result The method outperforms previous approaches on two datasets.
Transformer models outperform recurrent ones in modeling hierarchical data.
problem Modeling hierarchical structure in data.
method Introducing Multiresolution Transformer Networks leveraging self-attention.
result Multiresolution Transformer Networks significantly outperform state-of-the-art models on query suggestion datasets.
We replace the Hidden Markov Model (HMM) which is traditionally used in in continuous speech recognition with a bi-directional recurrent neural network encoder coupled to a recurrent neural network decoder that directly emits a stream of phonemes. The alignment between the input and output sequences is established usin…
The paper proposes a novel model to forecast patent citations using multi-attention recurrent networks.
problem Forecasting forward citations to patents to discover emerging technologies.
method The approach employs a sequence-to-sequence model with an attention-of-attention mechanism to capture dependencies in multiple time sequences.
result The proposed model outperforms state-of-the-art models in forward citation forecasting.
Sparse Attentive Backtracking selectively backpropagates long-term dependencies in recurrent networks.
problem Difficulty in learning long-term dependencies in BPTT due to computational impracticality and biased gradient estimates.
method Sparse Attentive Backtracking learns an attention mechanism over past hidden states and selectively backpropagates through high-weight paths.
result Model learns long-term dependencies with fewer backpropagation steps, addressing biased gradient issues.
New RNN model fuses sensor data from multiple stations.
problem Modeling distributed sensor networks for future behavior prediction.
method Multi-Encoder-Decoder RNN architecture with attention mechanism.
result Model improves prediction accuracy on real-world sensor datasets.
Convolutional-deconvolution networks can be adopted to perform end-to-end saliency detection. But, they do not work well with objects of multiple scales. To overcome such a limitation, in this work, we propose a recurrent attentional convolutional-deconvolution network (RACDNN). Using spatial transformer and recurrent …
Deep learning models predict ICU readmission with varying accuracy.
problem Predicting ICU readmission risk using deep learning architectures.
method Several deep learning architectures including attention-based models, recurrent layers, neural ODEs, and embeddings were trained on MIMIC-III data.
result Attention-based models with neural ODEs achieved highest predictive accuracy.
Neural network predicts cardiovascular events from EHRs with high accuracy.
problem Predicting onset of cardiovascular diseases from electronic health records.
method Multi-task gated recurrent units with attention mechanism.
result Model outperforms clinical risk scores in predicting stroke and myocardial infarction.
Study improves stock price prediction using advanced ML models.
problem Improving financial forecasting accuracy in stock markets.
method Evaluation of RNN architectures including LSTM, GRU, and attention-based models.
result Attention-based models outperform others in capturing complex dependencies.
AUTR generates sentences using a dynamic memory and attention mechanism.
problem Generating coherent sentences without explicit training data.
method Recurrent neural network with dynamic attention and canvas memory.
result AUTR learns meaningful latent representations and achieves competitive performance.
Hybrid model predicts flow and pressure in water systems.
problem Predicting flow and pressure in water distribution systems with complex spatial-temporal correlations.
method Hybrid dual-stage spatial-temporal attention-based recurrent neural networks (hDS-RNN).
result Our model outperformed 9 baseline models in flow and pressure series prediction.
Improved speech enhancement with MNTFA using time-frequency attention.
problem Speech enhancement with limited model size and memory.
method Designing MNTFA with self-attention modules for long sequences and joint training.
result MNTFA achieves better performance with fewer parameters than DPCRN.
Paper introduces methods to integrate external knowledge into RNNs using attention mechanisms.
problem Incorporating external knowledge into RNNs for improved performance.
method Proposes three methods: attentional concatenation, feature-based gating, and affine transformation.
result Attentional feature-based gating consistently improves performance across tasks.
New approach to understand recurrent policies as FSMs without minimization.
problem Minimization of FSMs obscures the semantics of policy decisions.
method Start with unminimized FSM, apply interpretable reductions, use attention tool.
result Reveals insights into policy decisions not previously noticed.
DAUM networks improve weather forecasting with attention and multistream features.
problem Improving weather forecasting accuracy using deep learning.
method Proposes DAUM networks combining tensorial unistream and multistream representations with attention mechanisms.
result Adding self-attention blocks enhances forecasting performance.
Paper proposes dual recurrent attention units for VQA models.
problem Comprehending visual and textual data for accurate question answering.
method Introduces and evaluates recurrent attention mechanisms in VQA models.
result Dual Recurrent Attention Units (RAUs) improve VQA performance.
Efficient graph generation with GRAN using attention and sampling.
problem Generating high-quality graphs efficiently.
method Graph Recurrent Attention Networks (GRAN) with attention mechanisms and sampling.
result State-of-the-art time efficiency and sample quality on benchmarks.
Study uses attention-based method to detect different types of online harassment.
problem Detecting different types of online harassment in social media content.
method Multi-attention based approach using Recurrent Neural Networks to address imbalanced data.
result Demonstrates effectiveness of attention-based mechanism for detecting various types of online harassment.
A new model uses attention and Gaussian processes for efficient time-series generation.
problem Computational inefficiency and uncertainty underestimation in sequence transduction.
method Attention-based Gaussian process network for real-valued sequence generation.
result The model improves training efficiency and learns factorized generative distribution.
Improved stock price prediction using attention modules and news sentiment.
problem Predicting stock prices with non-stationary and non-parametric data.
method α_{t}-RIM architecture with attention modules and exponentially smoothed recurrent neural network.
result The αt-RIM outperforms state-of-the-art models in predicting unseen data. Self-attention improves satellite time series classification without preprocessing.
problem Efficiently classifying raw satellite time series data.
method Comparison of deep learning models including self-attention, 1D-convolutions, recurrence, and random forest.
result Self-attention and recurrent neural networks outperform convolutional neural networks on raw satellite time series.
Paper improves Native ads CTR prediction using event embeddings and recurrent networks.
problem Hard CTR prediction for Native ads due to lack of direct query intent.
method Proposes a large-scale event embedding scheme and a recurrent neural network model.
result Significantly outperforms baseline and variants in CTR prediction.
LSTM-FCN improves time series classification with minimal model size increase.
problem Classifying time series sequences with high accuracy.
method Augmented fully convolutional networks with LSTM sub-modules and attention mechanism.
result LSTM-FCN achieves state-of-the-art performance.
Study improves forecasting in betting markets using novel neural networks.
problem Improving short-term price movement predictions in betting exchanges.
method Innovative convolutional attention mechanisms applied to recurrent neural networks and bi-dimensional layers.
result All proposed innovations positively impact classification task performance.
HRHN predicts time series by integrating exogenous data and temporal dynamics.
problem Challenges in predicting time series with exogenous data and temporal dynamics.
method Hierarchical attention-based Recurrent Highway Network (HRHN) that considers interactions among exogenous variables and temporal dynamics.
result HRHN outperforms state-of-the-art methods in time series prediction, especially in capturing sudden changes and oscillations.
RNN with attention detects system anomalies with high accuracy.
problem Lack of interpretability in deep learning models for system log anomaly detection.
method Recurrent Neural Network (RNN) with attention mechanisms.
result RNN with attention achieves high performance in anomaly detection.
Self-attention model improves HAR from wearable sensors.
problem Capturing spatio-temporal context from sensor data.
method Proposes a self-attention based neural network model.
result Significant performance improvement over state-of-the-art models.
FMA-ETA predicts travel time using FFN with attention.
problem Estimating travel time from spatial-temporal data.
method FFN with Multi-factor self-attention mechanism.
result FMA-ETA outperforms state-of-the-art methods in prediction accuracy with faster inference.
Proposes a multi-variable LSTM for accurate time series forecasting and variable importance.
problem Current attention mechanisms in recurrent neural networks fail to characterize variable importance in time series with exogenous variables.
method Develops a multi-variable LSTM with tensorized hidden states to learn variable importance and a mixture of temporal and variable attention.
result Demonstrates superior prediction performance and variable importance quantification compared to baselines.
Neural network model predicts alternating event-free periods.
problem Dynamic prediction of alternating recurrent events with statistical nuance.
method Developed an online dynamic prediction framework using neural network theory.
result Outstanding performance in predicting alternating recurrent event-free time.
Improved RNNs reduce memory decay and enhance language tasks.
problem Memory decay in RNNs affects performance in sequence prediction tasks.
method Introduced trainable scaling factors and a dependent bidirectional RNN to mitigate memory decay and improve performance.
result The proposed ELSTM and DBRNN models achieved up to 30% improvement in LAS compared to LSTM and GRU in dependency parsing.
Large-scale recurrent networks have drawn increasing attention recently because of their capabilities in modeling a large variety of real-world phenomena and physical mechanisms. This paper studies how to identify all authentic connections and estimate system parameters of a recurrent network, given a sequence of node …
Novel method segments heart sound signals using LSTMs with attention.
problem Heart sound segmentation for diagnostic applications.
method Bidirectional LSTMs with attention mechanisms.
result State-of-the-art performance on multiple benchmarks.
Proposes an interpretable LSTM for time series with exogenous variables.
problem Lack of variable importance characterization in recurrent neural networks.
method Develops a multi-variable LSTM with tensorized hidden states for learning variable-specific representations.
result Variable attention in real datasets is highly aligned with statistical causality.
Foresee predicts future road environments with neural networks.
problem Predicting future chaotic road environments for autonomous driving.
method Unidirectional gated recurrent units (GRUs) with attention for online training.
result Foresee outperforms state-of-the-art methods in future projection accuracy.
Study predicts blood pressure response to fluid bolus therapy with high accuracy.
problem Predicting successful response to fluid bolus therapy in hypotensive ICU patients.
method Used attention-based LSTM and GRU neural networks on a large ICU database.
result Stacked LSTM with attention mechanism achieved highest accuracy of 0.852.
EWS-GCN improves credit scoring by analyzing money transfer connections.
problem Improving credit scoring in transactional banking data.
method Edge Weight-Shared Graph Convolutional Network (EWS-GCN) combining graph and recurrent neural networks.
result EWS-GCN outperforms state-of-the-art models in credit scoring.
SeqSleepNet tackles automatic sleep staging as a sequence-to-sequence problem.
problem Automatic sleep staging as a sequence-to-sequence classification problem.
method End-to-end hierarchical recurrent neural network (SeqSleepNet) with filterbank and attention-based recurrent layers.
result SeqSleepNet achieves high accuracy (87.1% overall accuracy, 83.3% macro F1-score, 0.815 Cohen's kappa) on a publicly available dataset.