A new LSTM model reduces state updates and improves convergence for long sequences.
problem Vanishing gradient problem in RNNs and slow convergence on long sequences.
method Gaussian-gated LSTM (g-LSTM) with a time gate to control neuron updates.
result The g-LSTM model reduces state updates and computes by at least 10x compared to an equivalent LSTM.
Analyzes convergence rates for Gaussian-gated MoE model.
problem Theoretical understanding of Gaussian-gated MoE model is incomplete.
method Maximum likelihood estimation with novel Voronoi loss functions.
result MLE has distinct behaviors under different settings of Gaussian gating function parameters.
G-GLN extends GLNs to multiple regression and density modeling.
problem Learning features in deep neural networks.
method G-GLN uses a distributed and local credit assignment mechanism based on optimizing a convex objective.
result G-GLN achieves competitive or state-of-the-art performance on regression benchmarks.
Regularized MLE improves MoE models for high-dimensional data.
problem Modeling with high-dimensional predictors and feature selection.
method Gaussian gating network, ℓ1-regularized MLE, EM-Lasso algorithm, BIC-like criterion. result Regularized MLE outperforms standard MLE in clustering and regression tasks.
Paper proposes a novel method to accurately determine the number of experts in Gaussian-gated Gaussian MoE models.
problem Challenges in model selection for MoE models, especially with covariates.
method Introduces a novel extension using dendrograms of mixing measures to estimate the true number of mixture components.
result Achieves optimal convergence rates for parameter estimation and accurately approximates the regression function.
This paper tackles model selection for MoE models in high-dimensional data.
problem Model selection for Gaussian-gated localized MoE and block-diagonal covariance localized MoE regression models in high-dimensional data.
method Penalized maximum likelihood estimation framework with non-asymptotic risk bounds.
result Established non-asymptotic risk bounds for model selection in MoE models.
Proposes a new deep learning model for uncertainty quantification and propagation.
problem High-dimensional uncertainty quantification and propagation problems.
method Integrates U-net with Gaussian Gated Linear Network (GGLN) to create GLU-net.
result Less complex architecture with 44% fewer parameters than existing models.
Stacked LSTM improves weather forecasting accuracy by incorporating spatial information.
problem Improving temperature prediction accuracy in weather forecasting.
method 2-layer spatio-temporal stacked LSTM model with independent LSTM models per location in the first layer and combined hidden states in the second layer.
result The stacked LSTM model outperforms single LSTM models in most cases by utilizing spatial information.
Long short-term memory (LSTM) is normally used in recurrent neural network (RNN) as basic recurrent unit. However,conventional LSTM assumes that the state at current time step depends on previous time step. This assumption constraints the time dependency modeling capability. In this study, we propose a new variation of…
This paper compares LSTMs and attention mechanisms for financial time series forecasting.
problem Improving financial time series forecasting accuracy.
method Implemented an LSTM with attention mechanism and compared it to a standard LSTM.
result An LSTM with attention can outperform standalone LSTMs, but further investigation is needed.
Novel unsupervised LSTM method for video frame prediction.
problem Video frame prediction with limited labeled data.
method Inception-based LSTM architecture.
result Inception-based LSTM outperforms convolutional LSTM and has better performance than Inception v2 LSTM.
LSTM-FCN and ALSTM-FCN improve time series classification performance.
problem Improving time series classification performance.
method Ablation tests on LSTM-FCN and ALSTM-FCN, comparing z-normalizing techniques, dimension shuffle impact, and GRU replacement.
result LSTM and FCN blocks perform better together, and z-normalizing the whole dataset is more effective.
MC-LSTM extends LSTM to conserve mass in neural networks.
problem Conservation laws in real-world systems.
method Extending LSTM's inductive bias to conserve mass.
result MC-LSTM sets new state-of-the-art for predicting peak flows.
Proposes a model combining difference-attention and error-correction LSTMs for improved time series prediction.
problem Improving accuracy in time series prediction.
method Combines difference-attention LSTM and error-correction LSTM in a cascade approach.
result Improves prediction accuracy in time series.
Simplified LSTM models improve sentiment analysis on Twitter debate data.
problem Performing sentiment analysis on long sequence data from Twitter debates.
method Developed six parameter-reduced LSTM models (slim LSTM) for faster training and reduced computational cost.
result Slim LSTM models outperform standard LSTM model in sentiment analysis of GOP Debate Twitter dataset.
Enhanced LSTM model learns complex temporal dependencies.
problem Modeling long-term dependencies in sequential data.
method Gamma-LSTM with hierarchical memory units and gates.
result Gamma-LSTM outperforms regular and stacked LSTMs in sequence prediction.
Long short-term memory (LSTM) has been widely used for sequential data modeling. Researchers have increased LSTM depth by stacking LSTM cells to improve performance. This incurs model redundancy, increases run-time delay, and makes the LSTMs more prone to overfitting. To address these problems, we propose a hidden-laye…
Solves exploding and vanishing gradient problem in LSTMs.
problem Exploding and vanishing gradient problem in LSTM optimization.
method Introduces a simple stochastic algorithm (h-detach) to prevent suppression of gradient components through the cell state path in LSTM.
result Significant improvements in convergence speed, robustness, and generalization over vanilla LSTM training.
DA-LSTM adapts LSTM depth to non-uniform data, improving efficiency.
problem Non-uniform information distribution in sequential data cannot be accurately modeled by traditional LSTM.
method Developed DA-LSTM architecture that dynamically adjusts LSTM depth based on information distribution.
result DA-LSTM reduces computation resource usage and convergence time by 41.78% and 46.01% respectively.
Enhanced LSTM with multiple kernels and attention improves video action recognition.
problem Improving motion understanding in video analysis.
method Proposed a Network-in-LSTM approach with multiple convolutional kernels and layers, and an attention-based mechanism.
result Improves accuracy in supervised classification on UCF-101 and Sports-1M datasets.
EA-LSTM improves LSTM for time series prediction by evolving attention.
problem LSTMs struggle with assigning varying attention to sub-windows in time series data.
method Evolutionary attention-based LSTM with competitive random search.
result EA-LSTM achieves competitive performance in multivariate time series prediction.
BiLSTM models outperform LSTM and ARIMA in financial time series forecasting.
problem Comparing forecasting accuracy of ARIMA, LSTM, and BiLSTM models.
method Behavioral analysis and comparison of BiLSTM and LSTM models.
result BiLSTM models provide better predictions than ARIMA and LSTM models.
Because of their effectiveness in broad practical applications, LSTM networks have received a wealth of coverage in scientific journals, technical blogs, and implementation guides. However, in most articles, the inference formulas for the LSTM network and its parent, RNN, are stated axiomatically, while the training fo…
The driving force behind the recent success of LSTMs has been their ability to learn complex and non-linear relationships. Consequently, our inability to describe these relationships has led to LSTMs being characterized as black boxes. To this end, we introduce contextual decomposition (CD), an interpretation algorithm…
LSTMs show surprising few-shot learning ability, improving on MAML.
problem Few-shot learning with limited data.
method Revisited LSTM approach with Outer Product LSTM (OP-LSTM).
result OP-LSTM outperforms MAML on simple few-shot tasks, but not complex ones.
Paper questions RNN and LSTM's long-term memory and introduces a new definition.
problem Whether RNN and LSTM have long-term memory.
method Introduced a new definition of long-term memory and modified RNN and LSTM to test it.
result RNN and LSTM do not meet the new definition of long-term memory.
We present two simple ways of reducing the number of parameters and accelerating the training of large Long Short-Term Memory (LSTM) networks: the first one is "matrix factorization by design" of LSTM matrix into the product of two smaller matrices, and the second one is partitioning of LSTM matrix, its inputs and stat…
DP-LSTM predicts stock prices using financial news with improved accuracy and privacy.
problem Predicting stock prices with financial news articles.
method Integrates financial news articles into a sentiment-ARMA model, then uses an LSTM network with differential privacy.
result Achieves up to 65.79% improvement in MSE for S&P 500 prediction.
A new framework improves LSTM performance without adding more parameters.
problem Improving LSTM performance without increasing model complexity.
method A unifying framework of bilinear LSTMs that balances hidden state vector size and weight matrix approximation quality.
result Bilinear LSTMs achieve superior performance compared to linear LSTMs without additional parameters.
LSTM models improve macroeconomic forecasting with mixed frequency data.
problem Improving accuracy of macroeconomic forecasts using mixed frequency data.
method Adapted LSTM model to mixed frequency data, using U-MIDAS scheme.
result Proposed LSTM models outperform conventional MIDAS models in out-of-sample predictive performance.
DEKF maintains stability in LSTM learning with bounded perturbations.
problem Stability of DEKF in LSTM-based online learning.
method Modeling DEKF as a perturbed extended Kalman filter and deriving stability conditions.
result DEKF learns LSTM parameters with similar stability properties to the global extended Kalman filter.
Improved LSTM for industrial flow prediction with higher accuracy.
problem Predicting continuous time series variables in industrial settings.
method LSTM algorithm with multivariate tuning, incorporating periodic measurement and time window concepts.
result Significantly improved prediction accuracy (54.05% higher than traditional LSTM).
LSTMs improve bond yield forecasting with unique signals.
problem Improving bond yield forecasting accuracy.
method Long short-term memory (LSTM) networks with sequence-to-sequence architectures and LSTM-LagLasso methodology.
result Univariate LSTM models with additional memory can achieve similar results as multivariate MLP models using exogenous information.
Hybridizes LSTM and HMM for improved language model performance.
problem Improving the performance of LSTM and HMM models.
method Analyzes hidden state structures, compares predictive accuracy and hidden state output of LSTM and HMM.
result HMM can serve as an appropriate approximation of LSTM model.
DALC customizes LSTM models for detectors in large-scale traffic networks.
problem Fine-grained traffic prediction for large-scale transportation networks.
method Formulated as a finite Markov decision process, introduced ALC algorithm for automatic customization, and developed DALC for distributed customization.
result DALC provides higher prediction accuracy than Apache Spark MLlib approaches.
Recurrent neural networks like long short-term memory (LSTM) are important architectures for sequential prediction tasks. LSTMs (and RNNs in general) model sequences along the forward time direction. Bidirectional LSTMs (Bi-LSTMs) on the other hand model sequences along both forward and backward directions and are gene…
Stacked LSTM networks improve traffic volume forecasting.
problem Accurate traffic volume prediction for better planning.
method Applying stacked Long Short-Term Memory (LSTM) networks for time series forecasting.
result Stacked LSTM networks enhance the accuracy of traffic volume predictions.
This study provides benchmarks for different implementations of LSTM units between the deep learning frameworks PyTorch, TensorFlow, Lasagne and Keras. The comparison includes cuDNN LSTMs, fused LSTM variants and less optimized, but more flexible LSTM implementations. The benchmarks reflect two typical scenarios for au…
Extended LSTM improves language modeling performance.
problem Improving LSTM for large-scale language modeling.
method Exponential gating, modified memory structure, and residual stacking.
result xLSTM outperforms state-of-the-art models in performance and scaling.
MRC-LSTM predicts Bitcoin prices using CNN and LSTM.
problem Predicting Bitcoin price with high volatility and complex factors.
method Combines MRC and LSTM, focusing on multi-scale features and long-term dependencies.
result MRC-LSTM significantly outperforms other models in Bitcoin price prediction.
Paper shows how LSTM can remember long sequences by attending to persisted information.
problem LSTMs struggle with long sequences due to fading information and bias towards recent data.
method The paper introduces a mechanism that allows LSTMs to attend to information in memory based on how long it was persisted by the gating mechanism.
result The method improves LSTM's ability to process long sequences by retrieving information proportionally to its persistence in memory.
Study compares LSTM and Transformer models in financial time series prediction.
problem Comparing LSTM and Transformer models for financial time series prediction.
method Various LSTM-based and Transformer-based models compared on financial tasks; DLSTM and new Transformer architecture designed.
result Transformer-based models show limited advantage in absolute price sequence prediction, while LSTM-based models perform better on difference sequences.
DAG-LSTM improves DA classification in group chats.
problem DA classification in multi-party conversations.
method Directed-Acyclic-Graph LSTM (DAG-LSTM) exploiting turn-taking structure.
result DAG-LSTM outperforms existing methods by 0.8% in accuracy and 1.2% in macro-F1 score.
Paper proposes using LSTM for LSH-based sequence alignment.
problem Sequence alignment using deep learning models.
method Deep bidirectional LSTM for feature learning and LSH-based sequence alignment.
result Higher accuracy achieved with LSTM-based model.
Predicting the price correlation of two assets for future time periods is important in portfolio optimization. We apply LSTM recurrent neural networks (RNN) in predicting the stock price correlation coefficient of two individual stocks. RNNs are competent in understanding temporal dependencies. The use of LSTM cells fu…
In this paper, we use the house price data ranging from January 2004 to October 2016 to predict the average house price of November and December in 2016 for each district in Beijing, Shanghai, Guangzhou and Shenzhen. We apply Autoregressive Integrated Moving Average model to generate the baseline while LSTM networks to…
Study predicts cryptocurrency trends using LSTM model.
problem Predicting cryptocurrency price trends.
method Combination of window-sliding and prediction range method with LSTM model.
result Established model for cryptocurrency price trend prediction.
We analyse multimodal time-series data corresponding to weight, sleep and steps measurements. We focus on predicting whether a user will successfully achieve his/her weight objective. For this, we design several deep long short-term memory (LSTM) architectures, including a novel cross-modal LSTM (X-LSTM), and demonstra…