Batch-normalized RHN improves gradient control in recurrent networks.
problem Gradient vanishing or exploding in recurrent networks.
method Batch normalization applied at each recurrence loop in RHN.
result Batch-normalized RHN converges faster and performs better.
Transfer learning improves highway traffic forecasting using graph neural networks.
problem Lack of historical data for traffic forecasting on large highway networks.
method Developed a transfer learning approach for DCRNN, a graph neural network for highway forecasting.
result TL-DCRNN can forecast traffic on unseen regions of the highway network with high accuracy.
Graph-partitioning-based DCRNN improves traffic forecasting for large highways.
problem Challenges in accurately forecasting traffic on large highway networks.
method Graph-partitioning method to decompose large networks into smaller, independent networks.
result Demonstrated improved traffic forecasting on a large California highway network.
Recurrent Neural Networks (RNNs) play a major role in the field of sequential learning, and have outperformed traditional algorithms on many benchmarks. Training deep RNNs still remains a challenge, and most of the state-of-the-art models are structured with a transition depth of 2-4 layers. Recurrent Highway Networks …
Paper describes a video surveillance system for highway traffic events.
problem Detecting specific sequences of situations in highway traffic videos.
method Compares RNN and CNN architectures for analyzing video frames and sequences.
result Best architecture performs well in real conditions.
Time series prediction has been studied in a variety of domains. However, it is still challenging to predict future series given historical observations and past exogenous data. Existing methods either fail to consider the interactions among different components of exogenous variables which may affect the prediction ac…
A major contributing factor to the recent advances in deep neural networks is structural units that let sensory information and gradients to propagate easily. Gating is one such structure that acts as a flow control. Gates are employed in many recent state-of-the-art recurrent models such as LSTM and GRU, and feedforwa…
Despite all the impressive advances of recurrent neural networks, sequential data is still in need of better modelling. Truncated backpropagation through time (TBPTT), the learning algorithm most widely used in practice, suffers from the truncation bias, which drastically limits its ability to learn long-term dependenc…
Study compares deep learning models for traffic forecasting, highlighting graph elements' impact.
problem Challenges in forecasting spatial-temporal traffic patterns.
method In-depth comparative study of four deep neural network models with different basic elements.
result Graph attention improves long-term predictions in traffic forecasting models.
Neural Machine Translation (MT) has reached state-of-the-art results. However, one of the main challenges that neural MT still faces is dealing with very large vocabularies and morphologically rich languages. In this paper, we propose a neural MT system using character-based embeddings in combination with convolutional…
We describe a simple neural language model that relies only on character-level inputs. Predictions are still made at the word-level. Our model employs a convolutional neural network (CNN) and a highway network over characters, whose output is given to a long short-term memory (LSTM) recurrent neural network language mo…
Deep Recurrent Q-Network improves autonomous driving in urban areas with pedestrians.
problem Challenges in urban autonomous driving due to complex road structures and unpredictable pedestrian behavior.
method Combines Deep Q-Network with LSTM for long-term memory, designed a 3-D state representation, and uses a reward function.
result The proposed DRQN-based approach outperforms rule-based methods in dense urban scenarios.
We present PredRNN++, an improved recurrent network for video predictive learning. In pursuit of a greater spatiotemporal modeling capability, our approach increases the transition depth between adjacent states by leveraging a novel recurrent unit, which is named Causal LSTM for re-organizing the spatial and temporal m…
GHNet improves graph learning by balancing homogeneity and heterogeneity.
problem Over-smoothing in GCN leads to similar node representations.
method GHNet uses gating units to balance homogeneity and heterogeneity in feature propagation.
result GHNet achieves larger receptive fields without over-smoothing.
Proposes MSFCN and RFCN for video semantic segmentation improving accuracy by 9-15%.
problem Lack of temporal information in semantic segmentation algorithms for videos.
method Integrates motion cues and temporal consistency using recurrent and multi-stream FCN architectures.
result MSFCN-3 achieves significant accuracy improvements in various datasets.
Gating is a key technique used for integrating information from multiple sources by long short-term memory (LSTM) models and has recently also been applied to other models such as the highway network. Although gating is powerful, it is rather expensive in terms of both computation and storage as each gating unit uses a…
Study uses vehicle trajectory data to predict traffic incidents on highways.
problem Early detection of traffic incidents to reduce secondary crashes.
method Machine learning algorithms (Logistic Regression, Random Forest, Extreme Gradient Boost, Artificial Neural Network) applied to vehicle trajectory data.
result Random Forest model performs best for incident prediction.
Although there has been substantial research in software analytics for effort estimation in traditional software projects, little work has been done for estimation in agile projects, especially estimating user stories or issues. Story points are the most common unit of measure used for estimating the effort involved in…
Nontrivial connectivity has allowed the training of very deep networks by addressing the problem of vanishing gradients and offering a more efficient method of reusing parameters. In this paper we make a comparison between residual networks, densely-connected networks and highway networks on an image classification tas…
This paper uses machine learning to estimate how different types of crashes affect highway traffic.
problem Estimating the heterogeneous causal effects of crashes on highway traffic.
method Neyman-Rubin Causal Model, Conditional Shapley Value Index, Structural Causal Model, Doubly Robust Learning.
result Different types of crashes have varying impacts on traffic, with rear-end crashes causing the most severe congestion.
Deep RL for autonomous highway driving avoids unexpected scenarios.
problem Unexpected scenarios in AV operation space lead to poor decision-making.
method Deep reinforcement learning with safety checks for decision-making.
result Enhanced learning efficiency and safe behavior in highway driving.
HighD dataset captures naturalistic vehicle behavior on German highways for automated driving validation.
problem Current measurement methods fail to meet requirements for scenario-based validation of highly automated vehicles.
method Aerial perspective data collection fulfilling naturalistic behavior, quantity, and variety requirements.
result 16.5 hours of measurements from six locations, 110,000 vehicles, 45,000 km driven, 5600 lane changes.
The study compares different neural network architectures for option pricing accuracy and training time.
problem Evaluating the impact of network architectures on option pricing accuracy and training time.
method Empirical investigation of various neural network architectures (plain feed forward, highway, DGM) on option pricing problems.
result Generalized highway network architecture achieves the best performance in terms of mean squared error and training time.
Improved crash prediction model using deep belief network.
problem Time-consuming calibration process in crash prediction.
method Regularized deep belief network with two training steps.
result Improved prediction power with reduced human intervention.
This study identifies RwD crash patterns on rural two-lane highways under different lighting conditions.
problem Insufficient investigation of RwD crashes under varying lighting conditions.
method Data mining using association rules mining (ARM) on crash database.
result Interesting crash patterns and risk factors identified under different lighting conditions.
Interneurons improve learning in neural networks by accelerating convergence.
problem Rapid adaptation to changing input statistics in neural networks.
method Two mathematically tractable recurrent linear neural networks were compared: one with direct recurrent connections and the other with interneurons that mediate recurrent communication.
result The network with interneurons converges more quickly than the network with direct recurrent connections, scaling logarithmically with initialization spectrum.
Unified recurrent networks reveal differences in complexity levels of grammars.
problem Understanding the complexity and behavior of recurrent networks.
method Connecting recurrent networks with deterministic finite automata and formal grammars.
result Unified recurrent networks improve performance and match grammars from different complexity levels.
The paper examines how neural network topology affects adversarial robustness.
problem Understanding how neural network topology influences adversarial robustness.
method Investigated the graph of input traversing all layers of a neural network, comparing clean and adversarial inputs.
result Under-optimized edges in neural network graphs are a source of adversarial vulnerability and can be used to detect adversarial inputs.
This paper analyzes the generalization risk of unrolled neural networks using Stein's Unbiased Risk Estimator.
problem Analyzing the generalization risk of unrolled neural networks and its relationship to network design and train sample size.
method Using Stein's Unbiased Risk Estimator (SURE), the paper analyzes the generalization risk with bias and variance components for recurrent unrolled networks, focusing on the degrees-of-freedom (DOF) component and the trace of the end-to-end network Jacobian.
result DOF is well-approximated by the weighted path sparsity of the network under incoherence conditions on the trained weights, and DOF increases with train sample size and converges to the generalization risk for both recurrent and non-recurrent schemes.
Recurrent neural networks improve time series forecasting accuracy.
problem Time series forecasting is challenging, especially for sequential data.
method A recurrent neural network framework for feature engineering, prediction, and evaluation is presented.
result The LSTM and GRU networks outperform traditional methods in forecasting accuracy.
Paper proposes a statistical approach for predicting lane changes in highway scenarios.
problem Early recognition of lane changes for automated driving.
method Prototype trajectories generated from real data using Agglomerative Hierarchical Clustering. Maneuver prediction via Boosted Decision Trees and mixture model.
result Improved performance for lane change and trajectory prediction compared to a reference approach.
We introduce Neural Choice by Elimination, a new framework that integrates deep neural networks into probabilistic sequential choice models for learning to rank. Given a set of items to chose from, the elimination strategy starts with the whole item set and iteratively eliminates the least worthy item in the remaining …
RecNets use RNNs to process image channels in a compact, recurrent way.
problem Creating efficient neural network architectures for computer vision.
method Introducing RecNets with CRC layers that simulate recurrent processing of image channels.
result RecNets achieve superior size-accuracy trade-off compared to other compact models.
This work optimizes reservoir computing models by linking recurrence and non-linear dynamics.
problem Understanding how recurrence and non-linear dynamics in cortical networks contribute to their function.
method Transformed time-continuous, recurrent dynamics into an effective feed-forward structure of linear and non-linear temporal kernels.
result Optimal time-series classifiers can be built from random reservoir networks, demonstrating significant performance gains.
Bayesian approach improves neural network recurrence.
problem Improving neural network recurrence mechanisms.
method Introducing Bayesian recurrence relations and gates.
result Bayesian approach can perform as well as or better than conventional recurrent networks.
Stability is a fundamental property of dynamical systems, yet to this date it has had little bearing on the practice of recurrent neural networks. In this work, we conduct a thorough investigation of stable recurrent models. Theoretically, we prove stable recurrent neural networks are well approximated by feed-forward …
A hybrid CNN with shared parameters learns to incorporate recurrent structures.
problem Designing efficient recurrent neural networks.
method Parameter sharing scheme combining CNN and recurrent layers.
result Substantial parameter savings with competitive accuracy.
QRPNNs use quaternion-valued recurrent correlation neural networks to solve cross-talk issues.
problem Cross-talk problem in QRCNNs.
method Combining non-local projection learning with QRCNNs.
result QRPNNs exhibit greater storage capacity and noise tolerance.
Paper presents FPGA implementation for efficient recurrent neural networks.
problem Implementing recurrent neural networks on FPGAs for low latency.
method Developed hls4ml framework to implement LSTM and GRU layers.
result Demonstrated effective designs for both small and large models.
New method prunes recurrent networks efficiently, improving performance.
problem Pruning recurrent neural networks (RNNs) is challenging and often leads to poor performance.
method Data-efficient pruning objective derived from the spectrum of the recurrent Jacobian.
result 95% sparse GRUs significantly improve on existing baselines.
Leveraging advances in variational inference, we propose to enhance recurrent neural networks with latent variables, resulting in Stochastic Recurrent Networks (STORNs). The model i) can be trained with stochastic gradient methods, ii) allows structured and multi-modal conditionals at each time step, iii) features a re…
Scalable verifier for recurrent neural networks using polyhedral abstractions.
problem Certifying the correctness of recurrent neural networks.
method Combining sampling, optimization, and Fermat's theorem for polyhedral abstractions; gradient descent for refinement.
result Successfully verified challenging recurrent models in various domains.
Paper uses DMD to embed time in spatiotemporal forecasting.
problem Forecasting long-range seasonal dependencies in spatiotemporal data.
method Dynamic Mode Decomposition (DMD) for time representation.
result DMD-based embedding improves long-horizon forecasting accuracy.
Linear Memory Network separates memory and function in RNNs.
problem Complex transduction problems requiring memory and input-output exploitation.
method Conceptual separation between memory and function, using feedforward and autoencoder components.
result Efficient training and competitive performance on polyphonic music datasets.
SnAp approximates RTRL for online training of sparse recurrent networks.
problem Training large sparse recurrent networks online is computationally expensive.
method Sparse n-step Approximation (SnAp) of the RTRL influence matrix.
result SnAp with n=2 remains tractable for highly sparse networks and outperforms backpropagation through time.
Study on recurrent neural networks' feature selection and memorization using F1B test.
problem Conflict between feature selection and memorization in sequence learning.
method Flagged-1-Bit (F1B) test, four recurrent network models studied analytically and experimentally.
result Conflict can be resolved by gating mechanism or increasing state dimension.
Generalizes memory and forecasting capacities for nonlinear recurrent networks with dependent inputs.
problem Understanding memory and forecasting capabilities in networks with dependent inputs.
method Formulated bounds for memory and forecasting capacities in terms of network size and input properties.
result Proved that memory capacity for linear recurrent networks with independent inputs is given by the rank of the controllability matrix.
Recurrent-DBN models dynamic relational data with interpretable latent structures.
problem Interpreting dynamic relational data with hidden structures.
method Recurrent Dirichlet Belief Network framework with hierarchical latent structures and efficient inference strategy.
result Recurrent-DBN discovers interpretable latent structures and improves link prediction.