LSTM improves cross-network recommendations by capturing user preference changes and irregular time intervals.
problem Offline cross-network recommender solutions fail to capture user preference changes and dynamic environments.
method Proposes a multi-layered LSTM network with attention mechanisms, higher order interactions, and time-aware gates.
result The model consistently outperforms state-of-the-art in accuracy, diversity, and novelty.
Established recurrent neural networks are well-suited to solve a wide variety of prediction tasks involving discrete sequences. However, they do not perform as well in the task of dynamical system identification, when dealing with observations from continuous variables that are unevenly sampled in time, for example due…
Financial fraud detection in digital banking requires reasoning over multiple heterogeneous event streams.
problem Financial fraud detection in digital banking requires reasoning over multiple heterogeneous event streams.
method Multi-Stream Fraud Transformer (MSFT) architecture that encodes each event stream with independent Transformer encoders and fuses their representations through configurable mechanisms.
result Sequence models significantly outperform gradient-boosted trees operating on aggregated features.
New framework prevents forgetting in sequential learning tasks.
problem Catastrophic forgetting in Convolutional Neural Networks.
method Conditional channel gating modules and task classifier.
result Consistent improvement over existing methods in continual learning.
An accurate road surface friction prediction algorithm can enable intelligent transportation systems to share timely road surface condition to the public for increasing the safety of the road users. Previously, scholars developed multiple prediction models for forecasting road surface conditions using historical data. …
Time-aware fact-checking improves veracity predictions for time-sensitive claims.
problem Fact-checking decisions should consider temporal information of claims and evidence.
method Investigated four temporal ranking methods to optimize evidence ranking for fact-checking models.
result Time-aware evidence ranking surpasses relevance assumptions and improves veracity predictions for time-sensitive claims.
Time-aware deep learning methods improve spatial downscaling of atmospheric pollutants.
problem Transform coarse satellite data of atmospheric pollutants into high-resolution fields.
method Super-resolution deep residual networks and UNet architectures are extended with a temporal module encoding observation time.
result Temporal modules significantly improve downscaling performance and convergence speed.
SYNC learns time-aware causal representations to improve model generalization in evolving domains.
problem Spurious correlations and shortcut learning in existing EDG methods hinder model generalization.
method SYNC integrates dynamic causal factors and causal mechanism drifts into a sequential VAE framework.
result SYNC achieves superior temporal generalization performance on synthetic and real-world datasets.
A new method improves recommendation accuracy by learning from multiple networks and time-dependent user preferences.
problem Incomplete user profiles and dynamic user preferences degrade recommender quality.
method A cross-network time-aware recommender that learns from multiple source networks and develops current user models.
result The proposed solution achieves superior performance in accuracy, novelty, and diversity.
A new method for embedding temporal relationships in graphs.
problem Limited performance of existing time-aware graph embedding methods.
method Integrates temporal smoothness and task-oriented negative sampling.
result Improves performance in various tasks, especially entity/relationship/temporal scoping prediction.
Improving the performance of click-through rate (CTR) prediction remains one of the core tasks in online advertising systems. With the rise of deep learning, CTR prediction models with deep networks remarkably enhance model capacities. In deep CTR models, exploiting users' historical data is essential for learning user…
Enhances Ponzi scheme detection on Ethereum using time-aware metapaths.
problem Lack of temporal information in heterogeneous transaction graphs.
method Time-aware Metapath Feature Augmentation (TMFAug) module.
result Significant performance improvements in Ponzi scheme detection.
Prospective display advertising poses a great challenge for large advertising platforms as the strongest predictive signals of users are not eligible to be used in the conversion prediction systems. To that end efforts are made to collect as much information as possible about each user from various data sources and to …
TATD predicts missing entries in time-evolving tensors by exploiting temporal dependency and sparsity.
problem Predict missing entries in time-evolving tensors with temporal dependency and sparsity issues.
method TATD (Time-Aware Tensor Decomposition) integrates temporal dependency and time-varying sparsity through a smoothing regularization with Gaussian kernel and alternating optimization.
result TATD achieves state-of-the-art accuracy for decomposing temporal tensors.
FaStR improves scalability for time-aware RS with varying coefficients.
problem Limited applicability of structured regression models to large-scale data with categorical effects and many interactions.
method Combines structured additive regression and factorization approaches in a neural network-based model implementation.
result FaStR scales better and performs competitively with other time-aware RS in prediction performance.
Proposes a deep learning model for timely and accurate recommendations.
problem Inability to provide timely recommendations and ranking issues with implicit feedback.
method Unified cross-network solution using listwise ranking for implicit data.
result Superior performance in accuracy, novelty, and diversity compared to baselines.
FraudTransformer detects payment fraud by preserving event order and time gaps.
problem Detecting payment fraud in real-world banking streams with irregular time gaps.
method Augments a GPT-style architecture with a dedicated time encoder and a learned positional encoder.
result FraudTransformer outperforms classical and transformer baselines, achieving highest AUROC and PRAUC on held-out test set.
Word evolution refers to the changing meanings and associations of words throughout time, as a byproduct of human language evolution. By studying word evolution, we can infer social trends and language constructs over different periods of human history. However, traditional techniques such as word representation learni…
To date, most state-of-the-art sequence modeling architectures use attention to build generative models for language based tasks. Some of these models use all the available sequence tokens to generate an attention distribution which results in time complexity of O(n2). Alternatively, they utilize depthwise convoluti…
TASC improves synthetic control for time-series data with trends.
problem Inability of existing SC methods to fully utilize temporal structure in time-series data.
method TASC uses a state-space model with a constant trend and Kalman filter for counterfactual inference.
result TASC offers advantages in settings with strong temporal trends and high observation noise.
Proposes tPARAFAC2 for tracking evolving patterns in time-evolving data.
problem Lack of temporal regularization in tensor factorizations for capturing evolving patterns.
method Temporal PARAFAC2 (tPARAFAC2) with temporal regularization.
result tPARAFAC2 accurately captures evolving patterns better than existing methods.
DatedGPT prevents lookahead bias in financial forecasting models.
problem Lookahead bias in large language models trained on internet-scale data.
method Time-aware pretraining with annual data cutoffs and instruction fine-tuning.
result Models' knowledge is effectively bounded by their data cutoff year, improving forecasting validity.
The paper studies 4-qubit Clifford states and their properties.
problem Understanding the set and properties of 4-qubit Clifford states.
method Analyzing the 293760 4-qubit Clifford states, splitting them into 18 groups, and studying the action of CNOT gates and local gates.
result There are 293760 4-qubit Clifford states with specific entanglement entropies, and any pair can be connected with local gates and at most 3 CNOT gates.
Improved logistic MoE with sigmoid gate shows better sample efficiency.
problem Improving sample efficiency in logistic MoE models.
method Comprehensive analysis of multinomial logistic MoE with modified sigmoid gate, incorporating temperature parameter and using Euclidean score.
result The sigmoid gate leads to lower sample complexity than softmax gate for both parameter and expert estimation.
New method prepares 3-qubit states using local gates and controlled-Z gates.
problem Preparation of 3-qubit states using quantum gates.
method Uses Ry(θ) gates and controlled-Z gates, with an optimal number of controlled-Z gates. result Optimal number of controlled-Z gates for preparing 3-qubit states is four. TALBO optimizes latent spaces for evolving design objectives.
problem Temporal drift in design objectives.
method GP-prior variational autoencoder for time-varying latent space.
result Consistently outperforms LSBO baselines across varying drift speeds and objectives.
Sigmoid gating is more sample efficient than softmax in mixture of experts.
problem Softmax gating leads to unnecessary competition among experts, causing representation collapse.
method Theoretical analysis of a regression framework with mixture of experts, identifying identifiability conditions and convergence rates.
result Sigmoid gating requires fewer samples to achieve the same expert estimation error as softmax gating.
In quantum computation, series of quantum gates have to be arranged in a predefined sequence that led to a quantum circuit in order to solve a particular problem. What if the sequence of quantum gates is known but both the problem to be solved and the outcome of the so defined quantum circuit remain in the shadow? This…
Long Short-Term Memory (LSTM) infers the long term dependency through a cell state maintained by the input and the forget gate structures, which models a gate output as a value in [0,1] through a sigmoid function. However, due to the graduality of the sigmoid function, the sigmoid gate is not flexible in representing m…
Gated attention improves model curvature, enhancing performance on nonlinear tasks.
problem Understanding the geometric implications of gating in attention mechanisms.
method Modeling attention outputs as Gaussian distributions and analyzing Fisher--Rao geometry.
result Gated attention enables non-flat geometries, including positively curved manifolds.
Gated attention improves performance by using a hierarchical mixture of experts.
problem Improving performance of self-attention mechanisms in Transformers.
method Rigorously show that gated attention can be modeled as a hierarchical mixture of experts, providing a theoretical justification for its benefits.
result Gated attention is more sample-efficient than multi-head self-attention, requiring fewer data points to achieve the same estimation error.
Given the success of the gated recurrent unit, a natural question is whether all the gates of the long short-term memory (LSTM) network are necessary. Previous research has shown that the forget gate is one of the most important gates in the LSTM. Here we show that a forget-gate-only version of the LSTM with chrono-ini…
Improved HMoE models using Laplace gating function enhance expert specialization and performance.
problem Improving performance of hierarchical mixture of experts models.
method Used Laplace gating function instead of Softmax in hierarchical mixture of experts models.
result Laplace gating function accelerates expert convergence and enhances specialization.
The paper evaluates three variants of the Gated Recurrent Unit (GRU) in recurrent neural networks (RNN) by reducing parameters in the update and reset gates. We evaluate the three variant GRU models on MNIST and IMDB datasets and show that these GRU-RNN variant models perform as well as the original GRU RNN model while…
This study analyzes communication constraints in MoE architectures using information theory.
problem Communication constraints in Mixture-of-Experts (MoE) architectures.
method Developed a rate-distortion characterization of finite-rate gating in MoE architectures using information theory.
result Yielded capacity-aware limits for communication-constrained MoE systems.
Bayesian methods have been successfully applied to sparsify weights of neural networks and to remove structure units from the networks, e. g. neurons. We apply and further develop this approach for gated recurrent architectures. Specifically, in addition to sparsification of individual weights and neurons, we propose t…
Recurrent neural networks (RNNs) are powerful dynamical models for data with complex temporal structure. However, training RNNs has traditionally proved challenging due to exploding or vanishing of gradients. RNN models such as LSTMs and GRUs (and their variants) significantly mitigate these issues associated with trai…
Paper connects MoE and self-attention, proposing active-attention.
problem Improving efficiency and performance of self-attention mechanisms.
method Established connection between MoE and self-attention, analyzed quadratic gating functions, proposed active-attention mechanism.
result Active-attention outperforms standard self-attention in various tasks.
Gated recurrent neural networks have achieved remarkable results in the analysis of sequential data. Inside these networks, gates are used to control the flow of information, allowing to model even very long-term dependencies in the data. In this paper, we investigate whether the original gate equation (a linear projec…
Solves challenges in estimating parameters of softmax gating Gaussian mixture models.
problem Identifiability issues and complex interactions in Gaussian mixture of experts.
method Proposes novel Voronoi loss functions and establishes convergence rates of MLE.
result Connects convergence rate of MLE to a solvability problem of polynomial equations.
A major contributing factor to the recent advances in deep neural networks is structural units that let sensory information and gradients to propagate easily. Gating is one such structure that acts as a flow control. Gates are employed in many recent state-of-the-art recurrent models such as LSTM and GRU, and feedforwa…
Recently, a lot of techniques were developed to sparsify the weights of neural networks and to remove networks' structure units, e.g. neurons. We adjust the existing sparsification approaches to the gated recurrent architectures. Specifically, in addition to the sparsification of weights and neurons, we propose sparsif…
Gradient Gating improves deep GNNs by modulating message passing updates.
problem Oversmoothing and performance degradation in deep GNNs.
method Gradient gating mechanism for multi-rate message passing.
result G2 framework alleviates oversmoothing and achieves state-of-the-art performance. LSTMs were introduced to combat vanishing gradients in simple RNNs by augmenting them with gated additive recurrent connections. We present an alternative view to explain the success of LSTMs: the gates themselves are versatile recurrent models that provide more representational power than previously appreciated. We do…
The paper establishes convergence rates for MoE models in classification problems.
problem Understanding the behavior of MoE models in classification settings.
method Established convergence rates for density and parameter estimation in softmax gating multinomial logistic MoE models.
result Parameter estimation rates are significantly improved with a novel modified softmax gating function.
Spatiotemporal sequence prediction is an important problem in deep learning. We study next-frame(s) video prediction using a deep-learning-based predictive coding framework that uses convolutional, long short-term memory (convLSTM) modules. We introduce a novel reduced-gate convolutional LSTM(rgcLSTM) architecture that…
New insights into the top-K sparse softmax gating function for deep learning.
problem Understanding the theoretical effects of the top-K sparse softmax gating function on density and parameter estimations.
method Using a Gaussian mixture of experts, novel loss functions, and theoretical analysis.
result The convergence rates of density and parameter estimations are parametric under certain conditions, but slow under over-specified models.
Model accurately gates ocean microbes from high-frequency flow cytometry data.
problem Gating of high-frequency flow cytometry data for ocean microbes is challenging.
method Trend filtered mixture of experts with smooth parameter variation.
result Model accurately matches human-annotated gating and corrects errors.