New insights into how encoder-decoder networks generate attention matrices.
problem Understanding how encoder-decoder networks use attention matrices.
method Decomposing hidden states into temporal and input-driven components.
result Attention matrices are formed based on task requirements, not architecture type.
Unified framework analyzes and compares RFF and RoPE PEs for music generation.
problem Efficiently modeling music generation with positional encodings.
method Kernel methods to analyze and compare RFF and RoPE PEs.
result RoPEPool outperforms other methods in melody harmonization.
This paper studies the prediction of chord progressions for jazz music by relying on machine learning models. The motivation of our study comes from the recent success of neural networks for performing automatic music composition. Although high accuracies are obtained in single-step prediction scenarios, most models fa…
To act and plan in complex environments, we posit that agents should have a mental simulator of the world with three characteristics: (a) it should build an abstract state representing the condition of the world; (b) it should form a belief which represents uncertainty on the world; (c) it should go beyond simple step-…
Two autoencoding models learn latent traffic scene representations.
problem Learning latent representations of traffic scenarios.
method CNN and RNN models for spatio-temporal and temporal data, incorporating permutation invariance.
result Latent scenario embeddings can be used for clustering and similarity retrieval.
DRFormer uses dynamic tokenization and multi-scale transformer to forecast long time series.
problem Forecasting long-term time series data across diverse scales.
method Dynamic tokenizer, multi-scale transformer, dynamic sparse learning, rotary position encoding.
result DRFormer outperforms existing methods in forecasting accuracy.
Game-theoretic model captures investor interactions for stock price forecasting.
problem Complex market dynamics driving stock price movements.
method Game-theoretic modeling of heterogeneous investor interactions in a dynamic graph structure.
result Our method outperforms state-of-the-art stock price forecasting methods.
TIMeSynC combines financial service interactions for intent prediction.
problem Aligning and learning from multi-domain, multi-resolution sequences for accurate intent prediction.
method An encoder-decoder transformer model addressing sequence alignment, temporal dynamics, and dynamic/static sequence combination.
result Significant improvement in intent prediction over existing methods.
Model predicts travel time under rare conditions using a vector-space model.
problem Predicting travel time under rare temporal conditions (e.g., holidays, school vacations) is challenging due to limited historical data and other temporal changes.
method Presented a vector-space model for encoding rare temporal conditions, allowing coherent representation learning across different conditions.
result Increased performance for travel time prediction over different baselines when using the vector-space encoding for representing the temporal setting.
Multi-step passenger demand forecasting is a crucial task in on-demand vehicle sharing services. However, predicting passenger demand over multiple time horizons is generally challenging due to the nonlinear and dynamic spatial-temporal dependencies. In this work, we propose to model multi-step citywide passenger deman…
Financial fraud detection in digital banking requires reasoning over multiple heterogeneous event streams.
problem Financial fraud detection in digital banking requires reasoning over multiple heterogeneous event streams.
method Multi-Stream Fraud Transformer (MSFT) architecture that encodes each event stream with independent Transformer encoders and fuses their representations through configurable mechanisms.
result Sequence models significantly outperform gradient-boosted trees operating on aggregated features.
BiPE blends intra-segment and inter-segment encodings for better length extrapolation.
problem Improving length extrapolation in language models.
method Bilevel Positional Encoding (BiPE) that separates intra-segment and inter-segment encodings.
result BiPE enhances length extrapolation across various text modalities.
Model place cells as spatial embeddings for efficient path planning and cognitive map construction.
problem Encoding spatial navigation in the hippocampus.
method Model place cells using spectral decomposition of multi-step random walk transition kernels, inducing sparsity and adjacency.
result Place cells encode spatial information through non-negativity and inner-product structure, forming a cognitive map.
GTEA learns node representations in temporal interaction graphs.
problem Inductive representation learning on temporal interaction graphs.
method Integrates sequence model with time encoder and self-attention scheme for edge and node embeddings.
result GTEA learns comprehensive node representations capturing temporal and structural characteristics.
TG-GAN models dynamic graph evolution for continuous-time temporal graphs.
problem Challenges in modeling dynamic temporal graphs, especially in continuous time.
method Temporal Graph Generative Adversarial Network (TG-GAN) that models truncated edge sequences, time budgets, and node attributes.
result TG-GAN significantly outperforms existing methods in efficiency and effectiveness.
STRING improves 2D and 3D position encodings for better performance.
problem Efficient and accurate position encoding for 2D and 3D applications.
method STRING extends Rotary Position Encodings with a unifying theoretical framework, maintaining translation invariance and low computational cost.
result STRING shows substantial gains in open-vocabulary object detection and robotics.
Randomized positional encodings boost transformer performance on longer sequences.
problem Transformers struggle with generalizing to sequences of arbitrary length.
method Introduced randomized positional encodings that simulate longer sequences and randomly select positions.
result Randomized positional encodings increase test accuracy by 12.0% on average for sequences of unseen length.
Time-aware deep learning methods improve spatial downscaling of atmospheric pollutants.
problem Transform coarse satellite data of atmospheric pollutants into high-resolution fields.
method Super-resolution deep residual networks and UNet architectures are extended with a temporal module encoding observation time.
result Temporal modules significantly improve downscaling performance and convergence speed.
Monoaural audio source separation is a challenging research area in machine learning. In this area, a mixture containing multiple audio sources is given, and a model is expected to disentangle the mixture into isolated atomic sources. In this paper, we first introduce a challenging new dataset for monoaural source sepa…
New method for Transformer models to encode position information without sequential bias.
problem Lack of flexible and learnable position encoding for Transformer models.
method Continuous dynamical model to learn position encoding.
result Consistent improvements over baselines in various NLP tasks.
A new method detects financial fraud using graph transformers.
problem Detecting fraudulent transactions in financial data.
method Spatial-Temporal-Aware Graph Transformer (STA-GT) integrating GNNs and transformers.
result STA-GT outperforms general GNN models on financial fraud detection.
Paper introduces a new method for Transformers with linear complexity.
problem No efficient relative positional encoding for linear Transformer models.
method Stochastic Positional Encoding (SPE) that replaces classical RPE.
result SPE behaves like RPE and performs well on benchmarks.
Proposes a nonparametric approach for inferring spike train filters.
problem Modeling neuron information encoding from electrophysiological recordings.
method Gaussian process framework for joint inference of filters and hyperparameters.
result Automatic learning of filter temporal span and stimulus/history filters.
This paper tackles spatio-temporal information preservation in machine learning.
problem Conventional machine learning assumes orthogonal data attributes, disrupting spatio-temporal information.
method Shift-invariant k-means, convolutional dictionary learning, and spatio-temporal hypercomplex encoding schemes are proposed.
result Gabor feature extraction outperforms convolutional dictionary learning in spatio-temporal information preservation.
GAttNHP predicts future events in temporal knowledge graphs by encoding long-range dependencies and handling mutual excitation.
problem Forecasting future events in temporal knowledge graphs due to long-range dependencies, mutual excitation, and heavy-tailed inter-arrival times.
method GAttNHP uses a self-attention encoder, semantic soft-grouping, and NCQ regression to address these issues.
result GAttNHP improves entity and time prediction on six benchmark TKG datasets compared to state-of-the-art baselines.
We present the Video Ladder Network (VLN) for efficiently generating future video frames. VLN is a neural encoder-decoder model augmented at all layers by both recurrent and feedforward lateral connections. At each layer, these connections form a lateral recurrent residual block, where the feedforward connection repres…
TK-GCN forecasts spatiotemporal dynamics using Koopman-enhanced graph convolutional networks.
problem Forecasting complex spatiotemporal dynamics over irregular domains.
method Two-stage framework: Koopman-enhanced Graph Convolutional Network (K-GCN) for spatial encoding and Transformer for temporal modeling.
result TK-GCN outperforms state-of-the-art methods in spatiotemporal cardiac dynamics forecasting.
Deep learning models have shown state-of-the-art performance in many inverse reconstruction problems. However, it is not well understood what properties of the latent representation may improve the generalization ability of the network. Furthermore, limited models have been presented for inverse reconstructions over ti…
Kriformer uses graph transformers to estimate data in sparse sensor areas.
problem Sparse sensor deployment and unreliable data in spatiotemporal kriging tasks.
method Graph transformer model with positional encoding and attention mechanisms.
result Kriformer excels in representing unobserved locations in spatiotemporal kriging tasks.
New unsupervised learning task improves RL performance.
problem Reward-driven feature learning limitations in RL from images.
method Introduce Augmented Temporal Contrast (ATC) for unsupervised learning of image representations.
result Training encoders using ATC matches or outperforms end-to-end RL in most environments.
Nowadays, multivariate time series data are increasingly collected in various real world systems, e.g., power plants, wearable devices, etc. Anomaly detection and diagnosis in multivariate time series refer to identifying abnormal status in certain time steps and pinpointing the root causes. Building such a system, how…
Efficiently accelerates attention calculation for Transformers with relative positional encoding.
problem Quadratic complexity of attention in long sequences.
method Kernelized attention with Fast Fourier Transform (FFT) for RPE.
result Achieves O(n log n) time complexity, mitigates training instability, and outperforms other models.
Novel neural network solves PDEs with multi-scale resolution.
problem Solving time-dependent PDEs with varying spatial and temporal scales.
method Multi-scale message passing neural network with temporal and spatial gating modules.
result Outperforms baselines on PDEs with diverse scales.
Majority of semantic segmentation algorithms operate on a single frame even in the case of videos. In this work, the goal is to exploit temporal information within the algorithm model for leveraging motion cues and temporal consistency. We propose two simple high-level architectures based on Recurrent FCN (RFCN) and Mu…
Transformer struggles with arithmetic length but improves with explicit structure encoding.
problem Transformers fail to generalize length in arithmetic tasks.
method Explicitly encoding structural symmetries via modified number formatting and custom positional encodings.
result Transformer can generalize up to 50-digit numbers without additional data.
Bi-linear feature learning models, like the gated autoencoder, were proposed as a way to model relationships between frames in a video. By minimizing reconstruction error of one frame, given the previous frame, these models learn "mapping units" that encode the transformations inherent in a sequence, and thereby learn …
Proposes a THGNN for dynamic financial time series prediction.
problem Challenges in predicting stock market price movements.
method Temporal and heterogeneous graph neural network (THGNN) approach.
result Significantly improved prediction performance compared to state-of-the-art methods.
Spatio-temporal graphs such as traffic networks or gene regulatory systems present challenges for the existing deep learning methods due to the complexity of structural changes over time. To address these issues, we introduce Spatio-Temporal Deep Graph Infomax (STDGI)---a fully unsupervised node representation learning…
Model predicts future term connections in biomedical research.
problem Capturing temporal dynamics and unobserved connections in biomedical term relationships.
method Variational inference model for positive-unlabeled learning on dynamic graphs.
result Model effectively predicts term relationships in real-world datasets.
A novel approach predicts long-term stock price trends using 2D-convolutional encoders and semantic segmentation.
problem Predicting long-term daily stock price changes with deep learning models.
method Proposes a hierarchical CNN structure with Atrous Spatial Pyramid Pooling blocks to capture both long and short-term temporal relationships.
result Achieved overall accuracy and AUC of 78.18% and 0.88 for predicting trends over the next 20 days.
Interpersonal relations are fickle, with close friendships often dissolving into enmity. In this work, we explore linguistic cues that presage such transitions by studying dyadic interactions in an online strategy game where players form alliances and break those alliances through betrayal. We characterize friendships …
In recent years, Generative Adversarial Networks (GAN) have emerged as a powerful method for learning the mapping from noisy latent spaces to realistic data samples in high-dimensional space. So far, the development and application of GANs have been predominantly focused on spatial data such as images. In this project,…
TMTF improves time series visualization by separating dynamic regimes.
problem Misleading global transition matrix in time series analysis.
method Temporal chunking, local transition matrices, and image assembly.
result Temporal segmentation reveals distinct transition dynamics.
RTFN extracts robust temporal features for time series analysis.
problem Challenges in extracting sufficient shapelets from time series data.
method Combines temporal feature networks and attentional LSTM networks.
result RTFN outperforms in supervised and unsupervised time series analysis.
Bartnik mass is positive and non-decreasing for black holes
problem Quasilocal mass for black holes
method Defining a Bartnik mass and proving positivity and monotonicity
result Positive and non-decreasing Bartnik mass for black holes
We propose a conceptual design for a quantum blockchain. Our method involves encoding the blockchain into a temporal GHZ (Greenberger-Horne-Zeilinger) state of photons that do not simultaneously coexist. It is shown that the entanglement in time, as opposed to an entanglement in space, provides the crucial quantum adva…
New neural networks model for spatio-temporal data.
problem Building a mapping from spatially encoded time series covariates to real-valued response data.
method Proposed two novel extensions of Functional Neural Network (FNN) for spatio-temporal regression.
result Demonstrated effectiveness in handling varying spatial correlations through comprehensive simulation studies.
This paper explores the capabilities of convolutional neural networks to deal with a task that is easily manageable for humans: perceiving 3D pose of a human body from varying angles. However, in our approach, we are restricted to using a monocular vision system. For this purpose, we apply a convolutional neural networ…