This paper uses SPk languages to explore LDD characteristics and multi-element dependencies.
problem Understanding and modeling Long Distance Dependencies (LDDs) with multi-element interactions.
method Generated datasets with various properties using Strictly k-Piecewise languages, analyzed using mutual information.
result The number of interacting elements in a dependency is a key characteristic of LDDs.
We consider the task of detecting regulatory elements in the human genome directly from raw DNA. Past work has focused on small snippets of DNA, making it difficult to model long-distance dependencies that arise from DNA's 3-dimensional conformation. In order to study long-distance dependencies, we develop and release …
DRew dynamically rewires message passing to improve long-range tasks.
problem Over-squashing in MPNNs due to local message passing.
method Layer-dependent rewiring and delay mechanism for gradual densification.
result DRew outperforms graph Transformers and multi-hop MPNNs on long-range tasks.
In order to build efficient deep recurrent neural architectures, it is essential to analyze the complexityof long distance dependencies (LDDs) of the dataset being modeled. In this paper, we presentdetailed analysis of the dependency decay curve exhibited by various datasets. The datasets sampledfrom a similar process …
Physics: Similar long-distance properties can mask vastly different short-distance metrics.
problem Classifying homogeneous metrics on group manifolds by long-distance properties.
method Apply universality concept to geometry, focusing on metrics on Lie groups.
result Many metrics on low-dimensional Lie groups have similar long-distance properties despite differing short-distance properties.
HighwayGraph models long-distance node relations in GNNs with improved performance.
problem Limited-layer information propagation in GNNs hinders long-distance node relation modeling.
method Proposes two solutions: implicit and explicit modeling of long-distance node relations using shallow GNN architectures and a self-training framework.
result HighwayGraph achieves consistent and significant improvements over four GNNs on three benchmark datasets.
Combines PCA and message passing for better graph node embeddings.
problem Challenges of learning long-range dependencies in graph data.
method PCAPass: PCA + message passing for unsupervised node embeddings.
result Empirically shows competitive performance on node classification benchmarks.
The study examines Hawkes processes and their long-term behavior.
problem Understanding the long-term behavior of Hawkes processes.
method Proving functional limit theorems under various conditions on the dispersion of child events.
result Functional limit theorems hold for Hawkes processes with different levels of child event dispersion.
Estimates intrinsic dimension of data for GANs.
problem Estimating intrinsic dimension of high-dimensional data.
method Uses Wasserstein distances for estimation.
result Provides sample complexity bounds for GANs.
Estimates LRD in sequential data, improving RNNs.
problem Quantifying LRD in sequential data for better RNNs.
method Principled estimation procedure based on LRD theory for real-valued time series.
result Estimates LRD reliably in user behavior and Wikipedia article writing.
Deep RNNs excel at capturing long-term dependencies in sequential data.
problem Lack of a formal measure for RNNs' long-term memory capacity.
method Introduced a measure called Start-End separation rank to quantify RNNs' ability to model long-term dependencies.
result Deep RNNs support Start-End separation ranks that are combinatorially higher than shallow ones.
Language Models (LMs) are important components in several Natural Language Processing systems. Recurrent Neural Network LMs composed of LSTM units, especially those augmented with an external memory, have achieved state-of-the-art results. However, these models still struggle to process long sequences which are more li…
Paper proposes Gini distance statistics for estimating feature-label dependence.
problem Identifying statistical dependence between features and categorical labels.
method Generalized Gini distance in RKHS for feature-label dependence estimation.
result Gini distance statistics converge faster and have tighter error bounds than distance covariance.
Idioms pose problems to almost all Machine Translation systems. This type of language is very frequent in day-to-day language use and cannot be simply ignored. The recent interest in memory augmented models in the field of Language Modelling has aided the systems to achieve good results by bridging long-distance depend…
Uniform-in-time analysis for Stein Variational Gradient Descent across various metrics.
problem Understanding long-term behavior of finite-particle systems in relation to their mean-field limits.
method Developed uniform-in-time propagation-of-chaos results for continuous-time SVGD using cutoff strategies and finite-dimensional theories.
result Uniform-in-time propagation-of-chaos bounds in various metrics, including Langevin kernel Stein discrepancy, Wasserstein-1, and Wasserstein-2 distances.
We exhibit an efficient procedure for testing, based on a single long state sequence, whether an unknown Markov chain is identical to or ε-far from a given reference chain. We obtain nearly matching (up to logarithmic factors) upper and lower sample complexity bounds for our notion of distance, which is bas…
Machine learning can help us in solving problems in the context big data analysis and classification, as well as in playing complex games such as Go. But can it also be used to find novel protocols and algorithms for applications such as large-scale quantum communication? Here we show that machine learning can be used …
Improves retrieval accuracy for hierarchical documents, especially for distant matches.
problem Limited expressive power of dual encoder models in hierarchical retrieval.
method Proves feasibility of DEs for HR, introduces pretrain-finetune recipe to improve long-distance retrieval.
result Pretrain-finetune boosts recall on long-distance pairs from 19% to 76%.
Finite Goeritz groups for links with long bridge decompositions.
problem Characterizing Goeritz groups of links.
method Proving finite Goeritz groups for links with specific bridge decompositions.
result Finite Goeritz groups for links with distance at least 6.
We consider a point cloud Xn:={x1,…,xn} uniformly distributed on the flat torus Td:=Rd/Zd, and construct a geometric graph on the cloud by connecting points that are within distance ε of each other. We let P(Xn) be the space of probability …
Novel methods robustify Gromov-Wasserstein distance for cross-domain alignment.
problem Robustifying Gromov-Wasserstein distance for cross-domain alignment.
method Three novel techniques derived from robust statistics to improve GW and its variants.
result Empirical validation shows superior resilience to contamination.
BlockEcho method improves imputation of block-wise missing data.
problem Block-wise missing data reduces interpolation capability and predictive power.
method Integrates Matrix Factorization (MF) within Generative Adversarial Networks (GAN) to retain long-distance inter-element relationships.
result Superior performance on public datasets across three domains, especially at higher missing rates.
The following working document summarizes our work on the clustering of financial time series. It was written for a workshop on information geometry and its application for image and signal processing. This workshop brought several experts in pure and applied mathematics together with applied researchers from medical i…
We present a case-study demonstrating the usefulness of Bayesian hierarchical mixture modelling for investigating cognitive processes. In sentence comprehension, it is widely assumed that the distance between linguistic co-dependents affects the latency of dependency resolution: the longer the distance, the longer the …
We develop the distance dependent Chinese restaurant process (CRP), a flexible class of distributions over partitions that allows for non-exchangeability. This class can be used to model many kinds of dependencies between data in infinite clustering models, including dependencies across time or space. We examine the pr…
PBO methods improve RNN performance in learning long-term dependencies.
problem Training RNNs to learn long-term dependencies is challenging.
method Population-based global optimisation (PBO) techniques, including evolution strategies and particle swarm optimisation.
result PBO methods lead to performance improvements in RNNs for volatility forecasting.
Novel graph model forecasts urban traffic with reduced spatial complexity.
problem Challenges in traffic forecasting due to spatio-temporal complexity, especially in urban environments.
method MW-TGC network model that combines spatial and temporal dependencies using multi-weighted adjacency matrices and graph convolution operations.
result MW-TGC network outperforms other models in urban-core and urban-mix sites, reducing variance in heterogeneous environments.
Paper provides GOT convergence guarantees for sub-gamma distributions and dependent samples.
problem Estimating GOT distance under general settings.
method Gaussian-smoothed optimal transport (GOT) framework, sub-gamma distributions, dependent samples, kernel MMD distances.
result Convergence guarantees for GOT distance under more general settings.
We propose three measures of mutual dependence between multiple random vectors. All the measures are zero if and only if the random vectors are mutually independent. The first measure generalizes distance covariance from pairwise dependence to mutual dependence, while the other two measures are sums of squared distance…
Introduces Spectral Attention for better long-range time series forecasting.
problem Challenges in capturing long-range dependencies in time series forecasting.
method Spectral Attention mechanism that preserves temporal correlations and long-range dependencies.
result Achieves state-of-the-art results on 11 real-world time series datasets.
Researchers modify dp distance to handle long, thin splines.
problem Maintaining stability in convergence metrics with scalar curvature approaching positivity.
method Introducing and analyzing a modified dp distance to handle persistent splines. result The modified dp distance provides a stable estimate, useful for geometric stability. New empirical process bounds reveal trade-off between dependence and complexity in nonparametric learning.
problem Understanding generalization in nonparametric learning with temporal dependencies.
method Developed bounds on expected supremum of empirical processes under β/ρ-mixing assumptions. result Achieved rates similar to i.i.d. setting under long-range dependence with complex function classes.
A new RNN model tackles long-time dependencies with fast, invertible, and memory-efficient hidden states.
problem Challenges in processing sequential inputs with long-time dependencies in RNNs.
method A novel RNN architecture based on a Hamiltonian system of oscillators.
result The proposed RNN mitigates exploding and vanishing gradient problems, providing state-of-the-art performance.
Representation and learning of long-range dependencies is a central challenge confronted in modern applications of machine learning to sequence data. Yet despite the prominence of this issue, the basic problem of measuring long-range dependence, either in a given data source or as represented in a trained deep model, r…
Distance plays a fundamental role in measuring similarity between objects. Various visualization techniques and learning tasks in statistics and machine learning such as shape matching, classification, dimension reduction and clustering often rely on some distance or similarity measure. It is of tremendous importance t…
Fractal analysis is carried out on the stock market indices of seven European countries and the US. We find evidence of long range dependence in the log return series of the Mibtel (Italy) and the PX Glob (Czech Republic). Long range dependence implies that predictable patterns in the log returns do not dissipate quick…
A new regularizer boosts long-range dependency in sequence data.
problem Improving long-range dependency in sequence data models.
method Developed a mutual information regularizer to enhance sequence learning.
result The approach increases mutual information and likelihood on holdout data.
Combines CNN and Transformer for financial time series forecasting.
problem Forecasting financial time series, especially stock prices, is challenging due to short-term and long-term dependencies.
method Uses CNN for short-term dependencies and Transformer for long-term dependencies.
result Demonstrated superior performance in forecasting stock price changes compared to traditional methods.
Improved GNN handles long-range dependencies in multi-relational graphs.
problem Vanishing gradients in GNNs for multi-relational graphs.
method Proposes a Gated Graph Neural Network with improved long-range dependency handling.
result Outperforms popular GNN models in synthetic tasks.
Sequence models assign probabilities to variable-length sequences such as natural language texts. The ability of sequence models to capture temporal dependence can be characterized by the temporal scaling of correlation and mutual information. In this paper, we study the mutual information of recurrent neural networks …
FDBM models use fractional Brownian motion to model complex stochastic processes.
problem Capturing memory effects and long-range dependencies in stochastic processes.
method Developed a generative diffusion bridge framework using a Markovian approximation of fractional Brownian motion.
result FDBM outperforms standard models in predicting future states and unpaired data translation.
Study finds long-range dependence in financial markets, but deep generative models struggle to replicate it.
problem Long-range dependence in financial markets and challenges of deep generative models.
method Empirical analysis of financial data from three sectors, including LRD through various statistical methods and deep learning models.
result Deep generative models can reproduce stylized features but fail to capture long-range dependence structures.
We prove that every Teichmuller geodesic of a finite type surface contains a string of intersecting long, thick and dominant segments, such that the distance between consecutive segments is bounded. This is key to obtaining some results about Teichmuller geodesics which mimic those for hyperbolic geodesics. These resul…
Geography effect is investigated for the Chinese stock market including the Shanghai and Shenzhen stock markets, based on the daily data of individual stocks. The Shanghai city and the Guangdong province can be identified in the stock geographical sector. By investigating a geographical correlation on a geographical pa…
This paper presents empirical evidence using recently developed techniques in econophysics suggesting that the degree of long-range dependence in interest rates depends on the conduct of monetary policy. We study the term structure of interest rates for the US and find evidence that global Hurst exponents change dramat…
New RNN model handles long-term dependencies in irregularly-sampled time series.
problem Handling long-term dependencies in irregularly-sampled time series data.
method Designing ODE-LSTMs that separate memory from continuous-time state.
result ODE-LSTMs outperform other RNN-based models on non-uniformly sampled data with long-term dependencies.
Paper questions RNN and LSTM's long-term memory and introduces a new definition.
problem Whether RNN and LSTM have long-term memory.
method Introduced a new definition of long-term memory and modified RNN and LSTM to test it.
result RNN and LSTM do not meet the new definition of long-term memory.
Novel graph neural network combines random walks with local message passing.
problem Graph neural networks struggle with long-range dependencies.
method Combines random walks with local message passing in a novel architecture.
result Significant performance improvements on graph benchmarks.