Study examines asset pricing using various attention models, finding global self-attention and sliding window sparse attention models perform well.
problem Traditional asset pricing models miss temporal dependency and short memory issues.
method Investigates RNN attention models with various attention mechanisms for large-cap US stocks.
result Global self-attention and sliding window sparse attention models outperform in deriving returns and hedging risks, especially during the pandemic.
Investor attention predicts global equity market volatility during Ukraine invasion.
problem Predicting global equity market volatility during geopolitical events.
method Event-specific attention indices based on Google Trends, analyzed across 51 global equity markets.
result Investor attention significantly predicts volatility in countries with higher economic openness to Russia and closer to it.
GSANet improves semantic segmentation accuracy with selective and global attention.
problem Semantic segmentation accuracy improvement.
method Global and selective attention mechanism with ASPP and sparsemax.
result GSANet achieves state-of-the-art accuracy on ADE20k and Cityscapes datasets.
The paper investigates how Global Self-attention improves GCNs.
problem Improving expressive power and addressing overfitting in GCNs.
method Applying Global Self-attention mechanism over graph features.
result GSA mechanism enhances expressive power and mitigates overfitting and over-smoothing in GCNs.
Gradient descent converges geometrically to optimal self-attention parameters.
problem Training softmax self-attention layers for linear regression.
method Structure-aware gradient descent with preconditioner and regularizer.
result Gradient descent converges geometrically to global minima.
Global Memory Augmentation (GMAT) improves Transformer performance on long documents.
problem Large memory requirements of Transformer pairwise dot-product attention for long sequences.
method Integrates a dense global memory of length M into sparse Transformer blocks.
result Significant improvement on various tasks, including synthetic tasks, masked language modeling, and reading comprehension.
ETC improves Transformer models for long and structured inputs.
problem Scaling input length and encoding structured inputs in Transformers.
method Introduces global-local attention, relative position encodings, and CPC pre-training.
result Achieves state-of-the-art results on four natural language datasets.
Proposes a novel model for healthcare and SME credit risk prediction.
problem Lack of guidance from global view in sequence representation learning for time series modeling.
method Hierarchical Global View-guided (HGV) sequence representation learning framework with GGE and β β β -Attn modules. result Competitive prediction performance compared with other known baselines.
Paper proposes a forecasting model combining autoregressive models with spectral attention.
problem Time series forecasting across various domains.
method Combines deep autoregressive models with Spectral Attention (SA) module.
result SAAM consistently demonstrates improved forecasting accuracy compared to state-of-the-art approaches.
This paper improves GNNs' generalization by adding a Low-Rank Global Attention module.
problem Improving the generalization power of Graph Neural Networks (GNNs).
method Incorporating a Low-Rank Global Attention (LRGA) module into GNNs.
result Augmenting GNNs with LRGA aligns them with a powerful graph isomorphism test, 2-Folklore Weisfeiler-Lehman (2-FWL).
Tiled Squeeze-and-Excite improves channel attention with local spatial context.
problem Improving channel attention mechanisms in neural networks.
method Proposes tiled squeeze-and-excite (TSE) framework for channel attention.
result Local context of 7 rows or columns is sufficient for matching global context performance.
A new method for feature fusion in U-Net decoders using difference-based gating.
problem Precise fusion of high-level semantics and low-level details in U-Net decoder reconstruction.
method Proposes two difference-based gating approaches: Feature-difference gating (FDG) and Entropy-difference gating (EDG).
result Both FDG and EDG methods outperform existing attention-based fusion methods, with EDG showing superior performance.
Study local and global aspects of complex plane curve embeddings.
problem Local and global problems of complex plane curve embeddings.
method Braid monodromy, local and global analysis.
result Historical progress in understanding complex plane curve embeddings.
GAT-AGNN learns stock trends using graph and attention mechanisms.
problem Predicting dynamic stock trends in a complex market.
method Sequential graph structure with attention mechanisms.
result GAT-AGNN outperforms state-of-the-art methods in stock trend prediction.
EAGLE-Net enhances foundation models by integrating patch-level features for better tissue understanding.
problem Foundation models lack mechanisms for global tissue structure and local context in computational pathology.
method EAGLE-Net combines multi-scale spatial encoding, attention-guided loss functions, and background suppression to aggregate patch-level features into slide-level predictions.
result EAGLE-Net improves classification accuracy and concordance indices across multiple cancer types, producing biologically coherent attention maps.
Deep learning predicts drug prescriptions across global health records.
problem Predicting drug prescriptions in chronic disease patients.
method Adaptive cross-global attention graph kernel network with support vector machine.
result Model outperforms current methods in accuracy and interpretability.
New approach improves linear-time attention for language models.
problem Challenges of quadratic attention in long-sequence modelling, especially for discrete data.
method Reinterpreting linear attention through latent probabilistic graphical models, introducing asymmetric structure and recurrent parameterisation.
result Our model achieves competitive performance and outperforms existing linear attention variants on language modelling benchmarks.
Attention learns PCA on Gaussian data, proving its connection to principal component analysis.
problem Principal component analysis on Gaussian data.
method Analysis of attention mechanisms through PCA, covering finite and infinite prompt regimes.
result Attention aligns with principal eigenvectors of covariance matrices, converging to optimal solutions in the infinite-prompt limit.
LNUCB-TA improves MAB performance by dynamically adjusting exploration rates and recognizing spatiotemporal patterns.
problem Suboptimal performance in environments with rapidly changing reward structures and static exploration rates.
method Hybrid model combining linear and nonlinear estimation, with adaptive k-NN for temporal attention.
result Significantly outperforms state-of-the-art algorithms in cumulative and mean reward, convergence, and robustness.
Land use classification of low resolution spatial imagery is one of the most extensively researched fields in remote sensing. Despite significant advancements in satellite technology, high resolution imagery lacks global coverage and can be prohibitively expensive to procure for extended time periods. Accurately classi…
SpGAT learns graph representations using spectral attention for efficiency.
problem Efficiently capturing global graph patterns with minimal parameters.
method Introduces Spectral Graph Attention Network (SpGAT) using spectral domain attention mechanisms and a fast Chebychev approximation.
result SpGAT achieves better global pattern recognition with fewer parameters compared to GAT.
Conformer encoder reverses sequence in time dimension, affecting decoder training.
problem Reversal of sequence in Conformer encoder impacts decoder training.
method Analyzed initial behavior of decoder cross-attention and proposed methods to avoid flipping.
result Self-attention module of Conformer starts dominating, allowing only reversed information to pass.
Transformer++ improves neural machine translation BLEU scores.
problem Handling long-range dependencies in sentences.
method Proposes a new multi-head attention mechanism with context learning.
result Achieves new state-of-the-art BLEU scores on English-to-German and English-to-French translation tasks.
A new method for multi-criteria recommender systems using graph attention networks.
problem Lack of nuanced relationships between users and items based on specific criteria.
method MDGAT, a multi-edge bipartite graph with dual attention networks and contrastive learning.
result MDGAT achieves higher accuracy in predicting item ratings compared to baseline methods.
Since the pioneering work of Canham and Helfrich, variational formulations involving curvature-dependent functionals, like the classical Willmore functional, have proven useful for shape analysis of biomembranes. We address minimizers of the Canham-Helfrich functional defined over closed surfaces enclosing a fixed volu…
Deep learning model detects landmarks in X-ray cephalograms.
problem Automatically detecting landmarks in cephalometric X-ray images.
method 2-stage U-Net with attention mechanism and Expansive Exploration strategy.
result State-of-the-art results in landmark detection.
This research integrates attention into XAI frameworks for better model explanations.
problem Improving the interpretability of transformer models.
method Developed two novel explanation methods: Shapley value decomposition and token-level directional derivatives.
result Attention weights can be meaningfully incorporated into XAI frameworks, enhancing transformer explainability.
TabNet learns tabular data with attention, improving performance and interpretability.
problem Efficient learning and interpretability in tabular data.
method Sequential attention to feature selection, self-supervised learning.
result Outperforms other models on tabular datasets, interpretable feature attributions.
GA-Net selectively attends to parts of a sequence for text classification.
problem Inefficient global attention mechanisms on long sequences.
method Gated Attention Network (GA-Net) using an auxiliary network to dynamically select and attend to important parts of the sequence.
result GA-Net achieves better performance with less computation and interpretability.
PAGTN improves molecular property prediction by leveraging longer-range graph dependencies.
problem Local aggregation in GCNs misses higher-order graph properties.
method PAGTN uses path features and global attention layers to capture longer-range dependencies.
result PAGTN outperforms GCNs on various molecular property prediction datasets.
Transformers adapted to spherical geometry using space-filling curves.
problem Generalizing transformers to geometric domains like spheres.
method Attention heads following a space-filling curve.
result Introduction of the Spiroformer on a 2-sphere.
Meta algorithm solves multivariate optimization using univariate optimizers.
problem Multivariate global optimization problems.
method Meta algorithm combining univariate global optimizers.
result Meta algorithm provides robust regret guarantees.
Dropout is used to avoid overfitting by randomly dropping units from the neural networks during training. Inspired by dropout, this paper presents GI-Dropout, a novel dropout method integrating with global information to improve neural networks for text classification. Unlike the traditional dropout method in which the…
T-EMDE bridges the heterogeneity gap between image and text modalities.
problem Finding similarities between image and text modalities with non-related feature spaces.
method Inspired by EMDE, T-EMDE uses sketches for multimodal operations, avoiding self-attention's quadratic complexity.
result T-EMDE achieves state-of-the-art results and reduces model latency.
A3T-GCN improves traffic forecasting by capturing spatial and temporal dependencies.
problem Accurate real-time traffic forecasting in complex road networks.
method Attention Temporal Graph Convolutional Network (A3T-GCN) integrating recurrent units and graph convolutional network.
result Improved prediction accuracy through attention mechanism and global temporal information.
BigBird improves transformer performance on NLP tasks with longer sequences.
problem Quadratic dependency on sequence length in transformer models.
method Sparse attention mechanism reducing quadratic dependency to linear.
result Significant improvement in performance on NLP tasks like QA and summarization.
KFAtt improves CTR prediction by modeling user behavior with Kalman filtering attention.
problem Improving CTR prediction in personalized e-commerce search engines.
method KFAtt combines Kalman filtering with attention mechanisms to model user behavior.
result KFAtt outperforms existing methods in CTR prediction, achieving better performance in both offline and online settings.
Understanding the global optimality in deep learning (DL) has been attracting more and more attention recently. Conventional DL solvers, however, have not been developed intentionally to seek for such global optimality. In this paper we propose a novel approximation algorithm, BPGrad, towards optimizing deep models glo…
Transformers are powerful sequence models, but require time and memory that grows quadratically with the sequence length. In this paper we introduce sparse factorizations of the attention matrix which reduce this to O ( n n ) O(n \sqrt{n}) O ( n n ) . We also introduce a) a variation on architecture and initialization to train deeper net…
We focus our attention on the link prediction problem for knowledge graphs, which is treated herein as a binary classification task on neural embeddings of the entities. By comparing, combining and extending different methodologies for link prediction on graph-based data coming from different domains, we formalize a un…
Graph Neural Networks improve volatility prediction in financial markets.
problem Traditional models struggle with complex, non-linear interdependencies in financial markets.
method Temporal Graph Attention Network (Temporal GAT) combines GCNs and GATs to capture dynamic graph structures.
result Temporal GAT outperforms traditional GARCH models in volatility forecasting, especially for short- to mid-term predictions.
Softmax is found ineffective for NL block, leading to improved performance.
problem Inefficiency of softmax in NL block for global context modeling.
method Empirical analysis and replacement of softmax with scaling factor.
result Improved performance on various datasets with reduced computational cost.
Reinforcement learning in multi-agent scenarios is important for real-world applications but presents challenges beyond those seen in single-agent settings. We present an actor-critic algorithm that trains decentralized policies in multi-agent settings, using centrally computed critics that share an attention mechanism…
Study on multi-head softmax attention dynamics for in-context learning.
problem Understanding and optimizing multi-head softmax attention models for multi-task linear regression.
method Gradient flow analysis and spectral mapping technique.
result Gradient flow converges to optimal multi-head softmax attention model, with task allocation emerging during training.
We consider a Canham-Helfrich-type variational problem defined over closed surfaces enclosing a fixed volume and having fixed surface area. The problem models the shape of multiphase biomembranes. It consists of minimizing the sum of the Canham-Helfrich energy, in which the bending rigidities and spontaneous curvatures…
Orion-Bix combines biaxial attention and meta-learning for tabular few-shot learning.
problem Scaling and generalizing tabular models with mixed numeric and categorical fields, weak feature structure, and limited labeled data.
method Orion-Bix uses biaxial attention and meta-learned in-context reasoning to efficiently capture local and global dependencies.
result Orion-Bix outperforms gradient-boosting baselines and state-of-the-art tabular models on public benchmarks.
Self-attention optimizers converge to optimal weights, revealing bias patterns.
problem Understanding the implicit bias in self-attention mechanisms.
method Analysis of gradient-based optimization in self-attention layers.
result Adaptive step-size strategies can accelerate convergence to optimal weights.
The team predicts foreign exchange rates using clustering and attention models.
problem Complexity and unexpected events in foreign exchange markets.
method Clustering and attention models applied to historical data.
result Improved event-driven price prediction for oversold scenarios.