A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
Proposes a model combining difference-attention and error-correction LSTMs for improved time series prediction.
The Transformer is a fully attention-based alternative to recurrent networks that has achieved state-of-the-art results across a range of NLP tasks. In this paper, we analyze the structure of attention in a Transformer language model, the GPT-2 small pretrained model. We visualize attention for individual instances and…
New approach improves multi-head attention by making heads less similar.
Mathematical study shows post-hoc explanations are better than attention weights alone.
Attention-based models have recently shown great performance on a range of tasks, such as speech recognition, machine translation, and image captioning due to their ability to summarize relevant information that expands through the entire length of an input sequence. In this paper, we analyze the usage of attention mec…
A new method for feature fusion in U-Net decoders using difference-based gating.
Bitcoin's attention is linked to Google Trends data, not general uncertainty.
Attention mechanisms have become ubiquitous in NLP. Recent architectures, notably the Transformer, learn powerful context-aware word representations through layered, multi-headed attention. The multiple heads learn diverse types of word relationships. However, with standard softmax attention, all attention heads are de…
Paper proposes a new hierarchical attention mechanism for multi-scale data.
RoGAT enhances GAT robustness against adversarial attacks.
Study integrates attentional and spacing factors to improve category learning models.
Multi-head attention mechanism is capable of learning various representations from sequential data while paying attention to different subsequences, e.g., word-pieces or syllables in a spoken word. From the subsequences, it retrieves richer information than a single-head attention which only summarizes the whole sequen…
Graph convolutional networks (GCNs) have shown the powerful ability in text structure representation and effectively facilitate the task of text classification. However, challenges still exist in adapting GCN on learning discriminative features from texts due to the main issue of graph variants incurred by the textual …
Network embedding represents nodes in a continuous vector space and preserves structure information from the Network. Existing methods usually adopt a "one-size-fits-all" approach when concerning multi-scale structure information, such as first- and second-order proximity of nodes, ignoring the fact that different scal…
Graph Attention Networks (GATs) are the state-of-the-art neural architecture for representation learning with graphs. GATs learn attention functions that assign weights to nodes so that different nodes have different influences in the feature aggregation steps. In practice, however, induced attention functions are pron…
DAGR improves navigation by refining goal representations conditioned on the current state.
BERT-based architectures currently give state-of-the-art performance on many NLP tasks, but little is known about the exact mechanisms that contribute to its success. In the current work, we focus on the interpretation of self-attention, which is one of the fundamental underlying components of BERT. Using a subset of G…
Self-attentional models are a new paradigm for sequence modelling tasks which differ from common sequence modelling methods, such as recurrence-based and convolution-based sequence learning, in the way that their architecture is only based on the attention mechanism. Self-attentional models have been used in the creati…
Self-attention model improves HAR from wearable sensors.
Aligns attention distributions for improved accuracy and robustness.
Inspired by recent work in attention models for image captioning and question answering, we present a soft attention model for the reinforcement learning domain. This model uses a soft, top-down attention mechanism to create a bottleneck in the agent, forcing it to focus on task-relevant information by sequentially que…
First derived from human intuition, later adapted to machine translation for automatic token alignment, attention mechanism, a simple method that can be used for encoding sequence data based on the importance score each element is assigned, has been widely applied to and attained significant improvement in various task…
Study compares deep learning models for traffic forecasting, highlighting graph elements' impact.
New insights into how encoder-decoder networks generate attention matrices.
DAUM networks improve weather forecasting with attention and multistream features.
Elliptical Attention improves transformer performance by focusing on contextually relevant features.
Graph convolutional network (GCN) is generalization of convolutional neural network (CNN) to work with arbitrarily structured graphs. A binary adjacency matrix is commonly used in training a GCN. Recently, the attention mechanism allows the network to learn a dynamic and adaptive aggregation of the neighborhood. We pro…
LARF improves random forests with attention mechanisms and contamination models.
GACAN combines multi-granularity time series for traffic forecasting.
We present an attention mechanism inspired from definition of screened Coulomb potential. This attention mechanism was used to interpret the Graph Attention (GAT) model layers and training dataset by using a flexible and scalable framework (CoulGAT) developed for this purpose. Using CoulGAT, a forest of plain and resne…
The study reveals how attention paths in Transformers influence learning outcomes.
Paper introduces a conformer-based system for streaming language identification in long-form speech.
Time-based model improves recommendation performance.
Intelligent Momentum Transformer outperforms traditional trading strategies.
Unified framework for critical scaling of inverse temperature in self-attention.
Although deep neural networks generally have fixed network structures, the concept of dynamic mechanism has drawn more and more attention in recent years. Attention mechanisms compute input-dependent dynamic attention weights for aggregating a sequence of hidden states. Dynamic network configuration in convolutional ne…
A new model captures complex event data using attention and Fourier kernels.
LATTE tackles heterogeneous network embedding challenges with layer-stacked attention.
SpGAT learns graph representations using spectral attention for efficiency.
New algorithms improve multi-task learning across different environments.
Proposes GPCA module for channel attention in CNNs using Gaussian processes.
A new dynamic attention model improves vehicle routing problem solutions.
Self-attention networks localize when eigenspectrum variance is small.
Neural nets predict user attention from mouse movements.
This paper introduces new attention-based convolutional neural networks for selecting bands from hyperspectral images. The proposed approach re-uses convolutional activations at different depths, identifying the most informative regions of the spectrum with the help of gating mechanisms. Our attention techniques are mo…
This research integrates attention into XAI frameworks for better model explanations.
Visual Question Answering (VQA) requires AI models to comprehend data in two domains, vision and text. Current state-of-the-art models use learned attention mechanisms to extract relevant information from the input domains to answer a certain question. Thus, robust attention mechanisms are essential for powerful VQA mo…
The paper approaches the task of handwritten text recognition (HTR) with attentional encoder-decoder networks trained on sequences of characters, rather than words. We experiment on lines of text from popular handwriting datasets and compare different activation functions for the attention mechanism used for aligning i…