A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
CDAM improves attention maps for ViTs, making them more class-sensitive.
AReLU uses attention-based rectification to improve neural network performance.
Attention is an operation that selects some largest element from some set, where the notion of largest is defined elsewhere. Applying this operation to sequence to sequence mapping results in significant improvements to the task at hand. In this paper we provide the mathematical definition of attention and examine its …
Transformers' self-attention mechanism is mapped to a generalized Potts model.
Softmax is found ineffective for NL block, leading to improved performance.
Attention mechanisms and non-local mean operations in general are key ingredients in many state-of-the-art deep learning techniques. In particular, the Transformer model based on multi-head self-attention has recently achieved great success in natural language processing and computer vision. However, the vanilla algori…
Gated attention improves performance by using a hierarchical mixture of experts.
We derive the stress-energy tensor for polyharmonic maps between Riemannian manifolds. Moreover, we employ the stress-energy tensor to characterize polyharmonic maps where we pay special attention to triharmonic maps.
KATA improves associative recall by optimizing feature maps derived from nonnegative attention weights.
Centroid Transformers reduce memory and computation by summarizing inputs into centroids.
Self-attention model improves HAR from wearable sensors.
LUNA improves linear attention for long sequences without sacrificing accuracy.
Visual attention serves as a means of feature selection mechanism in the perceptual system. Motivated by Broadbent's leaky filter model of selective attention, we evaluate how such mechanism could be implemented and affect the learning process of deep reinforcement learning. We visualize and analyze the feature maps of…
Transformers can predict new tokens based on any number of context tokens, approximating continuous mappings with fixed resources.
We conduct large-scale studies on `human attention' in Visual Question Answering (VQA) to understand where humans choose to look to answer questions about images. We design and test multiple game-inspired novel attention-annotation interfaces that require the subject to sharpen regions of a blurred image to answer a qu…
This article reports on an investigation of the use of convolutional neural networks to predict the visual attention of chess players. The visual attention model described in this article has been created to generate saliency maps that capture hierarchical and spatial features of chessboard, in order to predict the pro…
Although group convolutional networks are able to learn powerful representations based on symmetry patterns, they lack explicit means to learn meaningful relationships among them (e.g., relative positions and poses). In this paper, we present attentive group equivariant convolutions, a generalization of the group convo…
A study on optimizing self-attention in tabular data using Optimal Transport.
Regularizes attention scores in vision transformers using bootstrapping.
Sequential modelling with self-attention has achieved cutting edge performances in natural language processing. With advantages in model flexibility, computation complexity and interpretability, self-attention is gradually becoming a key component in event sequence models. However, like most other sequence models, self…
We introduce a new local sparse attention layer that preserves two-dimensional geometry and locality. We show that by just replacing the dense attention layer of SAGAN with our construction, we obtain very significant FID, Inception score and pure visual improvements. FID score is improved from to on Im…
We incorporate Tensor-Product Representations within the Transformer in order to better support the explicit representation of relation structure. Our Tensor-Product Transformer (TP-Transformer) sets a new state of the art on the recently-introduced Mathematics Dataset containing 56 categories of free-form math word-pr…
Study proposes a statistical test for Vision Transformer's attention mechanisms.
Modern neural networks are often augmented with an attention mechanism, which tells the network where to focus within the input. We propose in this paper a new framework for sparse and structured attention, building upon a smoothed max operator. We show that the gradient of this operator defines a mapping from real val…
Paper investigates Lipschitz constants of self-attention modules in neural networks.
Introduces a novel spatial attention module for convolutional networks.
Transfer learning through fine-tuning a pre-trained neural network with an extremely large dataset, such as ImageNet, can significantly accelerate training while the accuracy is frequently bottlenecked by the limited dataset size of the new target task. To solve the problem, some regularization methods, constraining th…
This study investigates abrupt learning dynamics in Transformers, revealing plateau formation and internal representation collapse.
We consider symplectic Floer homology in the lowest nontrivial dimension, that is to say, for area-preserving diffeomorphisms of surfaces. Particular attention is paid to the quantum cap product; we show that it distinguishes the trivial element of the mapping class group from any nontrivial one.
Image super-resolution is a challenging task and has attracted increasing attention in research and industrial communities. In this paper, we propose a novel end-to-end Attention-based DenseNet with Residual Deconvolution named as ADRD. In our ADRD, a weighted dense block, in which the current layer receives weighted f…
A new method for feature fusion in U-Net decoders using difference-based gating.
3D Axial-Attention improves lung nodule classification accuracy.
Generalized are the investigated in other works of the author transports along paths in fibre bundles to transports along arbitrary maps in them. Their structure and some properties are studied. Special attention is paid to the linear case and the case when the map's domain is a Cartesian product of two sets. Also cons…
Bayesian Attention Networks compress data by focusing on key training samples.
OSA overcomes instability in skipless Transformers.
OLS predictions are shown to be similar to attention mechanisms in models.
Those maps of a closed surface to the three-dimensional torus that are homotopic to embeddings are characterized. Particular attention is paid to the somewhat intricate case when the surface is nonorientable.
Fetal brain imaging is a cornerstone of prenatal screening and early diagnosis of congenital anomalies. Knowledge of fetal gestational age is the key to the accurate assessment of brain development. This study develops an attention-based deep learning model to predict gestational age of the fetal brain. The proposed mo…
Incorporating human domain knowledge for breast tumor diagnosis is challenging, since shape, boundary, curvature, intensity, or other common medical priors vary significantly across patients and cannot be employed. This work proposes a new approach for integrating visual saliency into a deep learning model for breast t…
Inspired by the observation that humans are able to process videos efficiently by only paying attention where and when it is needed, we propose an interpretable and easy plug-in spatial-temporal attention mechanism for video action recognition. For spatial attention, we learn a saliency mask to allow the model to focus…
Action detection and recognition tasks have been the target of much focus in the computer vision community due to their many applications, namely, security, robotics and recommendation systems. Recently, datasets like AVA, provide multi-person, multi-label, spatiotemporal action detection and recognition challenges. Be…
KFAtt improves CTR prediction by modeling user behavior with Kalman filtering attention.
Transformers become faster by linearizing self-attention.
The study reveals the spectral structure of attention layers and its implications for generalization.
Though neural networks have achieved much progress in various applications, it is still highly challenging for them to learn from a continuous stream of tasks without forgetting. Continual learning, a new learning paradigm, aims to solve this issue. In this work, we propose a new model for continual learning, called Ba…
Attention mechanism learns to focus on sparse tokens efficiently.
In recent years a lot of attention has been paid to topological spaces which are a bit more general than smooth manifolds - orbifolds. Orbifolds are intuitively speaking manifolds with some singularities. The formal definition is also modelled on that of manifolds, an orbifold is a topological space which locally is ho…
Transformers can approximate posterior predictive distributions through in-context learning.