BAM model learns graph structure from data with robustness across linear and non-linear dependencies.
problem Detecting dependencies in datasets for graph structure learning.
method Proposes BAM, a neural network model using structural equation models and Chebyshev polynomials for training, with bilinear attention mechanism.
result Demonstrates robust generalizability and superior performance in graph estimation.
AIM models explain deep learning in attention layers, offering solvable insights.
problem Understanding how deep learning models learn in attention layers.
method Statistical mechanics and random matrix theory.
result Closed-form predictions for Bayes-optimal generalization error and gradient descent performance.
Financial time-series forecasting has long been a challenging problem because of the inherently noisy and stochastic nature of the market. In the High-Frequency Trading (HFT), forecasting for trading purposes is even a more challenging task since an automated inference system is required to be both accurate and fast. I…
Automatic emotion recognition (AER) is a challenging task due to the abstract concept and multiple expressions of emotion. Although there is no consensus on a definition, human emotional states usually can be apperceived by auditory and visual systems. Inspired by this cognitive process in human beings, it's natural to…
HKT improves sequence processing with multi-scale attention and kernel analysis.
problem Processing sequences at multiple scales with efficient attention mechanisms.
method Trainable causal downsampling and convex weights for level-specific score matrices.
result HKT achieves consistent gains over standard attention across various tasks.
Study introduces statistical mechanics for min-max problems.
problem Understanding the properties of min-max problems in high dimensions.
method Statistical mechanical formalism for analyzing min-max problems.
result Derives the relationship between training data and generalization error.
BiN normalizes financial time-series for better forecasting.
problem Non-stationarity and multimodality in financial time-series data.
method Bilinear Normalization (BiN) incorporated into TABL networks.
result BiN-TABL outperforms other normalization methods in financial forecasting.
Reasoning about graphs evolving over time is a challenging concept in many domains, such as bioinformatics, physics, and social networks. We consider a common case in which edges can be short term interactions (e.g., messaging) or long term structural connections (e.g., friendship). In practice, long term edges are oft…
Min-max formulations have attracted great attention in the ML community due to the rise of deep generative models and adversarial methods, while understanding the dynamics of gradient algorithms for solving such formulations has remained a grand challenge. As a first step, we restrict to bilinear zero-sum games and giv…
Active-memory mechanisms can replace self-attention in Transformers, but optimal results often require both.
problem Replacing self-attention with active-memory mechanisms in Transformers.
method Evaluation of various active-memory mechanisms in a Transformer model.
result Active-memory mechanisms can achieve comparable results to self-attention for language modeling, but optimal results are often achieved by combining both mechanisms.
Investigates the fundamental components of attention mechanisms.
problem Understanding the building blocks of attention in deep learning.
method Classified and studied three key mechanisms: additive, multiplicative output, and synaptic attention.
result Additive activation attention is central in proofs of lower bounds.
We consider differentiable games where the goal is to find a Nash equilibrium. The machine learning community has recently started using variants of the gradient method (GD). Prime examples are extragradient (EG), the optimistic gradient method (OG) and consensus optimization (CO), which enjoy linear convergence in cas…
Study reveals attention mechanism's similarity computation parallels traditional machine learning.
problem Understanding the essence and principles of attention mechanism in deep learning.
method Examined classic metrics and vector space properties in manifold learning, clustering, and supervised learning to identify key characteristics of similarity computation and information propagation.
result Self-attention mechanism in deep learning adheres to the same principles but operates more flexibly and adaptively.
Modern neural networks are often augmented with an attention mechanism, which tells the network where to focus within the input. We propose in this paper a new framework for sparse and structured attention, building upon a smoothed max operator. We show that the gradient of this operator defines a mapping from real val…
GOAT improves attention mechanisms by learning better priors.
problem Standard attention mechanisms use a naive uniform prior, limiting flexibility and generalization.
method GOAT introduces a trainable, continuous prior that replaces the uniform assumption, maintaining compatibility with optimized kernels.
result GOAT avoids representational trade-offs and learns an extrapolatable prior that combines positional flexibility with length generalization.
CoulGAT interprets GAT models by analyzing node interactions.
problem Understanding and interpreting the complex interactions within graph attention networks.
method Developed a CoulGAT framework to analyze and interpret GAT model layers and datasets.
result Extracted node-node and node-feature interactions to define a standard model for graph structure.
Attention mechanisms in deep neural networks have achieved excellent performance on sequence-prediction tasks. Here, we show that these recently-proposed attention-based mechanisms---in particular, the Transformer with its parallelizable self-attention layers, and the Memory Fusion Network with attention across modalit…
This article is an attempt to generalize Riemann's bilinear relations on compact Riemann surface of genus at least 2, which may lead to new structures in the theory of hyperbolic Riemann surfaces. No significant result is obtained, the article serves to bring the readers' attention to the observation made by [Bol-1949]…
The softmax content-based attention mechanism has proven to be very beneficial in many applications of recurrent neural networks. Nevertheless it suffers from two major computational limitations. First, its computations for an attention lookup scale linearly in the size of the attended sequence. Second, it does not enc…
SympFormer accelerates attention blocks using inertial dynamics on density spaces.
problem Improving the efficiency of self-attention blocks in Transformers.
method Introduced accelerated attention blocks derived from inertial Nesterov dynamics on density spaces.
result Accelerated attention blocks converge faster than classical blocks while preserving oracle calls.
Single-head attention approximates any function under various norms.
problem Universal approximation of functions using attention mechanisms.
method Interpreting attention as partitioning and summing linear transformations.
result Single-head attention can approximate any continuous function under L∞-norm and Lebesgue integrable functions under Lp-norm. Paper proposes a new hierarchical attention mechanism for multi-scale data.
problem Challenges in applying neural attention to multi-scale, multi-modal data.
method Developed a mathematical framework for multi-modal, multi-scale data and derived optimal neural attention mechanics.
result Proposed hierarchical attention mechanism improves transformer performance in multi-scale, multi-modal settings.
Scales attention for long contexts in LLMs.
problem Development of attention mechanisms for long context inference.
method Scale-invariant total attention and sparsity conditions, with a position-dependent transformation of logits.
result Scale-invariant attention scheme improves validation loss and long-context retrieval.
GA-Net selectively attends to parts of a sequence for text classification.
problem Inefficient global attention mechanisms on long sequences.
method Gated Attention Network (GA-Net) using an auxiliary network to dynamically select and attend to important parts of the sequence.
result GA-Net achieves better performance with less computation and interpretability.
New insights into attention mechanisms reveal dramatic trade-offs between rank and heads.
problem Dramatic trade-offs between rank and number of heads in attention mechanisms.
method Presented a simple target function and proved theoretical limits.
result Full-rank attention is necessary for long contexts, while low-rank is sufficient for short ones.
Enhances speech emotion recognition by integrating visual data with attention mechanisms.
problem Improving emotion detection accuracy by combining multiple modalities.
method Introducing an attention mechanism to combine audio, text, and video modalities.
result Significant improvement of 3.65% in weighted accuracy.
As an important part of speech recognition technology, automatic speech keyword recognition has been intensively studied in recent years. Such technology becomes especially pivotal under situations with limited infrastructures and computational resources, such as voice command recognition in vehicles and robot interact…
The paper investigates how Global Self-attention improves GCNs.
problem Improving expressive power and addressing overfitting in GCNs.
method Applying Global Self-attention mechanism over graph features.
result GSA mechanism enhances expressive power and mitigates overfitting and over-smoothing in GCNs.
Graph Neural Networks (GNNs) are powerful to learn the representation of graph-structured data. Most of the GNNs use the message-passing scheme, where the embedding of a node is iteratively updated by aggregating the information of its neighbors. To achieve a better expressive capability of node influences, attention m…
Improved convergence rates for saddle-point optimization algorithms.
problem Understanding last-iterate convergence rates for saddle-point optimization algorithms in constrained settings.
method Expanding the understanding of last-iterate convergence for Optimistic Gradient Descent Ascent (OGDA) and Optimistic Multiplicative Weights Update (OMWU) in the constrained setting.
result Linear last-iterate convergence achieved with a universal constant learning rate for OMWU in bilinear games over the simplex.
Graph attention is not always beneficial; conditions for perfect node classification are identified.
problem Understanding when graph attention mechanisms improve node classification performance.
method Theoretical analysis using Contextual Stochastic Block Models (CSBMs).
result Graph attention mechanisms are more effective when structure noise exceeds feature noise, and simpler graph convolution operations are better when feature noise predominates.
Visual Question Answering (VQA) requires AI models to comprehend data in two domains, vision and text. Current state-of-the-art models use learned attention mechanisms to extract relevant information from the input domains to answer a certain question. Thus, robust attention mechanisms are essential for powerful VQA mo…
Attention mechanism is effective in both focusing the deep learning models on relevant features and interpreting them. However, attentions may be unreliable since the networks that generate them are often trained in a weakly-supervised manner. To overcome this limitation, we introduce the notion of input-dependent unce…
Extracts important peaks from XRD spectra using Attention mechanism.
problem Identifying significant peaks in XRD patterns for material properties.
method Convolutional neural network with Attention mechanism to analyze deep features.
result Selected lattice constant predicts cathodic material cell voltage.
Investigates the benefits of multi-head attention in Transformers, deriving convergence and generalization guarantees.
problem Underexplored dynamics of multi-head attention in Transformer training and generalization.
method Derives convergence and generalization guarantees for gradient-descent training of a multi-head self-attention model.
result Establishes conditions for initialization that ensure multi-head attention's realizability.
Paper extends sparse alternatives to softmax for continuous domains, enabling efficient attention mechanisms.
problem Efficiently assigning zero probability to irrelevant categories in continuous domains.
method Extend alpha-entmax to continuous domains, introducing continuous-domain attention mechanisms.
result Continuous attention allows attending to time intervals and compact regions, improving text classification, machine translation, and visual question answering.
ABIForest improves anomaly detection using attention weights.
problem Anomaly detection in datasets.
method Attention mechanism integrated into Isolation Forest.
result ABIForest outperforms standard Isolation Forest on synthetic and real datasets.
Improved GAN performance with a novel local attention mechanism.
problem Enhancing the performance of Generative Adversarial Networks (GANs).
method Introducing a two-dimensional local attention mechanism that preserves geometry and locality, and using information flow graphs for design.
result Significant improvements in FID and Inception scores, from 18.65 to 15.94 on ImageNet.
The study reveals how attention paths in Transformers influence learning outcomes.
problem Understanding the theoretical basis of Transformers' performance.
method Developed a statistical mechanics theory for a simplified attention network.
result The predictor statistics are influenced by the combination of attention paths.
Paper uses AI to predict option volatility surfaces with improved accuracy.
problem Difficult to predict dynamic evolution of option volatility smile surface.
method Combines deep learning (LSTM) with attention mechanism.
result Predicted volatility surfaces lead to higher returns and Sharpe ratios.
MAT uses attention mechanism for molecule property prediction.
problem Designing a competitive neural network for molecule property prediction.
method Augmented attention mechanism using inter-atomic distances and molecular graph structure.
result MAT achieves state-of-the-art performance on diverse molecular prediction tasks.
WAVE improves time series forecasting by integrating AR and MA components.
problem Time series forecasting challenges.
method WAVE attention mechanism with AR and MA components.
result WAVE attention consistently improves TSF performance.
OLS predictions are shown to be similar to attention mechanisms in models.
problem OLS in traditional statistics and econometrics.
method Rewriting OLS as an attention mechanism in a transformed space.
result OLS can be understood as minimizing squared prediction errors via optimal embedding and decoding.
Linformer reduces transformer complexity to linear, improving efficiency.
problem High cost of training and deploying large transformer models for long sequences.
method Approximates self-attention with low-rank matrix, proposing Linformer with O(n) complexity. result Linformer performs similarly to standard transformers but is more memory- and time-efficient.
Neural networks using transformer-based architectures have recently demonstrated great power and flexibility in modeling sequences of many types. One of the core components of transformer networks is the attention layer, which allows contextual information to be exchanged among sequence elements. While many of the prev…
Paper introduces kernel deformed exponential families for sparse continuous attention.
problem Creating efficient attention mechanisms for sparse data.
method Developed kernel deformed exponential families, theoretically and experimentally.
result Kernel deformed exponential families can attend to multiple compact regions of data.
The paper explores how sinks and diagonal patterns prevent attention oversmoothing.
problem Preventing attention oversmoothing in neural networks.
method Analyzing geometric conditions and conditions for dense vs. sparse attention, proving equivalence between sinks and hard attention switch, and comparing the costs of sinks vs. diagonal patterns.
result Sinks and diagonal patterns effectively prevent attention oversmoothing, and diagonal patterns provide a more flexible approach.
Paper proposes an attention sampler for reducing attention mechanism computation.
problem Computational challenges in large-scale attention-based models.
method Importance sampling in streaming setting, attention sampler.
result Significantly reduces the computational burden of attention mechanisms.