SpGAT learns graph representations using spectral attention for efficiency.
problem Efficiently capturing global graph patterns with minimal parameters.
method Introduces Spectral Graph Attention Network (SpGAT) using spectral domain attention mechanisms and a fast Chebychev approximation.
result SpGAT achieves better global pattern recognition with fewer parameters compared to GAT.
Paper proposes a forecasting model combining autoregressive models with spectral attention.
problem Time series forecasting across various domains.
method Combines deep autoregressive models with Spectral Attention (SA) module.
result SAAM consistently demonstrates improved forecasting accuracy compared to state-of-the-art approaches.
Introduces Spectral Attention for better long-range time series forecasting.
problem Challenges in capturing long-range dependencies in time series forecasting.
method Spectral Attention mechanism that preserves temporal correlations and long-range dependencies.
result Achieves state-of-the-art results on 11 real-world time series datasets.
The study reveals the spectral structure of attention layers and its implications for generalization.
problem Understanding the spectral structure and generalization of trained attention layers.
method Empirical risk minimization in a single-head tied-attention layer, using random matrix theory, spin-glass theory, and approximate message passing.
result Exact high-dimensional characterization of training and test error, interpolation and recovery thresholds, and spectrum of the key and query matrices.
FastGAT reduces GNN computation time by 10x using graph sparsification.
problem High computational burden in attention-based GNNs.
method Spectral sparsification to generate optimal graph pruning.
result Per-epoch time is almost linear in graph nodes, reducing computational time by up to 10x.
A framework uses free probability to analyze Transformer models.
problem Understanding the dynamics and complexity of Transformer-based language models.
method Formal operator-theoretic analysis using free probability theory.
result Entropy-based generalization bounds derived under freeness assumptions.
This paper explains GNNs using graph signal denoising.
problem Understanding how GNNs work for node representation learning.
method Spectral graph convolutional networks and graph attention networks are analyzed from the perspective of graph signal denoising.
result GNNs implicitly solve graph signal denoising problems.
PRISMA uses PDE residuals for fast, robust, and accurate inference.
problem Slow gradient-based optimization and instability in PDE residual-based methods.
method Integrates PDE residuals directly into the model's architecture via attention mechanisms in the spectral domain.
result Competitive accuracy with significantly lower inference costs and faster speeds.
Language models fail to process hallucinated responses, and this study diagnoses the failure.
problem Language models fail to process hallucinated responses, leading to over-concentration or diffuse attention.
method The study uses forced scoring of benchmark-labeled responses to compute attention shapes and analyze the symmetric component of the degree-normalized attention operator.
result The study proves that every transpose-invariant spectral diagnostic of the attention operator is orientation-blind and bounds the sensitivity of any Lipschitz diagnostic by the asymmetry coefficient \(G\).
New method prevents entropy collapse in Transformer training, leading to more stable and robust models.
problem Training instability in Transformers, especially in attention layers.
method Spectral normalization with a learned scalar to prevent entropy collapse.
result Prevents entropy collapse, leading to more stable training.
Transformer models show distinct spectral fingerprints under voice changes.
problem Detecting architectural biases in transformer models.
method Spectral analysis of attention-induced token graphs.
result Clear architectural signatures in model fingerprints correlate with language-specific behavior.
This study analyzes why attention layers in neural networks can cause signal loss and proposes a solution.
problem Pathological behavior of attention layers in neural networks, leading to signal loss.
method Spectral analysis using Random Matrix Theory to identify and mitigate rank collapse in width.
result A novel solution to mitigate rank collapse in width by removing outlier eigenvalues.
Gradient descent converges geometrically to optimal self-attention parameters.
problem Training softmax self-attention layers for linear regression.
method Structure-aware gradient descent with preconditioner and regularizer.
result Gradient descent converges geometrically to global minima.
The paper compares spectral geometry in hyperbolic and spherical manifolds.
problem Understanding spectral geometry in spherical manifolds.
method Survey of known results and open problems.
result Analogous results hold in hyperbolic manifolds but not necessarily in spherical manifolds.
Attention-based GNNs can't prevent oversmoothing, leading to homogeneous node representations.
problem The issue of oversmoothing in attention-based GNNs.
method Viewed attention-based GNNs as nonlinear time-varying dynamical systems and used tools from the theory of products of inhomogeneous matrices and the joint spectral radius.
result Graph attention mechanism cannot prevent oversmoothing and loses expressive power exponentially.
New approach shows why overparameterized neural nets generalize well.
problem Understanding why overparameterized neural nets generalize well in practice.
method An alternative notion of capacity for attention-based models based on the effective rank of attention matrices.
result Generalization bound matches empirical scaling laws observed in large language models.
GACAN combines multi-granularity time series for traffic forecasting.
problem High dynamics and complex spatial-temporal dependency of road networks in traffic forecasting.
method Graph Attention-Convolution-Attention Networks (GACAN) with Att-Conv-Att (ACA) block.
result GACAN outperforms state-of-the-art baselines in traffic forecasting.
A panoply of multi-view clustering algorithms has been developed to deal with prevalent multi-view data. Among them, spectral clustering-based methods have drawn much attention and demonstrated promising results recently. Despite progress, there are still two fundamental questions that stay unanswered to date. First, h…
We investigate Relational Graph Attention Networks, a class of models that extends non-relational graph attention mechanisms to incorporate relational information, opening up these methods to a wider variety of problems. A thorough evaluation of these models is performed, and comparisons are made against established be…
Neural network model improves leaf spectral reflectance prediction for grapevines.
problem Inaccurate modeling of grapevine leaf spectral reflectance from traits.
method Multi-head attention neural network trained on grapevine-specific data.
result Model achieved high accuracy (R^2=0.84, NRMSE=1.52%) and outperformed PROSPECT-PRO.
LASE learns graph embeddings by unrolling GD iterations into a neural network.
problem Learning graph embeddings from graph inputs.
method Truncating and interpreting GD iterations as layers in a GNN.
result LASE outperforms eigendecomposition routines and GNNs in link prediction and node classification.
Attention learns PCA on Gaussian data, proving its connection to principal component analysis.
problem Principal component analysis on Gaussian data.
method Analysis of attention mechanisms through PCA, covering finite and infinite prompt regimes.
result Attention aligns with principal eigenvectors of covariance matrices, converging to optimal solutions in the infinite-prompt limit.
This work analyzes self-attention matrices using random matrix theory.
problem Understanding the theoretical behavior of self-attention layers in neural networks.
method Asymptotic spectral analysis of the attention matrix, Gaussian equivalence, and linearization.
result The singular value distribution of the attention matrix is asymptotically characterized by a linear model.
A new convolutional spectral kernel network learns hierarchical and local features.
problem Lack of deep learning in non-stationary spectral kernels.
method Introduces convolutional filters and deep architectures into non-stationary spectral kernels, derives generalization error bounds, and introduces regularizers.
result Validated the effectiveness of the convolutional spectral kernel network on real-world datasets.
GSAN learns adaptive node representations using geometric scattering and attention.
problem Oversmoothing in node representation learning.
method Attention-based architecture integrating geometric scattering and GCN channels.
result GSAN outperforms previous networks in semi-supervised node classification.
OLS is a special case of Transformer, revealing its linear nature.
problem Understanding the statistical essence of Transformer architecture.
method Algebraic proof and spectral decomposition of covariance matrix.
result Attention mechanism in Transformers is mathematically equivalent to OLS.
Detects corruption in agentic models during execution.
problem Inconsistent context, retrieval errors, or adversarial inputs corrupt intermediate steps of reasoning chains.
method Analyzes token graphs induced by attention and computes spectral statistics to emit accept/reject signals.
result A single threshold on the high frequency energy ratio optimally detects context inconsistency in agentic models.
New centrality-based graph shift operators improve graph neural networks.
problem Improving graph neural networks by enhancing graph shift operators.
method Proposed Centrality Graph Shift Operators (CGSOs) using global centrality metrics.
result CGSOs lead to improved performance in graph neural networks on real-world datasets.
We first show that a Laplace isospectral family of Riemannian orbifolds, satisfying a lower Ricci curvature bound, contains orbifolds with points of only finitely many isotropy types. If we restrict our attention to orbifolds with only isolated singularities, and assume a lower sectional curvature bound, then the numbe…
Graph signal processing detects hallucinations in large language models.
problem Detecting factual reasoning from hallucinations in large language models.
method Modeling transformer layers as dynamic graphs, using spectral analysis to define diagnostics.
result Spectral signatures can distinguish different types of hallucinations and achieve high accuracy.
WildCat efficiently compresses neural network attention mechanisms.
problem Expensive quadratic runtime of attention mechanisms in neural networks.
method Uses a weighted coreset with a fast subsampling algorithm to approximate attention with near-linear time complexity.
result Approximates exact attention with super-polynomial error decay and near-linear runtime.
Dimensionality reduction (DR) methods have attracted extensive attention to provide discriminative information and reduce the computational burden of the hyperspectral image (HSI) classification. However, the DR methods face many challenges due to limited training samples with high dimensional spectra. To address this …
We discuss generalizations of Ozsvath-Szabo's spectral sequence relating Khovanov homology and Heegaard Floer homology, focusing attention on an explicit relationship between natural Z (resp., 1/2 Z) gradings appearing in the two theories. These two gradings have simple representation-theoretic (resp., geometric) inter…
Improved performance of factorized neural layers through spectral initialization and Frobenius decay.
problem Improving the performance of factorized neural layers in various deep learning contexts.
method Spectral initialization and Frobenius decay for initialization and regularization.
result Spectral initialization and Frobenius decay lead to improved performance across multiple deep learning settings.
Muon dynamics study uses spectral Wasserstein flow for optimization stability.
problem Optimizing deep learning models with gradient normalization.
method Introduces Spectral Wasserstein distances for matrix flows, proving equivalence with Benamou--Brenier formulation.
result Gradient-flow interpretation of mean-field normalized training dynamics.
Proposes neural dynamic mode decomposition for end-to-end modeling of nonlinear dynamics.
problem Understanding and modeling nonlinear dynamical systems.
method Trains neural networks to minimize forecast error based on spectral decomposition in the lifted space.
result Demonstrates effectiveness in eigenvalue estimation and forecast performance.
Skyformer uses Gaussian kernel and Nyström method to speed up self-attention in transformers.
problem High computational cost of self-attention in transformers.
method Replaces softmax with Gaussian kernel and applies Nyström method for matrix approximation.
result Skyformer achieves comparable or better performance with fewer computation resources.
Hypergraph partitioning lies at the heart of a number of problems in machine learning and network sciences. Many algorithms for hypergraph partitioning have been proposed that extend standard approaches for graph partitioning to the case of hypergraphs. However, theoretical aspects of such methods have seldom received …
The study analyzes spectral algorithms for kernel methods and derives generalization error.
problem Estimating generalization error of spectral algorithms for kernel methods.
method Considered spectral algorithms including KRR and GD, derived generalization error as a functional of learning profile.
result Showed the loss localizes on certain spectral scales and conjectured universality of the loss for noisy observations.
In this paper, we propose the Self-Attention Generative Adversarial Network (SAGAN) which allows attention-driven, long-range dependency modeling for image generation tasks. Traditional convolutional GANs generate high-resolution details as a function of only spatially local points in lower-resolution feature maps. In …
Spectral decomposition of the Koopman operator is attracting attention as a tool for the analysis of nonlinear dynamical systems. Dynamic mode decomposition is a popular numerical algorithm for Koopman spectral analysis; however, we often need to prepare nonlinear observables manually according to the underlying dynami…
Spectral Clustering(SC) is a prominent data clustering technique of recent times which has attracted much attention from researchers. It is a highly data-driven method and makes no strict assumptions on the structure of the data to be clustered. One of the central pieces of spectral clustering is the construction of an…
Study on multi-head softmax attention dynamics for in-context learning.
problem Understanding and optimizing multi-head softmax attention models for multi-task linear regression.
method Gradient flow analysis and spectral mapping technique.
result Gradient flow converges to optimal multi-head softmax attention model, with task allocation emerging during training.
New method efficiently learns positive-definite curvature for neural nets.
problem Efficiently learn positive-definite curvature for neural net training.
method Spectral-factorized positive-definite curvature learning approach.
result Efficiently applies arbitrary matrix roots and generic curvature learning.
We present graph attention networks (GATs), novel neural network architectures that operate on graph-structured data, leveraging masked self-attentional layers to address the shortcomings of prior methods based on graph convolutions or their approximations. By stacking layers in which nodes are able to attend over thei…
Linearized attention fails to converge to NTK limit even at large widths.
problem Understanding the convergence of attention mechanisms to the kernel regime.
method Analyzes linearized attention and its relationship to the NTK limit, considering practical widths and conditions.
result Linearized attention does not converge to its NTK limit at any practical width, revealing a fundamental trade-off.
Unified framework for generating meteorological time series from text.
problem Lack of large-scale, physically grounded multimodal datasets and architectures ignoring spectral-temporal structure.
method Introduce MeteoCap-3B dataset and MTransformer model.
result State-of-the-art generation quality, accurate cross-modal alignment, strong semantic controllability.
Study reveals how attention helps in signal recovery from sequence models using random matrix theory.
problem Signal recovery from sequence models with attention mechanisms.
method Analysis of sample covariance matrices constructed from pooled sequence representations with attention weights.
result Optimal attention weights maximize signal-to-noise ratio and improve signal recovery.