RoGAT enhances GAT robustness against adversarial attacks.
problem Vulnerability of GAT to adversarial attacks.
method Dynamic adjustment of edge weights and features, with an extra attention score.
result RoGAT outperforms other defensive methods in robustness tests.
Aligns attention distributions for improved accuracy and robustness.
problem Improving the accuracy and robustness of neural networks using attention mechanisms.
method Alignment attention that encourages key and query distributions to match within each head.
result Alignment attention leads to better accuracy, uncertainty estimation, and robustness across various tasks.
Kernel PCA explains self-attention mechanisms in deep learning models.
problem Understanding and explaining self-attention mechanisms in deep learning models.
method Deriving self-attention from kernel principal component analysis (kernel PCA).
result RPC-Attention, a robust attention mechanism, outperforms softmax attention in various tasks.
Attention temperature improves robustness of ICL in high-dimensional settings.
problem ICL robustness failure under distribution shift in high dimensions.
method Analyzed a Transformer with approximate softmax attention, derived a closed-form error expression, and showed optimal temperature minimizes error.
result Optimal attention temperature minimizes ICL generalization error under distribution shift.
SE(3)-Transformers maintain equivariance for 3D data under rotations and translations.
problem Ensuring stable and predictable performance in 3D data under transformations.
method Introducing a self-attention module that is equivariant under continuous 3D roto-translations.
result The SE(3)-Transformer outperforms non-equivariant and non-attention models on real-world datasets.
Bayesian attention improves model performance and robustness.
problem Limited exploration of stochastic attention in neural networks.
method Introduces Bayesian attention belief networks using gamma and Weibull distributions.
result Outperforms deterministic and stochastic attention methods in accuracy and robustness.
TabPFN model shows strong robustness to noisy data.
problem TabPFN tackles robustness to noisy and imperfect tabular data.
method Empirical robustness analysis of TabPFN's attention mechanisms under various perturbations.
result TabPFN maintains high predictive performance and coherent internal behavior under noisy and imperfect data.
Neural network models have been very successful at achieving high accuracy on natural language inference (NLI) tasks. However, as demonstrated in recent literature, when tested on some simple adversarial examples, most of the models suffer a significant drop in performance. This raises the concern about the robustness …
Adversarial training has been successfully applied to build robust models at a certain cost. While the robustness of a model increases, the standard classification accuracy declines. This phenomenon is suggested to be an inherent trade-off. We propose a model that employs feature prioritization by a nonlinear attention…
AGNN improves network localization accuracy by 37-53% in NLOS conditions.
problem Massive network localization under Non-Line-of-Sight conditions.
method Attentional Graph Neural Network (AGNN) with Adjacency Learning Module (ALM) and Multiple Graph Attention Layers (MGAL).
result Significant improvement in localization accuracy, approaching fundamental lower bounds.
New method prevents entropy collapse in Transformer training, leading to more stable and robust models.
problem Training instability in Transformers, especially in attention layers.
method Spectral normalization with a learned scalar to prevent entropy collapse.
result Prevents entropy collapse, leading to more stable training.
Machine learning models that can exploit the inherent structure in data have gained prominence. In particular, there is a surge in deep learning solutions for graph-structured data, due to its wide-spread applicability in several fields. Graph attention networks (GAT), a recent addition to the broad class of feature le…
PRISMA uses PDE residuals for fast, robust, and accurate inference.
problem Slow gradient-based optimization and instability in PDE residual-based methods.
method Integrates PDE residuals directly into the model's architecture via attention mechanisms in the spectral domain.
result Competitive accuracy with significantly lower inference costs and faster speeds.
RTFN extracts robust temporal features for time series analysis.
problem Challenges in extracting sufficient shapelets from time series data.
method Combines temporal feature networks and attentional LSTM networks.
result RTFN outperforms in supervised and unsupervised time series analysis.
MAESTRO improves multimodal learning for dynamic time series with adaptive attention and robustness.
problem Challenges in multimodal learning, especially in healthcare and daily living.
method Dynamic intra- and cross-modal interactions, symbolic tokenization, adaptive attention budgeting, sparse cross-modal attention, MoE mechanism.
result Average relative improvements of 4% and 8% over existing multimodal and multivariate approaches, respectively, under complete observations.
Network embedding represents nodes in a continuous vector space and preserves structure information from the Network. Existing methods usually adopt a "one-size-fits-all" approach when concerning multi-scale structure information, such as first- and second-order proximity of nodes, ignoring the fact that different scal…
1-Lipschitz networks are as accurate as classical networks and offer robustness.
problem Misconceptions about 1-Lipschitz neural networks and their properties.
method Analysis of 1-Lipschitz neural networks' accuracy, robustness, and generalization.
result 1-Lipschitz neural networks are as accurate as classical networks and can fit arbitrarily difficult boundaries.
Graph attention improves node classification by distinguishing important edges.
problem Node classification in graph-based learning models.
method Theoretical analysis of graph attention networks for node classification.
result Graph attention can perfectly classify nodes in an 'easy' regime but fails in a 'hard' regime.
New attention model removes softmax, improving sequence length bias.
problem Softmax attention's limitations and sequence length bias.
method Replaces softmax with normalization in self-attention.
result Normalization leads to more robust and bias-free attention.
Elliptical Attention improves transformer performance by focusing on contextually relevant features.
problem Transformer models suffer from representation collapse and are vulnerable to contaminated samples.
method Uses Mahalanobis distance to define hyper-ellipsoidal neighborhoods for attention weights.
result Elliptical Attention reduces representation collapse and enhances model robustness.
Method matches noisy remote sensing images robustly.
problem Matching noisy remote sensing images.
method Combining attention mechanism with feature enhancement.
result More efficient and accurate matches achieved.
LLMs are vulnerable to task-irrelevant data changes, limiting their use for data fitting.
problem LLMs' sensitivity to task-irrelevant variations in data representation.
method Analysis of LLMs' performance and attention patterns under various data manipulations.
result LLMs are sensitive to task-irrelevant variations, leading to significant prediction errors.
The paper addresses adversarial robustness in in-context learning models.
problem Adversarial distribution shifts threaten the reliability of in-context learning models.
method A distributionally robust meta-learning framework is introduced to provide worst-case performance guarantees under Wasserstein-based distribution shifts.
result Model robustness scales with the square root of its capacity and is penalized by the square of the perturbation magnitude.
The concepts of risk-aversion, chance-constrained optimization, and robust optimization have developed significantly over the last decade. Statistical learning community has also witnessed a rapid theoretical and applied growth by relying on these concepts. A modeling framework, called distributionally robust optimizat…
Paper investigates Lipschitz constants of self-attention modules in neural networks.
problem Lipschitz constants of self-attention modules in neural networks.
method Proved standard dot-product self-attention is not Lipschitz for unbounded input domain. Proposed L2 self-attention that is Lipschitz. Derived upper bound on L2 self-attention's Lipschitz constant.
result Proved standard self-attention is not Lipschitz for unbounded input domain and proposed an alternative L2 self-attention that is Lipschitz.
Paper introduces a neural framework for accurate energy forecasting.
problem Challenges of forecasting energy demand and supply due to variability of renewable sources and dynamic consumption patterns.
method Integrates Neural ODEs, graph attention, multi-resolution wavelet transformations, and adaptive learning of frequencies.
result Consistently outperforms state-of-the-art baselines in various forecasting metrics across diverse datasets.
Deep neural networks, including recurrent networks, have been successfully applied to human activity recognition. Unfortunately, the final representation learned by recurrent networks might encode some noise (irrelevant signal components, unimportant sensor modalities, etc.). Besides, it is difficult to interpret the r…
Visual Question Answering (VQA) requires AI models to comprehend data in two domains, vision and text. Current state-of-the-art models use learned attention mechanisms to extract relevant information from the input domains to answer a certain question. Thus, robust attention mechanisms are essential for powerful VQA mo…
Learning distributed node representations in networks has been attracting increasing attention recently due to its effectiveness in a variety of applications. Existing approaches usually study networks with a single type of proximity between nodes, which defines a single view of a network. However, in reality there usu…
Attention mechanism combines bottom-up and top-down signals in neural networks.
problem Combining robust perception with bottom-up and top-down signals.
method Attention mechanism over modulated recurrent neural networks.
result Bidirectional information flow leads to improved performance in various tasks.
BAM model learns graph structure from data with robustness across linear and non-linear dependencies.
problem Detecting dependencies in datasets for graph structure learning.
method Proposes BAM, a neural network model using structural equation models and Chebyshev polynomials for training, with bilinear attention mechanism.
result Demonstrates robust generalizability and superior performance in graph estimation.
Looped Transformers improve robustness and expressivity in in-context learning for diverse tasks.
problem Improving robustness and expressivity in in-context learning for diverse tasks.
method Study in-context linear regression with diverse tasks, focusing on depth and looping.
result Looped Transformers exhibit similar expressive power and are provably robust under mild assumptions.
LSTM-based speaker verification usually uses a fixed-length local segment randomly truncated from an utterance to learn the utterance-level speaker embedding, while using the average embedding of all segments of a test utterance to verify the speaker, which results in a critical mismatch between testing and training. T…
Improved physics-integrated generative models with noise robustness and fidelity.
problem Enhancing generative models to produce outputs that comply with physical laws and improve generalization.
method Integrating variational autoencoder with planar normalizing flow and attention mechanisms to learn latent posterior distributions and mitigate noise.
result Significant improvement in reconstruction quality and robustness against noise.
Driver vigilance estimation is an important task for transportation safety. Wearable and portable brain-computer interface devices provide a powerful means for real-time monitoring of the vigilance level of drivers to help with avoiding distracted or impaired driving. In this paper, we propose a novel multimodal archit…
Transformer models waste resources on long-context tasks.
problem Redundant attention computations in Transformer models for long-context tasks.
method Reformulate sequence modeling as supervised learning, analyze attention sparsity, formulate attention optimization as linear coding problem, propose Dynamic Group Attention.
result DGA reduces computational costs while maintaining performance.
Paper explores robustness of CCS model for matrix completion.
problem Robustness of cross-concentrated sampling model against sparse outliers.
method Proposes Robust CUR Completion (RCURC) algorithm for efficient non-convex iterative matrix completion.
result Empirical validation of RCURC's efficiency and robustness in synthetic and real datasets.
Study reveals attention mechanism's similarity computation parallels traditional machine learning.
problem Understanding the essence and principles of attention mechanism in deep learning.
method Examined classic metrics and vector space properties in manifold learning, clustering, and supervised learning to identify key characteristics of similarity computation and information propagation.
result Self-attention mechanism in deep learning adheres to the same principles but operates more flexibly and adaptively.
New techniques improve channel prediction in noisy wireless systems.
problem Predicting channels in wireless communication systems from noisy observations.
method Adapted sequence-to-sequence models and transformers with reverse positional encoding and reversed encoder outputs.
result Improved robustness and relationship capture in channel prediction models.
Incorporating human domain knowledge for breast tumor diagnosis is challenging, since shape, boundary, curvature, intensity, or other common medical priors vary significantly across patients and cannot be employed. This work proposes a new approach for integrating visual saliency into a deep learning model for breast t…
GCL-LRR improves node classification in noisy graphs.
problem Noise in real-world graph data impairs GNNs' effectiveness.
method Two-stage transductive learning with low-rank regularization and attention.
result Improved node classification performance in noisy graphs.
CgNN uses network structure as IVs to estimate causal effects in networks.
problem Hidden confounders complicate causal effect estimation in network data.
method CgNN combines GNNs and attention mechanisms to leverage network structure as IVs.
result CgNN effectively mitigates hidden confounder bias and improves causal effect estimation.
Paper proposes integrating wavelet transform, channel attention, and LSTM for better stock price prediction.
problem Inherently difficult stock price prediction due to low signal-to-noise ratio.
method Wavelet transform convolution, channel attention, and LSTM integration.
result Robust performance in post-pandemic market conditions.
An important problem in multiview representation learning is finding the optimal combination of views with respect to the specific task at hand. To this end, we introduce NAM: a Neural Attentive Multiview machine that learns multiview item representations and similarity by employing a novel attention mechanism. NAM har…
We characterize when a convex risk measure associated to a law-invariant acceptance set in L∞ can be extended to Lp, 1≤p<∞, preserving finiteness and continuity. This problem is strongly connected to the statistical robustness of the corresponding risk measures. Special attention is paid to concre…
Clustering is one of the most important unsupervised problems in machine learning and statistics. Among many existing algorithms, kernel k-means has drawn much research attention due to its ability to find non-linear cluster boundaries and its inherent simplicity. There are two main approaches for kernel k-means: SVD o…
Support vector machines have attracted much attention in theoretical and in applied statistics. Main topics of recent interest are consistency, learning rates and robustness. In this article, it is shown that support vector machines are qualitatively robust. Since support vector machines can be represented by a functio…
New method improves robustness of smoothed classifiers against adversarial attacks.
problem Improving robustness of smoothed classifiers against adversarial attacks.
method Proposes worst-case adversarial loss over input distributions as a robustness certificate, and uses duality and smoothness properties to provide an easy-to-compute upper bound.
result Shows superior robustness performance over state-of-the-art certified or heuristic methods.