Paper proposes LAHA to improve XMTC by integrating document content and label correlation.
problem Challenges in tagging documents with most relevant labels from a large label set.
method Hybrid attention deep neural network model (LAHA) that combines multi-label self-attention and adaptive fusion strategies.
result LAHA outperforms state-of-the-art methods, especially on tail labels.
This paper improves neural tangent kernels for better generalization and local elasticity.
problem Performance gap between neural tangent kernels and real-world neural networks.
method Introduces label-aware kernels using Hoeffding decomposition.
result Models trained with proposed kernels simulate NNs better in terms of generalization and local elasticity.
LAGCN improves GCN performance by identifying and using valuable neighbors.
problem Existing GCN models do not identify valuable neighbors, potentially harming performance.
method LAGCN introduces a label-aware edge classifier to refine the graph and enhance learning performance.
result LAGCN significantly improves node classification performance on benchmark datasets.
MIND models user interests with multiple vectors for better recommendation.
problem Insufficient representation of user interests in deep learning models.
method Multi-Interest Network with Dynamic routing (MIND) using capsule routing and label-aware attention.
result MIND achieves superior performance in recommendation compared to state-of-the-art methods.
Orion-Bix combines biaxial attention and meta-learning for tabular few-shot learning.
problem Scaling and generalizing tabular models with mixed numeric and categorical fields, weak feature structure, and limited labeled data.
method Orion-Bix uses biaxial attention and meta-learned in-context reasoning to efficiently capture local and global dependencies.
result Orion-Bix outperforms gradient-boosting baselines and state-of-the-art tabular models on public benchmarks.
Given labeled points in a high-dimensional vector space, we seek a low-dimensional subspace such that projecting onto this subspace maintains some prescribed distance between points of differing labels. Intended applications include compressive classification. Taking inspiration from large margin nearest neighbor class…
Domain Adaptation in 6G wireless networks: When is it green?
problem Energy consumption of Domain Adaptation (UDA) compared to single-task training in 6G wireless networks.
method Investigate energy consumption and propose a method to determine the minimum number of target domains for UDA to be more energy-efficient than retraining.
result Proposed a method to determine the minimum number of target domains for UDA to be more energy-efficient than retraining.
A new model embeds word and label hierarchies in hyperbolic space for HMLC.
problem Learning mappings from word hierarchies to label hierarchies in hierarchical multi-label classification.
method Proposes a Hyperbolic Interaction Model (HyperIM) to learn label-aware document representations in hyperbolic space.
result Demonstrates improved performance for HMLC compared to state-of-the-art methods.
Proposes a new model for noisy labels considering multiple labelers and adversarial attacks.
problem Real-world noisy label models with multiple labelers and adversarial attacks.
method Labeler-dependent noise model with adversarial attack vectors.
result State-of-the-art approaches for learning from noisy labels are defeated by adversarial label attacks.
The paper resolves the paradox of using less data in machine learning.
problem The paradox of using less data in machine learning.
method Theoretical framework and data curation strategies.
result Small curated datasets can outperform full datasets under certain conditions.
Prototype selection improved using topological data analysis.
problem Improving prototype selection methods for data compression.
method Introducing two topological prototype selector variants: TPS and BoundaryTPS.
result BoundaryTPS achieves the lowest mean Friedman rank on H1 persistence-diagram preservation. New interpretation of attention in Transformers and Graph Attention Networks.
problem Understanding and improving attention mechanisms in deep learning models.
method Decomposed attention into a kernel and a normalization term; generalized the kernel function and norm.
result Generalized attention leads to better performance on various tasks.
New approach improves multi-head attention by making heads less similar.
problem Multi-head attention can lead to similar features, reducing model expressiveness.
method Proposes a non-parametric approach using Bayesian techniques to make heads repel each other.
result Improves feature diversity, leading to better representations and performance.
Neural process model improves real-time condition monitoring signal prediction.
problem Real-time adaptation for complex condition monitoring signals.
method Label-aware neural processes encoding and reconstruction.
result Advantages in real-time adaptation, enhanced signal prediction with uncertainty quantification, and joint prediction for labels and signals.
Regularizes attention scores in vision transformers using bootstrapping.
problem Noisy and diffused attention maps in ViT limit interpretability.
method Statistical learning techniques, bootstrapping of attention scores.
result Improves shrinkage and sparsity of attention scores.
Random forests with attention and self-attention improve regression performance.
problem Improving regression model performance on various datasets.
method Proposes new models using attention and self-attention mechanisms to solve regression problems.
result The models improve model performance on many datasets.
Aligns attention distributions for improved accuracy and robustness.
problem Improving the accuracy and robustness of neural networks using attention mechanisms.
method Alignment attention that encourages key and query distributions to match within each head.
result Alignment attention leads to better accuracy, uncertainty estimation, and robustness across various tasks.
A framework for transformer attention layers derived from SVR.
problem Developing principled attention mechanisms for transformers.
method Mapping self-attention to SVR, deriving new attention types.
result Improved transformer performance and efficiency.
Elliptical Attention improves transformer performance by focusing on contextually relevant features.
problem Transformer models suffer from representation collapse and are vulnerable to contaminated samples.
method Uses Mahalanobis distance to define hyper-ellipsoidal neighborhoods for attention weights.
result Elliptical Attention reduces representation collapse and enhances model robustness.
One of the most fundamental problems in machine learning is to compare examples: Given a pair of objects we want to return a value which indicates degree of (dis)similarity. Similarity is often task specific, and pre-defined distances can perform poorly, leading to work in metric learning. However, being able to learn …
Survey categorizes attention models across various domains.
problem Understanding and improving attention models in neural networks.
method Taxonomy and review of existing techniques.
result Provides a structured overview of attention models.
Kernel PCA explains self-attention mechanisms in deep learning models.
problem Understanding and explaining self-attention mechanisms in deep learning models.
method Deriving self-attention from kernel principal component analysis (kernel PCA).
result RPC-Attention, a robust attention mechanism, outperforms softmax attention in various tasks.
Study examines asset pricing using various attention models, finding global self-attention and sliding window sparse attention models perform well.
problem Traditional asset pricing models miss temporal dependency and short memory issues.
method Investigates RNN attention models with various attention mechanisms for large-cap US stocks.
result Global self-attention and sliding window sparse attention models outperform in deriving returns and hedging risks, especially during the pandemic.
LARF improves random forests with attention mechanisms and contamination models.
problem Improving accuracy in classification tasks with random forests.
method Introduces a two-level attention mechanism and uses a mixture of contamination models.
result Significantly improved classification performance on various datasets.
Gated attention improves performance by using a hierarchical mixture of experts.
problem Improving performance of self-attention mechanisms in Transformers.
method Rigorously show that gated attention can be modeled as a hierarchical mixture of experts, providing a theoretical justification for its benefits.
result Gated attention is more sample-efficient than multi-head self-attention, requiring fewer data points to achieve the same estimation error.
Linear attention model outperforms full attention in language tasks.
problem Quadratic scaling of full attention limits model size.
method Introduced a linear attention mechanism.
result Linear attention models achieve comparable performance to full attention models.
Investigates the fundamental components of attention mechanisms.
problem Understanding the building blocks of attention in deep learning.
method Classified and studied three key mechanisms: additive, multiplicative output, and synaptic attention.
result Additive activation attention is central in proofs of lower bounds.
Multi-head attention outperforms single-head in in-context linear regression tasks.
problem Comparing performance of transformer with single-/multi-head attention in in-context learning.
method Theoretical analysis of performance of transformers with different attention mechanisms in linear regression tasks.
result Multi-head attention with a substantial embedding dimension outperforms single-head attention in in-context linear regression tasks.
S3Attention improves long sequence attention with smoothed skeleton sketching.
problem Quadratic complexity of vanilla Attention makes it unsuitable for long sequence tasks.
method S3Attention uses smoothing and matrix sketching to balance information preservation and computation. result S3Attention significantly outperforms vanilla Attention and other Attention variants. Mathematical framework for understanding attention in neural networks.
problem Lack of theoretical understanding of attention in neural networks.
method Proposes a measure-theoretic model of attention and interprets self-attention as a system of self-interacting particles.
result Shows that attention is Lipschitz-continuous under suitable assumptions.
Bayesian attention improves model performance and robustness.
problem Limited exploration of stochastic attention in neural networks.
method Introduces Bayesian attention belief networks using gamma and Weibull distributions.
result Outperforms deterministic and stochastic attention methods in accuracy and robustness.
GOAT improves attention mechanisms by learning better priors.
problem Standard attention mechanisms use a naive uniform prior, limiting flexibility and generalization.
method GOAT introduces a trainable, continuous prior that replaces the uniform assumption, maintaining compatibility with optimized kernels.
result GOAT avoids representational trade-offs and learns an extrapolatable prior that combines positional flexibility with length generalization.
Improved language models with talking-heads attention.
problem Language model perplexity and quality issues.
method Added linear projections in multi-head attention.
result Better perplexities and quality in language tasks.
SympFormer accelerates attention blocks using inertial dynamics on density spaces.
problem Improving the efficiency of self-attention blocks in Transformers.
method Introduced accelerated attention blocks derived from inertial Nesterov dynamics on density spaces.
result Accelerated attention blocks converge faster than classical blocks while preserving oracle calls.
New approach improves linear-time attention for language models.
problem Challenges of quadratic attention in long-sequence modelling, especially for discrete data.
method Reinterpreting linear attention through latent probabilistic graphical models, introducing asymmetric structure and recurrent parameterisation.
result Our model achieves competitive performance and outperforms existing linear attention variants on language modelling benchmarks.
Bayesian theory explains abrupt emergence of copy subcircuit in attention.
problem Understanding the abrupt emergence of the copy subcircuit in attention during training.
method Deriving a closed-form posterior over the attention matrix and reducing it to a low-dimensional order parameter space.
result Derive a phase transition in the amount of training data.
Attention weights may not accurately highlight important parts due to combinatorial shortcuts.
problem Inaccurate interpretation of attention weights in models.
method Theoretical analysis and design of experiments to show combinatorial shortcuts. Proposed two methods to mitigate this issue.
result Proposed methods improve the interpretability of attention mechanisms.
Analyzes attention structure in GPT-2 model, revealing specific patterns.
problem Understanding attention mechanisms in Transformer models.
method Visualized and analyzed attention for GPT-2 model, examining interactions over a corpus.
result Attention targets different parts of speech at varying depths, aligning with dependency relations in middle layers.
Improved graph attention model for noisy graphs.
problem Understanding and improving graph attention in noisy graphs.
method Proposes SuperGAT, a self-supervised graph attention network.
result SuperGAT learns more expressive attention by encoding edges.
Dual-attention GCN improves text classification by adapting to textual complexity.
problem Challenges in learning discriminative features from texts due to graph variants.
method Proposes a dual-attention GCN with connection-attention and hop-attention mechanisms.
result Achieves state-of-the-art performance on text classification tasks.
Scales attention for long contexts in LLMs.
problem Development of attention mechanisms for long context inference.
method Scale-invariant total attention and sparsity conditions, with a position-dependent transformation of logits.
result Scale-invariant attention scheme improves validation loss and long-context retrieval.
Active-memory mechanisms can replace self-attention in Transformers, but optimal results often require both.
problem Replacing self-attention with active-memory mechanisms in Transformers.
method Evaluation of various active-memory mechanisms in a Transformer model.
result Active-memory mechanisms can achieve comparable results to self-attention for language modeling, but optimal results are often achieved by combining both mechanisms.
Single-head attention approximates any function under various norms.
problem Universal approximation of functions using attention mechanisms.
method Interpreting attention as partitioning and summing linear transformations.
result Single-head attention can approximate any continuous function under L∞-norm and Lebesgue integrable functions under Lp-norm. Self-supervised approach improves reinforcement learning with attention.
problem Previous attempts at integrating attention with reinforcement learning failed to produce significant improvements.
method Utilizes Markovian properties of state input and multiple simultaneous foci of attention.
result New state-of-the-art results in the Arcade Learning Environment.
3D Axial-Attention improves lung nodule classification accuracy.
problem Limited 3D attention in existing methods.
method Proposes 3D Axial-Attention network with 3D positional encoding.
result 3D Axial-Attention achieves state-of-the-art performance.
Transformers with linear space and time complexity for accurate attention estimation.
problem Efficiently estimating attention in large-scale tasks without relying on priors.
method Performers use Fast Attention Via positive Orthogonal Random features (FAVOR+) for linear approximation of softmax attention.
result Performers achieve competitive results on various tasks, demonstrating the effectiveness of their attention-learning approach.
Neural attention has become central to many state-of-the-art models in natural language processing and related domains. Attention networks are an easy-to-train and effective method for softly simulating alignment; however, the approach does not marginalize over latent alignments in a probabilistic sense. This property …
Enhances group convolutional networks with attention to learn meaningful relationships.
problem Lack of explicit means to learn meaningful relationships among symmetry patterns.
method Introduces attentive group equivariant convolutions, applying attention during convolution.
result Consistently outperforms conventional group convolutional networks on benchmark datasets.