Paper proposes multiscale self-attentive convolutions for vision and language.
problem Improving language and vision understanding models using self-attention.
method Developed 1D and 2D Self Attentive Convolutions (SAC), multiscale SAC (MSAC).
result MSAC enhances model performance for vision and language tasks.
Self-attention can replace convolutional layers in vision tasks.
problem The supremacy of convolutional layers in vision tasks.
method Analysis and experiments with self-attention layers compared to convolutional layers.
result Self-attention layers can perform as well as convolutional layers and learn to do so.
Improved hybrid acoustic model using interleaved self-attention and convolution.
problem Limited application of transformer in hybrid acoustic models.
method Proposed a model structure with interleaved self-attention and 1D convolution.
result Competitive recognition results on Librispeech dataset.
Model detects electricity theft with high accuracy.
problem Detecting electricity theft on imbalanced datasets.
method Multi-head self-attention mechanism with dilated convolutions and binary mask.
result Achieved AUC of 0.926, improving previous work by 17%.
Improved speech enhancement with MNTFA using time-frequency attention.
problem Speech enhancement with limited model size and memory.
method Designing MNTFA with self-attention modules for long sequences and joint training.
result MNTFA achieves better performance with fewer parameters than DPCRN.
ConViT combines CNN and ViT strengths, improving image classification.
problem Combining the strengths of CNNs and ViTs while avoiding their limitations.
method Introducing GPSA, a form of positional self-attention with a soft convolutional inductive bias.
result ConViT outperforms DeiT on ImageNet while offering improved sample efficiency.
The paper investigates how Global Self-attention improves GCNs.
problem Improving expressive power and addressing overfitting in GCNs.
method Applying Global Self-attention mechanism over graph features.
result GSA mechanism enhances expressive power and mitigates overfitting and over-smoothing in GCNs.
A new neural network separates vocals from music accompaniment.
problem Separating vocals from music accompaniment in recordings.
method Self-attention convolutional neural network (CNN) with densely-connected blocks.
result 19.5% relative improvement in vocals separation.
Advanced methods of applying deep learning to structured data such as graphs have been proposed in recent years. In particular, studies have focused on generalizing convolutional neural networks to graph data, which includes redefining the convolution and the downsampling (pooling) operations for graphs. The method of …
Transformers predict price movements from limit order books.
problem Predicting price movements from limit order books.
method Causal convolutional network with masked self-attention.
result Significantly outperforms existing architectures on FI-2010 dataset.
Active-memory mechanisms can replace self-attention in Transformers, but optimal results often require both.
problem Replacing self-attention with active-memory mechanisms in Transformers.
method Evaluation of various active-memory mechanisms in a Transformer model.
result Active-memory mechanisms can achieve comparable results to self-attention for language modeling, but optimal results are often achieved by combining both mechanisms.
LieTransformer extends self-attention to Lie groups for improved deep learning tasks.
problem Improving deep learning performance through group equivariant self-attention.
method LieSelfAttention layers that are equivariant to arbitrary Lie groups and their discrete subgroups.
result Competitive experimental results on various tasks.
Paper investigates Lipschitz constants of self-attention modules in neural networks.
problem Lipschitz constants of self-attention modules in neural networks.
method Proved standard dot-product self-attention is not Lipschitz for unbounded input domain. Proposed L2 self-attention that is Lipschitz. Derived upper bound on L2 self-attention's Lipschitz constant.
result Proved standard self-attention is not Lipschitz for unbounded input domain and proposed an alternative L2 self-attention that is Lipschitz.
Self-attention improves satellite time series classification without preprocessing.
problem Efficiently classifying raw satellite time series data.
method Comparison of deep learning models including self-attention, 1D-convolutions, recurrence, and random forest.
result Self-attention and recurrent neural networks outperform convolutional neural networks on raw satellite time series.
Self-attentional models improve chatbot efficiency and performance.
problem Training efficient task-oriented dialogue generation systems.
method Applied self-attentional models to three datasets for chatbot training.
result Self-attentional models outperform recurrence-based models in efficiency and performance.
A method to generate long-range human actions by leveraging graph convolutional networks and self-attention.
problem Generating long-range skeleton-based human actions is challenging due to small frame deviations.
method Proposes a variant of GCNs with self-attention to adaptively sparsify action graphs and capture structure information.
result Extensive experiments show superior performance compared to existing methods on human action datasets.
Paper tackles time series forecasting weaknesses in Transformers.
problem Transformer's locality-agnostic and memory-bottleneck issues in time series forecasting.
method Proposes convolutional self-attention and LogSparse Transformer to address locality and memory issues.
result Improves forecasting accuracy for fine-granularity, long-term dependency time series.
TSAM predicts directed temporal links using GCN and self-attention.
problem Predicting links in directed temporal networks.
method GCN, self-attention mechanism, autoencoder architecture, graph attentional layers, graph convolutional layers, graph recurrent unit layer.
result TSAM outperforms benchmarks on four realistic networks.
Multi-modal data comprising imaging (MRI, fMRI, PET, etc.) and non-imaging (clinical test, demographics, etc.) data can be collected together and used for disease prediction. Such diverse data gives complementary information about the patientś condition to make an informed diagnosis. A model capable of leveraging the i…
Vision Transformers show different internal representations compared to CNNs.
problem Understanding how Vision Transformers solve image classification tasks.
method Comparative analysis of ViT and CNN architectures on image classification benchmarks.
result ViT has more uniform representations across all layers, while CNNs have more varied representations.
Self-attention model improves HAR from wearable sensors.
problem Capturing spatio-temporal context from sensor data.
method Proposes a self-attention based neural network model.
result Significant performance improvement over state-of-the-art models.
Insurance companies must manage millions of claims per year. While most of these claims are non-fraudulent, fraud detection is core for insurance companies. The ultimate goal is a predictive model to single out the fraudulent claims and pay out the non-fraudulent ones immediately. Modern machine learning methods are we…
This paper offers an overview of neural network compression techniques.
problem Overparameterized neural networks are large and resource-intensive.
method Pruning, quantization, tensor decomposition, knowledge distillation.
result A comprehensive review of compression techniques for deep neural networks.
Trans-Unet predicts brain folding patterns from 3D point-clouds using novel 3D-to-2D transformation.
problem Challenges in learning high-fidelity 3D point-cloud features, including permutation invariance and fine-grained surface reconstruction.
method Transform 3D point-clouds into a 2D grid domain, then use a U-shaped hybrid model with CNNs and self-attention mechanisms.
result Trans-Unet achieves high-resolution predictions of brain patch growth, surpassing existing methods in fidelity and accuracy.
SAG-VAE learns data representations and feature relations end-to-end.
problem Vanilla VAEs cannot learn relations between features.
method Inspired by Graph Neural Networks, SAG-VAE jointly infers data representations and feature relations.
result SAG-VAE generates new data via graph convolution and is robust to perturbations.
TaLK Convolutions improve sequence modeling efficiency.
problem Efficiently modeling sequences with limited time complexity.
method Adaptive convolution operation that learns kernel size.
result Time complexity reduced to O(n), making sequence encoding linear. Paper develops a new model for predicting volatility surface.
problem Predicting volatility in financial markets is challenging due to its non-observable nature and complex dynamics.
method Physics-informed convolutional transformer architecture.
result The new model outperforms other deep-learning architectures in predicting volatility surface.
New model predicts ICU patients' stay duration efficiently.
problem Efficient ICU bed allocation under resource constraints.
method Temporal Pointwise Convolution (TPC) model combining temporal and pointwise convolutions.
result Significant performance improvements over LSTM and Transformer models.
GSA-Nets apply group equivariance to self-attention for vision tasks.
problem Improving self-attention networks for vision tasks.
method Define group-equivariant positional encodings.
result GSA-Nets outperform non-equivariant self-attention networks on vision benchmarks.
In this paper, we propose the Self-Attention Generative Adversarial Network (SAGAN) which allows attention-driven, long-range dependency modeling for image generation tasks. Traditional convolutional GANs generate high-resolution details as a function of only spatially local points in lower-resolution feature maps. In …
Self-attention prefers sparse functions of input sequences, reducing sample complexity.
problem Understanding the inductive biases of self-attention in modeling long-range dependencies.
method Theoretical analysis and synthetic experiments to probe sample complexity of learning sparse functions with Transformers.
result Bounded-norm Transformer networks can represent sparse functions of the input sequence with logarithmic sample complexity.
Self-attention models benefit equally from width and depth, but beyond a certain point, depth becomes less efficient.
problem Understanding the optimal balance between depth and width in self-attention models.
method Theoretical predictions and empirical ablations on networks of varying depths and widths.
result An optimal width of 30K is recommended for a 1-Trillion parameter network, marking a significant width for self-attention models.
Improves sequential recommendation with relation-aware self-attention.
problem Improving accuracy in sequential recommendation.
method Integrates Transformer's self-attention mechanism with a probabilistic model of recommendation context.
result Significant improvements over recent baseline models.
Study algebraic invariants from lightning self-attention models.
problem Understanding polynomial coefficients of self-attention mechanisms.
method Identify algebraic invariants using polynomial coefficients and coordinate geometry.
result Found linear and nonlinear families of algebraic invariants.
Random forests with attention and self-attention improve regression performance.
problem Improving regression model performance on various datasets.
method Proposes new models using attention and self-attention mechanisms to solve regression problems.
result The models improve model performance on many datasets.
Kernel PCA explains self-attention mechanisms in deep learning models.
problem Understanding and explaining self-attention mechanisms in deep learning models.
method Deriving self-attention from kernel principal component analysis (kernel PCA).
result RPC-Attention, a robust attention mechanism, outperforms softmax attention in various tasks.
Proposes a faster Transformer decoding method by truncating target-side self-attention windows.
problem Efficiency in Transformer decoding with minimal BLEU score loss.
method N-gram assumption to truncate target-side self-attention windows.
result N-gram masked self-attention model maintains BLEU score for N values from 4 to 8. Study reveals self-attention's role in learning and generalizing interactions.
problem Understanding self-attention's theoretical role in neural architectures.
method Interacting entities analysis, including multi-agent RL and genetic sequences.
result Self-attention efficiently represents, learns, and generalizes pairwise interactions.
Improved non-intrusive load monitoring with a novel neural network.
problem Accurately disaggregating household electricity consumption without dedicated meters.
method Developed a scale- and context-aware network with multi-scale features and contextual information.
result Significantly improved accuracy compared to state-of-the-art methods.
Linformer reduces transformer complexity to linear, improving efficiency.
problem High cost of training and deploying large transformer models for long sequences.
method Approximates self-attention with low-rank matrix, proposing Linformer with O(n) complexity. result Linformer performs similarly to standard transformers but is more memory- and time-efficient.
Graph Attention Networks improve image classification with superpixels.
problem Classifying images with irregular shapes and edges.
method Transform images into superpixel graphs, then apply GATs.
result GATs outperform other GNN models in image classification.
Transformer models perform slower than convolutional networks in learning hierarchical language structures.
problem Understanding how neural networks learn hierarchical language structures.
method Theoretical scaling laws and empirical validation of neural network performance.
result Convolutional networks outperform transformers in learning hierarchical language structures.
Skeinformer accelerates self-attention for long sequences with linear complexity.
problem Efficiency of Transformer models in processing long sequences.
method Matrix sketching and column sampling to reduce quadratic complexity to linear.
result Skeinformer outperforms alternatives with smaller time/space footprint.
Deep autoregressive sequence-to-sequence models have demonstrated impressive performance across a wide variety of tasks in recent years. While common architecture classes such as recurrent, convolutional, and self-attention networks make different trade-offs between the amount of computation needed per layer and the le…
Paper connects MoE and self-attention, proposing active-attention.
problem Improving efficiency and performance of self-attention mechanisms.
method Established connection between MoE and self-attention, analyzed quadratic gating functions, proposed active-attention mechanism.
result Active-attention outperforms standard self-attention in various tasks.
Gradient descent converges geometrically to optimal self-attention parameters.
problem Training softmax self-attention layers for linear regression.
method Structure-aware gradient descent with preconditioner and regularizer.
result Gradient descent converges geometrically to global minima.
MGMC method handles missing data in medical datasets for accurate disease classification.
problem Handling missing data in incomplete medical datasets for accurate disease classification.
method Multigraph Geometric Matrix Completion (MGMC) using multiple graph convolutional networks.
result MGMC achieves superior classification and imputation performance compared to state-of-the-art approaches.
New method embeds time span into self-attention for better temporal pattern recognition.
problem Capturing temporal patterns in event sequences without recurrent networks.
method Functional time representation learning with Bochner's and Mercer's Theorems.
result Proposed methods outperform baseline models in various continuous-time event sequence prediction tasks.