BiPE blends intra-segment and inter-segment encodings for better length extrapolation.
problem Improving length extrapolation in language models.
method Bilevel Positional Encoding (BiPE) that separates intra-segment and inter-segment encodings.
result BiPE enhances length extrapolation across various text modalities.
STRING improves 2D and 3D position encodings for better performance.
problem Efficient and accurate position encoding for 2D and 3D applications.
method STRING extends Rotary Position Encodings with a unifying theoretical framework, maintaining translation invariance and low computational cost.
result STRING shows substantial gains in open-vocabulary object detection and robotics.
Randomized positional encodings boost transformer performance on longer sequences.
problem Transformers struggle with generalizing to sequences of arbitrary length.
method Introduced randomized positional encodings that simulate longer sequences and randomly select positions.
result Randomized positional encodings increase test accuracy by 12.0% on average for sequences of unseen length.
New method for Transformer models to encode position information without sequential bias.
problem Lack of flexible and learnable position encoding for Transformer models.
method Continuous dynamical model to learn position encoding.
result Consistent improvements over baselines in various NLP tasks.
Paper introduces a new method for Transformers with linear complexity.
problem No efficient relative positional encoding for linear Transformer models.
method Stochastic Positional Encoding (SPE) that replaces classical RPE.
result SPE behaves like RPE and performs well on benchmarks.
Efficiently accelerates attention calculation for Transformers with relative positional encoding.
problem Quadratic complexity of attention in long sequences.
method Kernelized attention with Fast Fourier Transform (FFT) for RPE.
result Achieves O(n log n) time complexity, mitigates training instability, and outperforms other models.
Transformer struggles with arithmetic length but improves with explicit structure encoding.
problem Transformers fail to generalize length in arithmetic tasks.
method Explicitly encoding structural symmetries via modified number formatting and custom positional encodings.
result Transformer can generalize up to 50-digit numbers without additional data.
Paper proposes LCP for structural encodings, outperforming existing methods.
problem Improving Graph Neural Networks performance through effective structural encodings.
method Geometric perspective, Local Curvature Profiles (LCP) for structural encodings, combining with global positional encodings, comparing with rewiring techniques.
result LCP significantly outperforms existing structural encodings and combining LCP with global positional encodings improves performance.
GTA improves transformer-based NVS models by encoding geometric structure.
problem Suboptimal positional encoding for 3D vision tasks.
method Geometry-aware attention mechanism encoding geometric structure of tokens.
result GTA improves learning efficiency and performance of NVS models.
Unified framework analyzes and compares RFF and RoPE PEs for music generation.
problem Efficiently modeling music generation with positional encodings.
method Kernel methods to analyze and compare RFF and RoPE PEs.
result RoPEPool outperforms other methods in melody harmonization.
Binary encoding enables neural networks to extrapolate periodic functions.
problem Extrapolating periodic functions without prior knowledge of their form.
method Normalized Base-2 Encoding (NB2E) for continuous numerical values.
result MLPs using NB2E can successfully extrapolate diverse periodic signals.
Graph attention networks improve performance on heterogeneous graphs.
problem Complex performance of GNNs on heterogeneous graphs.
method Integrating positional encodings into graph attention networks.
result Graph attention networks excel in node classification and link prediction.
This paper explores GNN functions on random graphs, highlighting the importance of node Positional Encodings.
problem Understanding the expressive power of GNNs on large random graphs.
method General convergence notions, input node features, and Positional Encodings (PEs).
result GNNs can converge to certain functions on large random graphs, emphasizing the role of PEs.
New analysis shows PE in Transformers increases generalization gap and vulnerability.
problem Understanding the impact of PE on Transformer generalization and robustness.
method Generalization analysis and adversarial Rademacher bounds for a single-layer Transformer with trainable PE.
result PE systematically enlarges the generalization gap and makes models more vulnerable to attacks.
PE-GQNN improves spatial data prediction and uncertainty quantification.
problem Poor calibration of predictive distributions in spatial data models.
method Combines PE-GNNs with Quantile Neural Networks and recalibration techniques.
result PE-GQNN outperforms existing methods in predictive accuracy and uncertainty quantification.
GSA-Nets apply group equivariance to self-attention for vision tasks.
problem Improving self-attention networks for vision tasks.
method Define group-equivariant positional encodings.
result GSA-Nets outperform non-equivariant self-attention networks on vision benchmarks.
New method for directed graphs using learnable spectral positional encodings.
problem Challenges in magnetic Laplacians and unitary gauge invariance for directed graphs.
method Learnable spectral PEs of the form hθ(Aq)R, computed in Hermitian block Krylov subspace.
result Gauge-invariant and computationally efficient solution for directed graphs.
A new method for spectral positional encodings in directed graphs using Hermitian block Krylov subspaces.
problem Challenges in spectral positional encodings for directed graphs, including computational complexity and gauge invariance issues.
method Learnable spectral positional encodings of the form hθ(Aq)R, computed in a Hermitian block Krylov subspace from sparse matrix-vector products. result The method is gauge-invariant and converges to the exact eigendecomposition oracle as the depth grows.
New techniques improve channel prediction in noisy wireless systems.
problem Predicting channels in wireless communication systems from noisy observations.
method Adapted sequence-to-sequence models and transformers with reverse positional encoding and reversed encoder outputs.
result Improved robustness and relationship capture in channel prediction models.
Conformer encoder reverses sequence in time dimension, affecting decoder training.
problem Reversal of sequence in Conformer encoder impacts decoder training.
method Analyzed initial behavior of decoder cross-attention and proposed methods to avoid flipping.
result Self-attention module of Conformer starts dominating, allowing only reversed information to pass.
Brain networks in fMRI are typically identified using spatial independent component analysis (ICA), yet mathematical constraints such as sparse coding and positivity both provide alternate biologically-plausible frameworks for generating brain networks. Non-negative Matrix Factorization (NMF) would suppress negative BO…
Affine spiking neural networks learn efficiently and generalize well.
problem Learning with spiking neural networks, especially with positive weights.
method Affine encoders and decoders, continuous parameter dependence, gradient-based training.
result Affine spiking neural networks can approximate shallow ReLU networks and generalize well.
Financial fraud detection in digital banking requires reasoning over multiple heterogeneous event streams.
problem Financial fraud detection in digital banking requires reasoning over multiple heterogeneous event streams.
method Multi-Stream Fraud Transformer (MSFT) architecture that encodes each event stream with independent Transformer encoders and fuses their representations through configurable mechanisms.
result Sequence models significantly outperform gradient-boosted trees operating on aggregated features.
This paper proposes grid cells encode position via a conformal isometric embedding of 2D physical space.
problem Hexagonal grid firing patterns in grid cells.
method Learning a distance-preserving position embedding in neural space using a recurrent neural network.
result The conformal isometric embedding of 2D physical space into neural space explains hexagonal grid firing patterns.
New insights into how encoder-decoder networks generate attention matrices.
problem Understanding how encoder-decoder networks use attention matrices.
method Decomposing hidden states into temporal and input-driven components.
result Attention matrices are formed based on task requirements, not architecture type.
CT improves neural network performance on cell complex data.
problem Improving predictive performance of neural networks on complex data.
method Introducing the Cellular Transformer (CT) that generalizes graph-based transformers to cell complexes.
result CT achieves state-of-the-art performance on cell complex datasets without complex enhancements.
ETC improves Transformer models for long and structured inputs.
problem Scaling input length and encoding structured inputs in Transformers.
method Introduces global-local attention, relative position encodings, and CPC pre-training.
result Achieves state-of-the-art results on four natural language datasets.
Introduce Collapsed Effective Operators for higher-order structures.
problem Existing spectral operators decompose topology into separate ranks, leaving practitioners to fuse information back to vertices.
method Introduce Collapsed Effective Operators via Schur complementation of a graded Laplacian.
result Preserves positive semi-definiteness, lowers system energy under higher-order connectivity.
This paper introduces a new task to better understand Transformers in quantitative contexts.
problem Understanding Transformers in high-stakes quantitative and scientific applications.
method Introduces a novel contextual counting task and analyzes it with causal and non-causal Transformer architectures.
result Causal attention is better suited for the contextual counting task, and no positional embeddings lead to the best accuracy.
FraudTransformer detects payment fraud by preserving event order and time gaps.
problem Detecting payment fraud in real-world banking streams with irregular time gaps.
method Augments a GPT-style architecture with a dedicated time encoder and a learned positional encoder.
result FraudTransformer outperforms classical and transformer baselines, achieving highest AUROC and PRAUC on held-out test set.
cMIM improves representation learning without positive-pair augmentations.
problem Learning robust representations for diverse tasks.
method Contrastive Mutual Information Machine (cMIM) framework.
result cMIM outperforms MIM and InfoNCE on classification and regression tasks.
We construct a combinatorial invariant of Legendrian knots in standard contact three-space. This invariant, which encodes rational relative Symplectic Field Theory and extends contact homology, counts holomorphic disks with an arbitrary number of positive punctures. The construction uses ideas from string topology.
Researchers prove Transformers are Turing-complete and analyze their components.
problem Understanding the computational power and limitations of Transformers.
method Analyzed Turing-completeness of vanilla and modified Transformers, and necessity of components.
result Transformers with positional masking and positional encodings are Turing-complete.
SymPE breaks symmetries in equivariant networks, improving performance across various tasks.
problem Equivariant networks cannot break symmetries, leading to poor performance in tasks with symmetrical inputs.
method Novel equivariant conditional distributions and randomized canonicalization.
result SymPE significantly improves performance of group-equivariant and graph neural networks.
In this paper, we propose an auto-encoder based generative neural network model whose encoder compresses the inputs into vectors in the tangent space of a special Lie group manifold: upper triangular positive definite affine transform matrices (UTDATs). UTDATs are representations of Gaussian distributions and can strai…
The paper describes fitting submanifolds to data using Sussmann's orbit theorem.
problem Fitting an immersed submanifold to random samples.
method Uses Sussmann's orbit theorem to ensure submanifold fitting. Reconstruction involves encoding times and decoding via flows of vector fields.
result A high-probability bound on excess risk for the reconstruction error.
Space2Vec learns multi-scale spatial representations from grid cell insights.
problem Encoding spatial features with varying scales from GIS data.
method Proposes Space2Vec, a multi-scale representation learning model using grid cell insights.
result Space2Vec outperforms baselines in predicting POI types and image classification with geo-locations.
Transformer learns graph structure better with subgraph info.
problem Transformer struggles with structural similarity in graph learning.
method Structure-Aware Transformer with subgraph attention.
result Improves graph prediction benchmarks significantly.
New analysis shows RPE-based Transformers can't approximate all functions.
problem Understanding the limitations of RPE-based Transformers in approximating continuous functions.
method Mathematical analysis and development of a novel attention module (URPE) to overcome limitations.
result RPE-based Transformers can't approximate all continuous sequence-to-sequence functions, even with depth and width.
Generative models have achieved impressive results in many domains including image and text generation. In the natural sciences, generative models have led to rapid progress in automated drug discovery. Many of the current methods focus on either 1-D or 2-D representations of typically small, drug-like molecules. Howev…
In this paper, we present our method of using fixed-size ordinally forgetting encoding (FOFE) to solve the word sense disambiguation (WSD) problem. FOFE enables us to encode variable-length sequence of words into a theoretically unique fixed-size representation that can be fed into a feed forward neural network (FFNN),…
In this paper, a multi-layer architecture (in a hierarchical fashion) by stacking various Kernel Ridge Regression (KRR) based Auto-Encoder for one-class classification is proposed and is referred as MKOC. MKOC has many layers of Auto-Encoders to project the input features into new feature space and the last layer was r…
We consider sphere bundles P and P' of totally null planes of maximal dimension and opposite self-duality over a 4-dimensional manifold equipped with a Weyl or Riemannian geometry. The fibre product PP' of P and P' is found to be appropriate for the encoding of both the selfdual and the Einstein-Weyl equations for the …
New method to encode Weinstein 4-manifolds using multisections with divides.
problem Encoding and manipulating Weinstein 4-manifolds.
method Using multisection diagrams with divides and algorithms based on Kirby-Weinstein handle decompositions and PALFs.
result Definition and symplectic implementation of monodromy on multisections with divides.
GraphReach improves GNN performance by incorporating node positions.
problem Existing GNNs fail to capture node positions, leading to inaccurate predictions.
method GraphReach uses reachability estimations from anchor nodes to capture global node positions.
result GraphReach achieves up to 40% relative improvement in accuracy compared to state-of-the-art GNNs.
We propose to formulate multi-label learning as a estimation of class distribution in a non-linear embedding space, where for each label, its positive data embeddings and negative data embeddings distribute compactly to form a positive component and negative component respectively, while the positive component and nega…
Block and Weinberger show that an arithmetic manifold can be endowed with a positive scalar curvature metric if and only if its $\rationals$-rank exceeds 2. We show in this article that these metrics are never in the same coarse class as the natural metric inherited from the base Lie group. Furthering the coarse $C^\as…
Medical imaging models may encode demographic attributes without violating fairness, depending on the approach.
problem Discrimination in medical imaging models due to encoding demographic attributes.
method Examined marginal and class-conditional representation invariance, traditional fairness notions, and counterfactual fairness.
result Demographically invariant models may not necessarily be fair, and encoding demographic attributes can be advantageous.