Transformation Equivariant Representations (TERs) aim to capture the intrinsic visual structures that equivary to various transformations by expanding the notion of {\em translation} equivariance underlying the success of Convolutional Neural Networks (CNNs). For this purpose, we present both deterministic AutoEncoding…
New Feedback Transformer architecture improves model performance by exposing past representations to future.
problem Limitations of Transformers in fully exploiting sequential input.
method Proposes Feedback Transformer exposing all past representations to future.
result Demonstrates improved performance with smaller, shallower models.
Unified framework for learning function representations using INRs and Transformers.
problem Scalability and efficiency limitations in existing generative models.
method Integrates INRs and Transformer-based hypernetworks into latent variable models.
result Improved scalability, expressiveness, and generalization over existing models.
Transformer learns graph structure better with subgraph info.
problem Transformer struggles with structural similarity in graph learning.
method Structure-Aware Transformer with subgraph attention.
result Improves graph prediction benchmarks significantly.
The Weyl transform is introduced as a rich framework for data representation. Transform coefficients are connected to the Walsh-Hadamard transform of multiscale autocorrelations, and different forms of dyadic periodicity in a signal are shown to appear as different features in its Weyl coefficients. The Weyl transform …
Relational data representations have become an increasingly important topic due to the recent proliferation of network datasets (e.g., social, biological, information networks) and a corresponding increase in the application of statistical relational learning (SRL) algorithms to these domains. In this article, we exami…
PatchGT uses non-trainable graph patches to improve graph representation learning.
problem Learning high-level information in graph tasks with direct Transformer models.
method PatchGT segments graphs into non-trainable patches, uses GNN for patch-level learning, and Transformer for graph-level learning.
result PatchGT achieves higher expressiveness and competitive performance on benchmark datasets.
We present a signal representation framework called the sparse manifold transform that combines key ideas from sparse coding, manifold learning, and slow feature analysis. It turns non-linear transformations in the primary sensory signal space into linear interpolations in a representational embedding space while maint…
Improves contrastive learning invariance with novel training objectives and feature averaging.
problem Contrastive learning's implicit invariance is insufficient for robust performance.
method Introduces a novel training objective and feature averaging approach to enforce invariance.
result Improved performance and robustness to transformations on downstream tasks.
Transformer learns representations from time series data for money laundering detection.
problem Detecting money laundering using structured time series data.
method Contrastive learning for representation learning, followed by scoring and thresholding.
result Transformer outperforms rule-based and LSTM methods in detecting money laundering with controlled false positives.
Researchers analyze the geometric and statistical properties of transformer model representations.
problem Understanding the semantic structure of large transformer models across various data types.
method Characterization of geometric and statistical properties through analysis of intrinsic dimension and neighbor composition.
result The semantic information of the dataset is better expressed at the end of the first peak in transformer models.
A neural scene representation framework enforcing 3D transformations.
problem Learning 3D scene representations from images without 3D supervision.
method Introducing a loss enforcing equivariance of the scene representation with 3D transformations.
result Real-time neural rendering with comparable results to models requiring minutes for inference.
A framework uses free probability to analyze Transformer models.
problem Understanding the dynamics and complexity of Transformer-based language models.
method Formal operator-theoretic analysis using free probability theory.
result Entropy-based generalization bounds derived under freeness assumptions.
We incorporate Tensor-Product Representations within the Transformer in order to better support the explicit representation of relation structure. Our Tensor-Product Transformer (TP-Transformer) sets a new state of the art on the recently-introduced Mathematics Dataset containing 56 categories of free-form math word-pr…
Transforms solutions of Davey-Stewartson II equation geometrically.
problem Solving the Davey-Stewartson II equation.
method Moutard transform and spinor representation of surfaces.
result Constructs examples of solutions with smooth initial data losing regularity.
Researchers present and compare different representations of dissipative Hamiltonian DAE systems.
problem Understanding and transforming dissipative Hamiltonian DAE systems.
method Global geometric and algebraic points of view, translations between representations, characterizations, and numerical methods for computing structural information.
result A general DAE system can be transformed into a dissipative Hamiltonian or port-Hamiltonian DAE system.
Representations in the auditory cortex might be based on mechanisms similar to the visual ventral stream; modules for building invariance to transformations and multiple layers for compositionality and selectivity. In this paper we propose the use of such computational modules for extracting invariant and discriminativ…
The key to success in machine learning (ML) is the use of effective data representations. Traditionally, data representations were hand-crafted. Recently it has been demonstrated that, given sufficient data, deep neural networks can learn effective implicit representations from simple input representations. However, fo…
This study investigates abrupt learning dynamics in Transformers, revealing plateau formation and internal representation collapse.
problem Abrupt learning in Transformers, particularly during the loss plateau.
method Investigates mechanisms of abrupt learning in shallow Transformers, focusing on attention maps and hidden states.
result Reveals plateau formation, internal representation collapse, and strong repetition bias in outputs.
How can intelligent agents solve a diverse set of tasks in a data-efficient manner? The disentangled representation learning approach posits that such an agent would benefit from separating out (disentangling) the underlying structure of the world into disjoint parts of its representation. However, there is no generall…
Learning invariant representations is an important problem in machine learning and pattern recognition. In this paper, we present a novel framework of transformation-invariant feature learning by incorporating linear transformations into the feature learning algorithms. For example, we present the transformation-invari…
The paper reinterprets knot group invariants using affine transformations.
problem Alexander invariants of knots and their geometric interpretation.
method Representation varieties of knot groups into extrmAGL1(C). result Alexander polynomial as the singular locus of a coherent sheaf.
CRATE-MAE learns structured representations from unlabeled data.
problem Learning structured representations from unlabeled data.
method Structured Diffusion with White-Box Transformers.
result CRATE-MAE achieves highly promising performance on large-scale imagery datasets.
Transforms data into separable subspaces for clustering.
problem Data is not always separable into subspaces.
method Embeds subspace clustering techniques into transform learning.
result Improves upon state-of-the-art clustering techniques.
IGT learns graph representations without supervision.
problem Building deep unsupervised graph representations.
method Generic complex-valued spectral graph architecture from Fourier transform generalization, greedy concave objective for discriminative and invariant features.
result IGT learns both discriminative and invariant features from graph topology.
We present a 2x2 Lax representation for discrete circular nets of constant negative Gauß curvature. It is tightly linked to the 4D consistency of the Lax representation of discrete K-nets (in asymptotic line parametrization). The description gives rise to Bäcklund transformations and an associated family. All the membe…
New representations for discrete surfaces derived from dual transforms.
problem Constructing discrete surfaces in differential geometry.
method Using Ω-dual transform and lightlike Gauss maps in Laguerre geometry. result All discrete linear Weingarten surfaces arise via Weierstrass-type representations.
Normality equations describe Newtonian dynamical systems admitting normal shift of hypersurfaces. These equations were first derived in Euclidean geometry. Then very soon they were rederived in Riemannian and in Finslerian geometry. Recently I have found that normality equations can be derived in geometry given by clas…
Enhances autoencoders to represent transformations explicitly.
problem Lack of explicit representation of transformations in autoencoders.
method Extended variational autoencoders to include latent transformations, using hierarchical graphical models.
result Inferred latent transformations reflect interpretable properties in the observation space.
Vision Transformers show different internal representations compared to CNNs.
problem Understanding how Vision Transformers solve image classification tasks.
method Comparative analysis of ViT and CNN architectures on image classification benchmarks.
result ViT has more uniform representations across all layers, while CNNs have more varied representations.
Paper recovers latent causal structure and linear transformation from indirect observations.
problem Recovering latent causal structure and linear transformation from indirect observations.
method Established sufficient conditions for DAG recovery, leveraged score function properties, and used soft/hard interventions.
result Perfect recovery of latent DAG structure and linear transformation up to scaling using soft interventions, hard interventions with additional hypothesis testing.
We construct a certain cross product of two copies of the braided dual H~ of a quasitriangular Hopf algebra H, which we call the elliptic double EH, and which we use to construct representations of the punctured elliptic braid group extending the well-known representations of the planar braid group attache…
Transformers reduce redundancy by focusing on invariant relational quantities.
problem Substantial internal redundancy in Transformer models due to coordinate-dependent representations and continuous symmetries.
method Reformulate representations, attention mechanisms, and optimization dynamics in terms of invariant relational quantities, eliminating redundant degrees of freedom by construction.
result Architectures that operate directly on relational structures, providing a principled geometric framework for reducing parameter redundancy and analyzing optimization.
CascadeXML improves multi-resolution learning for XMC with transformer features.
problem Learning subset labels from millions of choices with trade-offs between performance and computation.
method End-to-end multi-resolution learning pipeline using transformer multi-layer architecture.
result Significantly outperforms existing approaches on benchmark datasets.
Transformer autoencoder learns musical style from performances.
problem Learning high-level controls over symbolic music generation.
method Aggregates encodings of input data across time to obtain global style representation.
result Improves control over performance style and melody in music generation tasks.
Sensory data are often comprised of independent content and transformation factors. For example, face images may have shapes as content and poses as transformation. To infer separately these factors from given data, various ``disentangling'' models have been proposed. However, many of these are supervised or semi-super…
Transformer-M learns molecular data in 2D or 3D formats.
problem Learning models for molecules are limited to specific data formats.
method Developed a Transformer-based model that can handle 2D and 3D molecular data.
result Transformer-M achieves strong performance on both 2D and 3D molecular tasks.
The Interaction-Transformation (IT) is a new representation for Symbolic Regression that restricts the search space into simpler, but expressive, function forms. This representation has the advantage of creating a smoother search space unlike the space generated by Expression Trees, the common representation used in Ge…
Study geometric and representation theory of statistical transformation models.
problem Understand relationships between induced structures and actions on measure spaces.
method Investigate geometric properties and symplectic actions on induced structures.
result Show equivariance of action and relationships between tangent bundles and projectivizations.
Transformer model improves source code summarization.
problem Generating readable summaries of source code.
method Transformer model with self-attention mechanism for code representation.
result Transformer model outperforms state-of-the-art techniques.
This paper tackles causal representation learning with linear and general transformations.
problem Identify and recover latent causal variables and graphs under unknown transformations.
method Score-based algorithms that use gradients of log-density functions for identifiability and achievability.
result Two stochastic hard interventions per node are sufficient for identifiability of general transformations.
The complex wave representation (CWR) converts unsigned 2D distance transforms into their corresponding wave functions. Here, the distance transform S(X) appears as the phase of the wave function φ(X)---specifically, φ(X)=exp(iS(X)/τwhere τis a free parameter. In this work, we prove a novel result using the higher-orde…
A new model improves CT image quality from low-dose scans.
problem Improving CT image quality from low-dose scans.
method Multi-layer Residual Sparsifying Transform (MRST) learning model for low-dose CT reconstruction.
result The MRST model outperforms conventional methods in maintaining subtle details.
New unsupervised learning technique learns independent kernels for better machine learning tasks.
problem Improving unsupervised representation learning for machine learning tasks.
method Stacking convolutional transforms using alternating proximal minimization scheme.
result DCTL outperforms shallow version CTL on benchmark datasets.
Multimodalities provide promising performance than unimodality in most tasks. However, learning the semantic of the representations from multimodalities efficiently is extremely challenging. To tackle this, we propose the Transformer based Cross-modal Translator (TCT) to learn unimodal sequence representations by trans…
Weierstrass-type representations have been used extensively in surface theory to create surfaces with special curvature properties. In this paper we give a unified description of these representations in terms of classical transformation theory of Ω-surfaces.
This paper compares Transformers and RNNs in various tasks, showing size differences.
problem Comparing representational capabilities of Transformers and RNNs across tasks.
method Analysis of differences in tasks like index lookup, nearest neighbor, and string equality.
result Size differences in Transformers and RNNs for various tasks.
Study of SL(2,R) representations on a once-punctured torus, showing Cantor set spectrum.
problem Characterizing SL(2,R) representations on a once-punctured torus.
method Introduction of spectrum as a subset of projective measured laminations, analysis of dynamics of cocycles.
result Spectrum of a generic representation on a once-punctured torus is a Cantor set.