Proposes QQE for transforming and embedding data distributions.
problem Transforming and embedding data distributions for better representation or visualization.
method Quantile-Quantile Embedding (QQE) using quantile-quantile plot concept.
result QQE allows for better discrimination of classes in some cases.
Geometrically transforms word embeddings into a common space for better comparison.
problem Comparing embeddings from different sources is challenging.
method Applies orthogonal rotations and Mahalanobis scaling to transform embeddings into a shared latent space.
result The method improves word similarity and analogy tasks.
A new invariant captures geometric features of circle embeddings.
problem Capturing geometric features of circle embeddings invariantly.
method Chordal distance transform and persistent homology.
result Persistent homology of chordal distance transform is invariant.
Transformers learn to recall with non-orthogonal embeddings in realistic settings.
problem Understanding how transformers store and retrieve knowledge in practical scenarios.
method Analyzing a single-layer transformer with random embeddings trained on a token-retrieval task.
result Explicit formulas for the model's storage capacity reveal a multiplicative dependence on sample size, embedding dimension, and sequence length.
Small neural networks embed arbitrary metric spaces into Gaussian mixtures.
problem Embedding arbitrary metric spaces into a fixed space with low distortion.
method Probabilistic transformers of small depth and width.
result Embeddings with low metric distortion for various metric spaces.
Minimal token perturbations reveal how Transformer models process information.
problem Understanding information propagation in Transformer models for interpretability.
method Study of minimal token perturbations on embedding space.
result Rare tokens cause larger shifts, and input information mixes deeper.
We develop embeddings for nonlinear subspaces preserving vector norms.
problem Preserving vector norms in nonlinear subspaces.
method Low-distortion embeddings for subspaces under nonlinear transformations.
result First low-distortion embeddings for a wide class of nonlinear functions.
We break down transformer embeddings into interpretable components revealing hidden geometric structures.
problem Understanding the hidden geometry and interpretability of transformer models.
method Decomposed transformer embeddings into position, context, and residual components.
result Pervasive mathematical structure in transformer embeddings, including position and context vectors.
Paper unifies propositionalization and embedding for relational learning.
problem Data fusion from diverse input formats into a single table.
method Unified framework combining propositionalization and embedding.
result New algorithms outperform existing relational learners.
The complexity of a learning task is increased by transformations in the input space that preserve class identity. Visual object recognition for example is affected by changes in viewpoint, scale, illumination or planar transformations. While drastically altering the visual appearance, these changes are orthogonal to r…
InfiniteWalk connects deep network embeddings to spectral graph theory with a nonlinear transformation.
problem Learning node representations from networks with deep learning methods.
method Study of the DeepWalk objective in the limit as window size goes to infinity, linking to spectral graph embeddings with a nonlinear transformation.
result Simple binary thresholding of the Laplacian pseudoinverse can approximate DeepWalk embeddings.
FREDE efficiently embeds graphs using linear space and guarantees quality.
problem Efficiently embedding graphs with quality guarantees and linear space complexity.
method FREDE combines matrix sketching with a nonlinear transform of PageRank similarities to achieve linear space and quality guarantees.
result FREDE provides column-covariance approximation guarantees that are nearly as good as SVD, even with limited node similarities.
Generative Adversarial Nets (GANs) and Variational Auto-Encoders (VAEs) provide impressive image generations from Gaussian white noise, but the underlying mathematics are not well understood. We compute deep convolutional network generators by inverting a fixed embedding operator. Therefore, they do not require to be o…
When analyzing weighted networks using spectral embedding, a judicious transformation of the edge weights may produce better results. To formalize this idea, we consider the asymptotic behavior of spectral embedding for different edge-weight representations, under a generic low rank model. We measure the quality of dif…
Link prediction is an important and frequently studied task that contributes to an understanding of the structure of knowledge graphs (KGs) in statistical relational learning. Inspired by the success of graph convolutional networks (GCN) in modeling graph data, we propose a unified GCN framework, named TransGCN, to add…
ANT learns sparse embeddings for large vocabularies efficiently.
problem Lack of scalable methods for embedding large vocabularies in neural networks.
method Anchor & Transform (ANT) algorithm that learns a small set of anchor embeddings and a sparse transformation matrix.
result ANT achieves stronger performance with fewer parameters (up to 40x compression) compared to existing methods.
Predictive Sparse Manifold Transform learns dynamic video sequences.
problem Learning and predicting natural dynamics in video sequences.
method Two-layer framework: sparse coding and manifold learning.
result PSMT with dynamic embedding space outperforms static baselines in future frame prediction.
Improved graph embedding through refined linear transformation and community recovery.
problem Identifying meaningful latent communities in graph data.
method Refined graph encoder embedding via linear transformation, self-training, and latent community recovery.
result Improved vertex embedding and better decision boundaries for vertex classification.
Transformers learn topic structure through embedding and attention mechanisms.
problem Understanding how transformers capture semantic structure in text.
method Combination of mathematical analysis and experiments on Wikipedia and synthetic data.
result Embedding and attention layers encode topic structure in transformers.
Transformers show strengths and weaknesses in complexity analysis.
problem Understanding the strengths and limitations of attention layers in transformers.
method Analysis of representation power through complexity parameters and task-specific constructions.
result Transformers can solve sparse averaging tasks with logarithmic complexity, but triple detection tasks require linear complexity.
We study the Penrose transform for the `quaternionic objects' whose twistor spaces are complex manifolds endowed with locally complete families of embedded Riemann spheres with positive normal bundles.
Graphs from van der Corput sequence embed into Chamanara surface.
problem Embedding graphs from van der Corput sequence into surfaces.
method Constructed 4-regular graphs from van der Corput sequence and Kronecker sequence, embedded into torus and Chamanara surface. result Graphs from van der Corput sequence embed into Chamanara surface with one edge removal.
Maps asymptotically embed conic transforms from circle bundles.
problem Embedding conic transforms from circle bundles.
method Asymptotic embeddings using equivariant Szegő projectors.
result Maps embed conic transforms from circle bundles.
Study geodesic properties of time series data using Wasserstein metric.
problem Modeling nonlinear time series with transport-based metrics.
method Generalized Wasserstein metric and signed cumulative distribution transforms.
result Geodesic properties provide added interpretability and robustness in time series classifiers.
EmDT generates synthetic fraud data to improve detection accuracy.
problem Imbalanced datasets in fraud detection lead to poor performance on rare fraudulent transactions.
method EmDT uses UMAP clustering to identify fraudulent patterns and a Transformer denoising network to generate synthetic data.
result EmDT significantly improves classification performance compared to existing methods.
AI-enhanced product embeddings boost demand analysis accuracy.
problem Traditional demand analysis struggles with nuanced product attributes.
method Combining text, images, and tabular data with transformer embeddings for causal inference.
result AI-enhanced embeddings improve sales rank and price predictions.
Tautness of submanifolds in spheres is preserved under Lie sphere transformations.
problem Invariance of tautness under Lie sphere transformations.
method Extending tautness definition to Legendre submanifolds and using Lie sphere transformations to show invariance.
result Tautness is invariant under Lie sphere transformations.
This paper introduces a new task to better understand Transformers in quantitative contexts.
problem Understanding Transformers in high-stakes quantitative and scientific applications.
method Introduces a novel contextual counting task and analyzes it with causal and non-causal Transformer architectures.
result Causal attention is better suited for the contextual counting task, and no positional embeddings lead to the best accuracy.
Transformers encode latent distributions in text, improving performance in out-of-distribution cases.
problem What should embeddings from language models represent?
method Connecting autoregressive prediction to sufficient statistics, identifying three settings.
result Transformers encode latent generating distributions, improving performance.
This work improves KG embeddings by integrating hyperbolic and attention mechanisms.
problem Preserving hierarchical and logical patterns in KGs with low-dimensional embeddings.
method Combines hyperbolic reflections/rotations with attention mechanisms to capture complex relational patterns.
result Improves MRR by up to 6.1% on standard benchmarks and new state-of-the-art results in high dimensions.
This paper clarifies VAE's property through geometric and information-theoretic interpretations.
problem The transparency of VAE model is an underlying issue.
method Quantitative understanding of VAE through differential geometry and information theory.
result VAE can be mapped to an implicit isometric embedding with a scale factor derived from the posterior parameter.
DEN learns diverse tasks to generalize to unseen tasks.
problem Generalization from a diverse set of classification tasks with limited data.
method Three-block architecture: covariate transformation, distribution embedding, and classification.
result DEN outperforms existing methods in various synthetic and real tasks.
BERTopic improves financial text analysis with FinTextSim's contextual embeddings.
problem Analyzing financial text data for insights and predictions.
method Integrates BERTopic with FinTextSim for topic modeling and clustering.
result BERTopic performs better with FinTextSim's embeddings, improving topic clarity and reducing misclassification.
Sequence feature embedding is a challenging task due to the unstructuredness of sequence, i.e., arbitrary strings of arbitrary length. Existing methods are efficient in extracting short-term dependencies but typically suffer from computation issues for the long-term. Sequence Graph Transform (SGT), a feature embedding …
RR-GCN uses random transformations instead of learned weights for node embeddings.
problem Learning node embeddings in KGs.
method Random Relational Graph Convolutional Network (RR-GCN) with untrained parameters.
result RR-GCN can compete with fully trained R-GCNs in node classification and link prediction.
Transform learning improves K-means clustering for document analysis.
problem Improving K-means clustering for document analysis.
method Embedding K-means clustering loss into transform learning framework and solving jointly using ADMM.
result Improves over state-of-the-art in document clustering.
The Joint Optimization of Fidelity and Commensurability (JOFC) manifold matching methodology embeds an omnibus dissimilarity matrix consisting of multiple dissimilarities on the same set of objects. One approach to this embedding optimizes the preservation of fidelity to each individual dissimilarity matrix together wi…
The paper extends optimal transport for linear separability of sheared distributions in supervised learning.
problem Learning on the space of probability measures using shifts and scalings.
method Embedding probability measures into L2 spaces using optimal transport, then applying regular machine learning techniques. result Sheared distributions can be linearly separated under certain conditions, with bounds on transformations.
The kernel embedding algorithm is an important component for adapting kernel methods to large datasets. Since the algorithm consumes a major computation cost in the testing phase, we propose a novel teacher-learner framework of learning computation-efficient kernel embeddings from specific data. In the framework, the h…
A new quantization strategy reduces Transformer model size and inference time.
problem Heavy computation load and memory overhead in Transformer models for mobile devices.
method Mixed precision quantization with varying bits per word in embedding blocks.
result 11.8x smaller model size and 3.5x speed up for on-device NMT.
Word embeddings are commonly obtained as optimizers of a criterion function f of a text corpus, but assessed on word-task performance using a different evaluation function g of the test data. We contend that a possible source of disparity in performance on tasks is the incompatibility between classes of transformat…
In this paper, we generalize the famous Hasimoto's transformation by showing that the dynamics of a closed unidimensional vortex filament embedded in a three-dimensional manifold of constant curvature gives rise under Hasimoto's transformation to the non-linear Schrodinger equation. We also give a natural interpretatio…
Although nonstationary data are more common in the real world, most existing causal discovery methods do not take nonstationarity into consideration. In this letter, we propose a kernel embedding-based approach, ENCI, for nonstationary causal model inference where data are collected from multiple domains with varying d…
CoT enhances transformer accuracy on serial tasks by enabling serial computation.
problem Improving accuracy of large language models on inherently serial problems.
method Integrating a chain of thought (CoT) into decoder-only transformers to enable serial computation.
result Constant-depth transformers with CoT can solve problems in AC^0, surpassing TC^0 without CoT.
A fast graph embedding method for large graphs.
problem Efficiently embedding large graphs for various applications.
method One-hot graph encoder embedding with linear complexity.
result Graph encoder embedding is approximately normally distributed and converges to its mean.
Although modern recommendation systems can exploit the structure in users' item feedback, most are powerless in the face of new users who provide no structure for them to exploit. In this paper we introduce ImplicitCE, an algorithm for recommending items to new users during their sign-up flow. ImplicitCE works by trans…
This research studies affine invariance in continuous-domain convolutional neural networks.
problem Recognizing patterns and features under affine transformations in continuous domains.
method Introduces a new criterion for assessing affine invariance, embeds images into the affine Lie group, and analyzes convolution over this group.
result Extends the scope of geometrical transformations that deep-learning pipelines can handle.
MPP trains a transformer to predict multiple physical systems, improving accuracy across various tasks.
problem Training models for specific physical systems is inefficient and requires fine-tuning.
method MPP trains a shared transformer on multiple heterogeneous physical systems, projecting fields into a shared embedding space.
result A single MPP-pretrained transformer outperforms task-specific models on all pretraining sub-tasks and downstream tasks.