Estimates kernel eigenvalues for compositional dot-product kernels.
problem Improving estimates for kernel eigenvalues.
method Eigenvalue decay estimates of integral operators associated with dot-product kernels.
result Improved estimates for kernel volumes in reproducing kernel Hilbert spaces.
Approximating non-linear kernels using feature maps has gained a lot of interest in recent years due to applications in reducing training and testing times of SVM classifiers and other kernel based learning algorithms. We extend this line of work and present low distortion embeddings for dot product kernels into linear…
Improves efficiency of random feature approximations for dot product kernels.
problem Efficiency of random feature approximations for dot product kernels.
method Generalization of existing random feature approximations using complex-valued random features, theoretical analysis of variances, data-driven optimization approach.
result Complex-valued random features can significantly reduce the variances of approximations.
Transformers improve with Fourier integral attentions.
problem Inefficiency of dot-product attention in capturing feature dependencies.
method Interpreted attention as kernel regression, proposed FourierFormer with generalized Fourier integral kernels.
result FourierFormer achieves better accuracy and reduces redundancy.
The paper analyzes learning curves for kernel ridge regression with dot-product kernels.
problem Understanding the learning curves for different scaling regimes of data and model.
method Precise formulas for mean test error, bias, and variance in the mo∞ with m/dr constant regime. result A peak in the learning curve at m≈dr/r! for any integer r. In this paper we study the concentration properties for the eigenvalues of kernel matrices, which are central objects in a wide range of kernel methods and, more recently, in network analysis. We present a set of concentration inequalities tailored for each individual eigenvalue of the kernel matrix with respect to its…
In this paper, we propose the distributed tree kernels (DTK) as a novel method to reduce time and space complexity of tree kernels. Using a linear complexity algorithm to compute vectors for trees, we embed feature spaces of tree fragments in low-dimensional spaces where the kernel computation is directly done with dot…
Random features enhance control of complex systems.
problem Flexible nonlinear models for control-affine systems.
method Random features approximations for control-affine structure.
result Methods formalized and shown to relate to ADP and AD kernels.
Kernel methods form a powerful, versatile, and theoretically-grounded unifying framework to solve nonlinear problems in signal processing and machine learning. The standard approach relies on the kernel trick to perform pairwise evaluations of a kernel function, which leads to scalability issues for large datasets due …
This work analyzes when contrastive models are close to PCA or kernel methods.
problem Understanding when contrastive models are equivalent to kernel methods or PCA.
method Analyzing the training dynamics of two-layer contrastive models with non-linear activation.
result Wide contrastive models with cosine similarity based losses are close to PCA.
A new model captures complex event data using attention and Fourier kernels.
problem Capturing complex non-linear temporal dependencies in discrete event data.
method Integrates attention mechanism into point processes' conditional intensity function and uses Fourier kernel embedding.
result Established theoretical properties and demonstrated competitive performance.
Revisits neural collaborative filtering vs. matrix factorization, showing dot product superiority.
problem Comparing neural collaborative filtering to matrix factorization in recommendation systems.
method Revisited experiments using MLPs as similarity functions, comparing dot product to MLP outputs.
result Simple dot product outperforms MLP-based learned similarities in practical settings.
We present an approach to model time series data from resting state fMRI for autism spectrum disorder (ASD) severity classification. We propose to adopt kernel machines and employ graph kernels that define a kernel dot product between two graphs. This enables us to take advantage of spatio-temporal information to captu…
We analyze how kernel machines and neural networks learn different frequency modes of the target function as data size increases.
problem Understanding how kernel machines and neural networks learn different frequency modes of the target function as data size increases.
method Theoretical methods from Gaussian processes and statistical physics, combined with simulations on synthetic data and MNIST dataset.
result Kernel machines and neural networks fit successively higher spectral modes of the target function as the size of the training set grows.
Traditionally, multi-layer neural networks use dot product between the output vector of previous layer and the incoming weight vector as the input to activation function. The result of dot product is unbounded, thus increases the risk of large variance. Large variance of neuron makes the model sensitive to the change o…
Skyformer uses Gaussian kernel and Nyström method to speed up self-attention in transformers.
problem High computational cost of self-attention in transformers.
method Replaces softmax with Gaussian kernel and applies Nyström method for matrix approximation.
result Skyformer achieves comparable or better performance with fewer computation resources.
IDPGs extend RDPGs with a Poisson process for random latent positions.
problem Modeling randomness in latent positions for graph structure.
method Introduce IDPGs using Poisson point processes on latent Euclidean space.
result Continuous analogues of adjacency matrices link latent structure to observed graphs.
Study analyzes learnability of RKHS under L∞ norm for kernel methods.
problem Understand performance of kernel methods and random feature models.
method Relate L∞ learnability to kernel spectrum decay and establish sample complexity bounds.
result Conditions for efficient L∞ learning of RKHS identified.
Many interesting machine learning problems are best posed by considering instances that are distributions, or sample sets drawn from distributions. Previous work devoted to machine learning tasks with distributional inputs has done so through pairwise kernel evaluations between pdfs (or sample sets). While such an appr…
Extends random dot product graph model to handle multiple graphs.
problem Modeling and analyzing multiple graphs with shared nodes.
method Jointly embed adjacency matrices into a latent space.
result Node representations converge to latent positions with Gaussian error.
Transformers are explained as infinite-dimensional kernel machines.
problem Understanding the mechanics of Transformers in AI.
method Characterized Transformers' attention mechanism as a kernel learning method on Banach spaces.
result Transformer's kernel has infinite feature dimension and can learn any binary non-Mercer reproducing kernel Banach space pair.
Linear time algorithm for random walk kernels on sparse graphs.
problem Efficient computation of general random walk kernels for large graphs.
method Sample dependent random walks to compute graph embeddings without direct graph product.
result Up to 27x faster and scalable to 128x larger graphs than previous methods.
Convex optimization method infers latent structure in random dot product graphs.
problem Inferring latent probability matrix of random dot product graphs.
method Conic programming with nuclear norm regularization.
result Asymptotic consistency of probability estimates and recovery of latent structure.
This paper defines the notion of class discrepancy for families of functions. It shows that low discrepancy classes admit small offline and streaming coresets. We provide general techniques for bounding the class discrepancy of machine learning problems. As corollaries of the general technique we bound the discrepancy …
Despite their successes, what makes kernel methods difficult to use in many large scale problems is the fact that storing and computing the decision function is typically expensive, especially at prediction time. In this paper, we overcome this difficulty by proposing Fastfood, an approximation that accelerates such co…
New algorithms improve community detection and parameter estimation for PABM.
problem Improving community detection and parameter estimation for PABM.
method Connecting PABM to GRDPG, constructing new algorithms, and deriving asymptotic properties.
result Absolute number of community detection errors tends to zero as graph vertices increase.
Vectors of data are at the heart of machine learning and data mining. Recently, vector quantization methods have shown great promise in reducing both the time and space costs of operating on vectors. We introduce a vector quantization algorithm that can compress vectors over 12x faster than existing techniques while al…
New clustering method recovers hidden tree structure from data.
problem Recovering hidden hierarchical structure in data.
method Maximum average dot product for merging clusters in hierarchical clustering.
result The algorithm produces a tree that accurately represents the underlying generative hierarchical structure.
In this work we show that, using the eigen-decomposition of the adjacency matrix, we can consistently estimate latent positions for random dot product graphs provided the latent positions are i.i.d. from some distribution. If class labels are observed for a number of vertices tending to infinity, then we show that the …
New method uses manifold learning to infer latent positions of 1D submanifolds in random dot product graphs.
problem Inference on latent positions of unknown 1D submanifolds in RDPGs.
method Apply Isomap for manifold learning to estimate arc lengths on the unknown submanifold.
result Test statistics based on Isomap converge to known submanifold power as auxiliary vertices increase.
The random dot product graph (RDPG) is an independent-edge random graph that is analytically tractable and, simultaneously, either encompasses or can successfully approximate a wide range of random graphs, from relatively simple stochastic block models to complex latent position graphs. In this survey paper, we describ…
We prove a central limit theorem for the components of the largest eigenvectors of the adjacency matrix of a finite-dimensional random dot product graph whose true latent positions are unknown. In particular, we follow the methodology outlined in \citet{sussman2012universally} to construct consistent estimates for the …
At the core of any inference procedure in deep neural networks are dot product operations, which are the component that require the highest computational resources. A common approach to reduce the cost of inference is to reduce its memory complexity by lowering the entropy of the weight matrices of the neural network, …
The paper examines how well node similarities are preserved by random projections in graph embeddings.
problem The preservation of node similarities under random projections in graph embeddings.
method Investigation of dot product and cosine similarity preservation by random projections over graph matrix rows.
result Random projections produce unreliable embeddings for dot product, especially for high-degree nodes.
Paper explores embedding methods for detecting pseudo-cliques in random graphs, showing limitations and potential.
problem Detecting planted pseudo-cliques in random dot product graphs.
method Adjacency Spectral Embedding (ASE) and Graph Encoder Embedding (GEE).
result These methods can localize pseudo-cliques with additional clean network data, but not without it.
Power of network tests degrades when vertices are misaligned.
problem Power loss in network hypothesis testing due to vertex shuffling.
method Theoretical analysis and simulations of Frobenius norm differences in random dot product and stochastic block models.
result Shuffling vertices can significantly reduce the power of network tests.
Transformers become faster by linearizing self-attention.
problem Quadratic complexity of transformers makes them slow for long sequences.
method Expressed self-attention as a linear dot-product and used matrix product associativity to reduce complexity.
result Linear transformers are up to 4000x faster on long sequences.
GraphMoE generates random graphs using neural networks and graphlets.
problem Learning generative models for random graphs.
method GraphMoE uses a neural network trained with graphlets and subgraph counts to match the distribution of random graphs.
result GraphMoE can generate graphs that mimic various real-world datasets and fool graph classifiers.
Elliptical Attention improves transformer performance by focusing on contextually relevant features.
problem Transformer models suffer from representation collapse and are vulnerable to contaminated samples.
method Uses Mahalanobis distance to define hyper-ellipsoidal neighborhoods for attention weights.
result Elliptical Attention reduces representation collapse and enhances model robustness.
Spectral embedding is a procedure which can be used to obtain vector representations of the nodes of a graph. This paper proposes a generalisation of the latent position network model known as the random dot product graph, to allow interpretation of those vector representations as latent position estimates. The general…
SDPA is shown to be an optimal transport problem in deep learning.
problem The mathematical foundation and optimization perspective of SDPA.
method SDPA is shown to be the exact solution to a degenerate, one-sided Entropic Optimal Transport (EOT) problem.
result The SDPA mechanism is a principled mechanism where the forward pass performs optimal inference and the backward pass implements a rational, manifold-aware learning update.
SGPA calibrates transformer uncertainty for safety-critical tasks.
problem Uncertainty estimation in transformer models for safety-critical domains.
method Bayesian inference in transformer's output space using sparse Gaussian processes.
result SGPA-based Transformers improve in-distribution calibration and out-of-distribution robustness.
In statistical relational learning, the link prediction problem is key to automatically understand the structure of large knowledge bases. As in previous studies, we propose to solve this problem through latent factorization. However, here we make use of complex valued embeddings. The composition of complex embeddings …
This paper improves GNNs' generalization by adding a Low-Rank Global Attention module.
problem Improving the generalization power of Graph Neural Networks (GNNs).
method Incorporating a Low-Rank Global Attention (LRGA) module into GNNs.
result Augmenting GNNs with LRGA aligns them with a powerful graph isomorphism test, 2-Folklore Weisfeiler-Lehman (2-FWL).
NNLMs optimize poorly for word probabilities due to embedding space structure.
problem NNLMs assign suboptimal probabilities to some words.
method Analyzed the inductive bias of NNLMs and the structure of word embeddings.
result Words on the convex hull have bounded probability, affecting others.
The paper extends RDPG model to handle weighted graphs, enabling better analysis of network data.
problem Modeling networks with weighted edges to capture heterogeneous weight distributions.
method Proposes a nonparametric W-RDPG model with latent positions and moment-generating functions.
result Establishes statistical guarantees for estimating nodal latent positions and sampling graphs.
OmniMatch algorithm perfectly matches graphs without edge correlation.
problem Graph matching in the absence of edge correlation.
method OmniMatch algorithm for seeded multiple graph matching.
result OmniMatch aligns O(sα) unseeded vertices across multiple networks efficiently and perfectly. The vanishing gradient problem was a major obstacle for the success of deep learning. In recent years it was gradually alleviated through multiple different techniques. However the problem was not really overcome in a fundamental way, since it is inherent to neural networks with activation functions based on dot produc…