arXiv research

Approximating non-linear kernels using feature maps has gained a lot of interest in recent years due to applications in reducing training and testing times of SVM classifiers and other kernel based learning algorithms. We extend this line of work and present low distortion embeddings for dot product kernels into linear…

We present eigenvalue decay estimates of integral operators associated with compositional dot-product kernels. The estimates improve on previous ones established for power series kernels on spheres. This allows us to obtain the volumes of balls in the corresponding reproducing kernel Hilbert spaces. We discuss the cons…

Improves efficiency of random feature approximations for dot product kernels.

Transformers improve with Fourier integral attentions.

The paper analyzes learning curves for kernel ridge regression with dot-product kernels.

In this paper we study the concentration properties for the eigenvalues of kernel matrices, which are central objects in a wide range of kernel methods and, more recently, in network analysis. We present a set of concentration inequalities tailored for each individual eigenvalue of the kernel matrix with respect to its…

Random features enhance control of complex systems.

In this paper, we propose the distributed tree kernels (DTK) as a novel method to reduce time and space complexity of tree kernels. Using a linear complexity algorithm to compute vectors for trees, we embed feature spaces of tree fragments in low-dimensional spaces where the kernel computation is directly done with dot…

Kernel methods form a powerful, versatile, and theoretically-grounded unifying framework to solve nonlinear problems in signal processing and machine learning. The standard approach relies on the kernel trick to perform pairwise evaluations of a kernel function, which leads to scalability issues for large datasets due …

Revisits neural collaborative filtering vs. matrix factorization, showing dot product superiority.

This work analyzes when contrastive models are close to PCA or kernel methods.

We are interested in approximation of a multivariate function $f(x_1,\dots,x_d)$ by linear combinations of products $u^1(x_1)\cdots u^d(x_d)$ of univariate functions $u^i(x_i)$, $i=1,\dots,d$. In the case $d=2$ it is a classical problem of bilinear approximation. In the case of approximation in the $L_2$ space the bili…

Traditionally, multi-layer neural networks use dot product between the output vector of previous layer and the incoming weight vector as the input to activation function. The result of dot product is unbounded, thus increases the risk of large variance. Large variance of neuron makes the model sensitive to the change o…

We present a novel attention-based model for discrete event data to capture complex non-linear temporal dependence structures. We borrow the idea from the attention mechanism and incorporate it into the point processes' conditional intensity function. We further introduce a novel score function using Fourier kernel emb…

IDPGs extend RDPGs with a Poisson process for random latent positions.

We present an approach to model time series data from resting state fMRI for autism spectrum disorder (ASD) severity classification. We propose to adopt kernel machines and employ graph kernels that define a kernel dot product between two graphs. This enables us to take advantage of spatio-temporal information to captu…

We derive analytical expressions for the generalization performance of kernel regression as a function of the number of training samples using theoretical methods from Gaussian processes and statistical physics. Our expressions apply to wide neural networks due to an equivalence between training them and kernel regress…

Formula derived for spherical growth series of specific groups.

Extends random dot product graph model to handle multiple graphs.

Linear time algorithm for random walk kernels on sparse graphs.

Convex optimization method infers latent structure in random dot product graphs.

Vectors of data are at the heart of machine learning and data mining. Recently, vector quantization methods have shown great promise in reducing both the time and space costs of operating on vectors. We introduce a vector quantization algorithm that can compress vectors over 12x faster than existing techniques while al…

Skyformer uses Gaussian kernel and Nyström method to speed up self-attention in transformers.

Let $G$ be a countable group that splits as a free product of groups of the form $G=G_1\ast\dots\ast G_k\ast F_N$, where $F_N$ is a finitely generated free group. We identify the closure of the outer space $P\mathcal{O}(G,\{G_1,\dots,G_k\})$ for the axes topology with the space of projective minimal, \emph{very small} …

Study analyzes learnability of RKHS under L∞ norm for kernel methods.

New algorithms improve community detection and parameter estimation for PABM.

New clustering method recovers hidden tree structure from data.

Study on the topology of tensorial bodies, showing they are homeomorphic to a product space.

In this work we show that, using the eigen-decomposition of the adjacency matrix, we can consistently estimate latent positions for random dot product graphs provided the latent positions are i.i.d. from some distribution. If class labels are observed for a number of vertices tending to infinity, then we show that the …

Many interesting machine learning problems are best posed by considering instances that are distributions, or sample sets drawn from distributions. Previous work devoted to machine learning tasks with distributional inputs has done so through pairwise kernel evaluations between pdfs (or sample sets). While such an appr…

New method uses manifold learning to infer latent positions of 1D submanifolds in random dot product graphs.

The random dot product graph (RDPG) is an independent-edge random graph that is analytically tractable and, simultaneously, either encompasses or can successfully approximate a wide range of random graphs, from relatively simple stochastic block models to complex latent position graphs. In this survey paper, we describ…

Transformers become faster by linearizing self-attention.

We prove a central limit theorem for the components of the largest eigenvectors of the adjacency matrix of a finite-dimensional random dot product graph whose true latent positions are unknown. In particular, we follow the methodology outlined in \citet{sussman2012universally} to construct consistent estimates for the …

At the core of any inference procedure in deep neural networks are dot product operations, which are the component that require the highest computational resources. A common approach to reduce the cost of inference is to reduce its memory complexity by lowering the entropy of the weight matrices of the neural network, …

This paper defines the notion of class discrepancy for families of functions. It shows that low discrepancy classes admit small offline and streaming coresets. We provide general techniques for bounding the class discrepancy of machine learning problems. As corollaries of the general technique we bound the discrepancy …

The paper examines how well node similarities are preserved by random projections in graph embeddings.

Study on stability of half-harmonic maps from R to S, proving non-degeneracy and quantitative stability.

Transformers are explained as infinite-dimensional kernel machines.

Paper explores embedding methods for detecting pseudo-cliques in random graphs, showing limitations and potential.

Despite their successes, what makes kernel methods difficult to use in many large scale problems is the fact that storing and computing the decision function is typically expensive, especially at prediction time. In this paper, we overcome this difficulty by proposing Fastfood, an approximation that accelerates such co…

Let $G=G_1\ast\dots\ast G_k\ast F$ be a countable group which splits as a free product, where all groups $G_i$ are freely indecomposable and not isomorphic to $\mathbb{Z}$, and $F$ is a finitely generated free group. If for all $i\in\{1,\dots,k\}$, both $G_i$ and its outer automorphism group $\text{Out}(G_i)$ satisfy t…

Power of network tests degrades when vertices are misaligned.

Elliptical Attention improves transformer performance by focusing on contextually relevant features.

Spectral embedding is a procedure which can be used to obtain vector representations of the nodes of a graph. This paper proposes a generalisation of the latent position network model known as the random dot product graph, to allow interpretation of those vector representations as latent position estimates. The general…

SDPA is shown to be an optimal transport problem in deep learning.

We introduce the $2$-simplicial Transformer, an extension of the Transformer which includes a form of higher-dimensional attention generalising the dot-product attention, and uses this attention to update entity representations with tensor products of value vectors. We show that this architecture is a useful inductive …

In statistical relational learning, the link prediction problem is key to automatically understand the structure of large knowledge bases. As in previous studies, we propose to solve this problem through latent factorization. However, here we make use of complex valued embeddings. The composition of complex embeddings …