arXiv research

Approximating non-linear kernels using feature maps has gained a lot of interest in recent years due to applications in reducing training and testing times of SVM classifiers and other kernel based learning algorithms. We extend this line of work and present low distortion embeddings for dot product kernels into linear…

Revisits neural collaborative filtering vs. matrix factorization, showing dot product superiority.

Traditionally, multi-layer neural networks use dot product between the output vector of previous layer and the incoming weight vector as the input to activation function. The result of dot product is unbounded, thus increases the risk of large variance. Large variance of neuron makes the model sensitive to the change o…

We present eigenvalue decay estimates of integral operators associated with compositional dot-product kernels. The estimates improve on previous ones established for power series kernels on spheres. This allows us to obtain the volumes of balls in the corresponding reproducing kernel Hilbert spaces. We discuss the cons…

IDPGs extend RDPGs with a Poisson process for random latent positions.

Improves efficiency of random feature approximations for dot product kernels.

Formula derived for spherical growth series of specific groups.

Extends random dot product graph model to handle multiple graphs.

Convex optimization method infers latent structure in random dot product graphs.

Vectors of data are at the heart of machine learning and data mining. Recently, vector quantization methods have shown great promise in reducing both the time and space costs of operating on vectors. We introduce a vector quantization algorithm that can compress vectors over 12x faster than existing techniques while al…

Let $G$ be a countable group that splits as a free product of groups of the form $G=G_1\ast\dots\ast G_k\ast F_N$, where $F_N$ is a finitely generated free group. We identify the closure of the outer space $P\mathcal{O}(G,\{G_1,\dots,G_k\})$ for the axes topology with the space of projective minimal, \emph{very small} …

New algorithms improve community detection and parameter estimation for PABM.

Transformers improve with Fourier integral attentions.

New clustering method recovers hidden tree structure from data.

The paper analyzes learning curves for kernel ridge regression with dot-product kernels.

Study on the topology of tensorial bodies, showing they are homeomorphic to a product space.

In this work we show that, using the eigen-decomposition of the adjacency matrix, we can consistently estimate latent positions for random dot product graphs provided the latent positions are i.i.d. from some distribution. If class labels are observed for a number of vertices tending to infinity, then we show that the …

New method uses manifold learning to infer latent positions of 1D submanifolds in random dot product graphs.

The random dot product graph (RDPG) is an independent-edge random graph that is analytically tractable and, simultaneously, either encompasses or can successfully approximate a wide range of random graphs, from relatively simple stochastic block models to complex latent position graphs. In this survey paper, we describ…

We prove a central limit theorem for the components of the largest eigenvectors of the adjacency matrix of a finite-dimensional random dot product graph whose true latent positions are unknown. In particular, we follow the methodology outlined in \citet{sussman2012universally} to construct consistent estimates for the …

At the core of any inference procedure in deep neural networks are dot product operations, which are the component that require the highest computational resources. A common approach to reduce the cost of inference is to reduce its memory complexity by lowering the entropy of the weight matrices of the neural network, …

The paper examines how well node similarities are preserved by random projections in graph embeddings.

Paper explores embedding methods for detecting pseudo-cliques in random graphs, showing limitations and potential.

Let $G=G_1\ast\dots\ast G_k\ast F$ be a countable group which splits as a free product, where all groups $G_i$ are freely indecomposable and not isomorphic to $\mathbb{Z}$, and $F$ is a finitely generated free group. If for all $i\in\{1,\dots,k\}$, both $G_i$ and its outer automorphism group $\text{Out}(G_i)$ satisfy t…

Power of network tests degrades when vertices are misaligned.

Elliptical Attention improves transformer performance by focusing on contextually relevant features.

Spectral embedding is a procedure which can be used to obtain vector representations of the nodes of a graph. This paper proposes a generalisation of the latent position network model known as the random dot product graph, to allow interpretation of those vector representations as latent position estimates. The general…

SDPA is shown to be an optimal transport problem in deep learning.

We introduce the $2$-simplicial Transformer, an extension of the Transformer which includes a form of higher-dimensional attention generalising the dot-product attention, and uses this attention to update entity representations with tensor products of value vectors. We show that this architecture is a useful inductive …

In statistical relational learning, the link prediction problem is key to automatically understand the structure of large knowledge bases. As in previous studies, we propose to solve this problem through latent factorization. However, here we make use of complex valued embeddings. The composition of complex embeddings …

The paper extends RDPG model to handle weighted graphs, enabling better analysis of network data.

We are interested in approximation of a multivariate function $f(x_1,\dots,x_d)$ by linear combinations of products $u^1(x_1)\cdots u^d(x_d)$ of univariate functions $u^i(x_i)$, $i=1,\dots,d$. In the case $d=2$ it is a classical problem of bilinear approximation. In the case of approximation in the $L_2$ space the bili…

NNLMs optimize poorly for word probabilities due to embedding space structure.

OmniMatch algorithm perfectly matches graphs without edge correlation.

Random features enhance control of complex systems.

The study explores special hypersurfaces in Riemannian products, focusing on elliptic Weingarten conditions.

The vanishing gradient problem was a major obstacle for the success of deep learning. In recent years it was gradually alleviated through multiple different techniques. However the problem was not really overcome in a fundamental way, since it is inherent to neural networks with activation functions based on dot produc…

New method recovers graph latent positions under edge differential privacy.

We introduce a new routing algorithm for capsule networks, in which a child capsule is routed to a parent based only on agreement between the parent's state and the child's vote. The new mechanism 1) designs routing via inverted dot-product attention; 2) imposes Layer Normalization as normalization; and 3) replaces seq…

The paper corrects for node degree in spectral clustering using random walk Laplacian.

Online CPD for weighted and directed graphs using RDPG model.

New method embeds dynamic networks with stability for node behavior.

In statistical relational learning, knowledge graph completion deals with automatically understanding the structure of large knowledge graphs---labeled directed graphs---and predicting missing relationships---labeled edges. State-of-the-art embedding models propose different trade-offs between modeling expressiveness, …

Using a coordinate free characterization of hyperplanes intersection, we provide explicitly a set of local generators for a smooth affine distribution given by those smooth vector fields $X\in\mathfrak{X}(U)$ defined eventually on an open subset $U\subseteq M$ of a smooth Riemannian manifold $(M,g)$, that verifies the …

In this paper we study the concentration properties for the eigenvalues of kernel matrices, which are central objects in a wide range of kernel methods and, more recently, in network analysis. We present a set of concentration inequalities tailored for each individual eigenvalue of the kernel matrix with respect to its…

The paper examines deformations of simple dotted graphs made of circles.

Vertex clustering in a stochastic blockmodel graph has wide applicability and has been the subject of extensive research. In thispaper, we provide a short proof that the adjacency spectral embedding can be used to obtain perfect clustering for the stochastic blockmodel and the degree-corrected stochastic blockmodel. We…

If $A$ is an abelian group and $φ$ is an integer, let $A(φ)$ be the subgroup of $A$ consisting of elements $a \in A$ such that $φ\cdot a=0$. We prove that if $D$ is a diagram of a classical link $L$ and $0=φ_0,φ_1,\dots,φ_{n-1}$ are the invariant factors of an adjusted Goeritz matrix of $D$, then the group $\mathcal{D}…