arXiv research

Gradient descent, or negative gradient flow, is a standard technique in optimization to find minima of functions. Many implementations of gradient descent rely on discretized versions, i.e., moving in the gradient direction for a set step size, recomputing the gradient, and continuing. In this paper, we present an appr…

Gradient-trained embeddings select important tokens in text.

Gradient estimates for special harmonic functions on manifolds.

Motivated by manifold learning techniques, we give an explicit lower bound for how far a smoothly embedded compact submanifold in ${\mathbb R}^N$ can move in a normal direction and remain an embedding. In addition, given a penalty function $P : \text{Emb}(M,\mathbb{R}^N) \rightarrow \mathbb{R}$ on the space of embeddi…

Transformers learn to recall with non-orthogonal embeddings in realistic settings.

The paper studies how adding an ℓ2 penalty affects network embeddings.

Existence of balanced embedding proved for complex manifold into infinite-dimensional space.

Boosted embeddings improve time series forecasting models.

In this paper, we propose a scalable algorithm for spectral embedding. The latter is a standard tool for graph clustering. However, its computational bottleneck is the eigendecomposition of the graph Laplacian matrix, which prevents its application to large-scale graphs. Our contribution consists of reformulating spect…

The paper explores how gradient descent trains associative memories, revealing oscillations and convergence issues.

Corrected CBOW performs similarly to Skip-gram.

Ask-n-Learn uses gradient embeddings for active learning in image classification.

Embeds sparsity in deep neural networks, allowing exact zero parameters.

New algorithm tackles dynamic query routing to multiple embedding models.

Optimization with inequality constraints using embedded gradient vector field method

We unify subsampling methods for network embeddings and prove their asymptotic distribution.

STRATA generates code adversarial examples efficiently without gradients.

Enhances graph embeddings by preserving graph topology.

SpecNet2 improves spectral embedding without orthogonalization, achieving better performance and efficiency.

New method learns state embeddings from demonstrations for improved reinforcement learning.

Optimizes shapes of curves using Möbius energy gradients.

Recently, some works have suggested methods to combine variational probabilistic inference with Monte Carlo sampling. One promising approach is via local optimal transport. In this approach, a gradient steepest descent method based on local optimal transport principles is formulated to transform deterministically point…

A hyperbolic space has been shown to be more capable of modeling complex networks than a Euclidean space. This paper proposes an explicit update rule along geodesics in a hyperbolic space. The convergence of our algorithm is theoretically guaranteed, and the convergence rate is better than the conventional Euclidean gr…

Embedded ensembles improve neural network performance efficiently.

This paper presents an end-to-end text-independent speaker verification framework by jointly considering the speaker embedding (SE) network and automatic speech recognition (ASR) network. The SE network learns to output an embedding vector which distinguishes the speaker characteristics of the input utterance, while th…

This paper tackles variance issues in GNN training by proposing a method to reduce both embedding and gradient variances.

Many applications today, such as NLP, network analysis, and code analysis, rely on semantically embedding objects into low-dimensional fixed-length vectors. Such embeddings naturally provide a way to perform useful downstream tasks, such as identifying relations among objects or predicting objects for a given context, …

Mathematical analysis of SNE and t-SNE for dimension reduction.

Gradient rollback explains neural models by identifying influential training examples.

Gradient-based meta-learning techniques are both widely applicable and proficient at solving challenging few-shot learning and fast adaptation problems. However, they have practical difficulties when operating on high-dimensional parameter spaces in extreme low-data regimes. We show that it is possible to bypass these …

Topolow embeds dissimilarity data into Euclidean space robustly against non-metricity and sparsity.

Parametric t-SNE improves generalization for streaming data.

In this work, a novel sequential Monte Carlo filter is introduced which aims at efficient sampling of high-dimensional state spaces with a limited number of particles. Particles are pushed forward from the prior to the posterior density using a sequence of mappings that minimizes the Kullback-Leibler divergence between…

We develop computationally efficient Riemannian manifolds for graph embeddings.

Innovative method solves nonconvex optimization on manifolds.

Gradient flow expands curves to round shapes.

This paper tackles efficient optimization for nonlinear embeddings in similarity learning.

Stochastic neighbor embedding (SNE) and related nonlinear manifold learning algorithms achieve high-quality low-dimensional representations of similarity data, but are notoriously slow to train. We propose a generic formulation of embedding algorithms that includes SNE and other existing algorithms, and study their rel…

We describe the $ν$-lines of curvature of an embedding of the double torus into $\mathbb R^4$, defined as the link of the real part of the Milnor fibration of a polynomial, where $ν$ is its gradient. Through this analysis, we present a complete description of the foliation of lines of curvature of the embedding, define…

We present a novel event embedding algorithm for crime data that can jointly capture time, location, and the complex free-text component of each event. The embedding is achieved by regularized Restricted Boltzmann Machines (RBMs), and we introduce a new way to regularize by imposing a $\ell_1$ penalty on the conditiona…

New method neutralizes gender bias in word embeddings without losing semantic information.

New DP optimization methods for sparse gradients, improving on existing algorithms.

In deep neural nets, lower level embedding layers account for a large portion of the total number of parameters. Tikhonov regularization, graph-based regularization, and hard parameter sharing are approaches that introduce explicit biases into training in a hope to reduce statistical complexity. Alternatively, we propo…

Conventional embedding methods directly associate each symbol with a continuous embedding vector, which is equivalent to applying a linear transformation based on a "one-hot" encoding of the discrete symbols. Despite its simplicity, such approach yields the number of parameters that grows linearly with the vocabulary s…

The paper proves a reverse isoperimetric inequality and applies it to analyze surface flows.

This paper improves topic modeling by embedding words and topics together.

We characterize those spacetimes which admit a isometric (or conformal) embedding in some Lorentz-Minkowski space L^N. In particular, any globally hyperbolic spacetime can be isometrically embedded in L^N. This is proven by a result of its own interest: the construction of a smooth time function whose gradient is bound…

Embedding methods such as word embedding have become pillars for many applications containing discrete structures. Conventional embedding methods directly associate each symbol with a continuous embedding vector, which is equivalent to applying linear transformation based on "one-hot" encoding of the discrete symbols. …