arXiv research

SGNs use Hamiltonian mechanics for invertible deep generative modeling.

Stochastic SGN method converges faster than SGD for DNNs.

We revisit skip-gram negative sampling (SGNS), one of the most popular neural-network based approaches to learning distributed word representation. We first point out the ambiguity issue undermining the SGNS model, in the sense that the word vectors can be entirely distorted without changing the objective value. To res…

We study the following $1$-Yamabe equation on a connected finite graph $Δ_1u+g\mathrm{Sgn}(u)=h|u|^{α-1}\mathrm{Sgn}(u),$ where $Δ_1$ is the discrete $1$-Laplacian, $α>1$ and $g, h>0$ are known. We show that the above $1$-Yamabe equation always has a nontrivial solution $u\geq0$, $u\neq0$.

We simplify word embeddings by removing sigmoid in SGNS, revealing connections to hyperbolic spaces.

The Global Vectors for word representation (GloVe), introduced by Jeffrey Pennington et al. is reported to be an efficient and effective method for learning vector representations of words. State-of-the-art performance is also provided by skip-gram with negative-sampling (SGNS) implemented in the word2vec tool. In this…

New method improves generalization in deep learning models.

This work shows dimension regularization can replace skip-gram negative sampling for graph embeddings, improving efficiency and performance.

New perspective on SGD reveals short-range memory effects in deep learning.

In this note, we investigate the well-known Yau rigidity theorem for minimal submanifolds in spheres. Using the parameter method of Yau and the DDVV inequality verified by Lu, Ge and Tang, we prove that if $M$ is an $n$-dimensional oriented compact minimal submanifold in the unit sphere $S^{n+p}(1)$, and if $K_{M}\geq\…

Let $M$ be an $n(\geq 4)$-dimensional compact submanifold in the simply connected space form $F^{n+p}(c)$ with constant curvature $c\geq 0$, where $H$ is the mean curvature of $M$. We verify that if the scalar curvature of $M$ satisfies $R>n(n-2)(c+H^2)$, and if $Ric_M\geq (n-2-\frac{2σ_n}{2n-σ_n})(c+H^2)$, then $M$ is…

Unified framework for word embedding models using noise examples.

What enables Stochastic Gradient Descent (SGD) to achieve better generalization than Gradient Descent (GD) in Neural Network training? This question has attracted much attention. In this paper, we study the distribution of the Stochastic Gradient Noise (SGN) vectors during the training. We observe that for batch sizes …

Note on subgaussian bounds for sign-quantized linear maps.

This paper takes a step towards theoretical analysis of the relationship between word embeddings and context embeddings in models such as word2vec. We start from basic probabilistic assumptions on the nature of word vectors, context vectors, and text generation. These assumptions are well supported either empirically o…

We present a comprehensive study of utility function of the minority game in its efficient regime. We develop an effective description of state of the game. For the payoff function $g(x)=\sgn (x)$ we explicitly represent the game as the Markov process and prove the finitness of number of states. We also demonstrate bou…

EGN optimizes deep neural networks with exact Gauss-Newton for large-scale problems.

Previous studies indicate that nonlinear properties of Gaussian time series with long-range correlations, $u_i$, can be detected and quantified by studying the correlations in the magnitude series $|u_i|$, i.e., the ``volatility''. However, the origin for this empirical observation still remains unclear, and the exact …

Self-training improves weak classifiers in mixture models.

Word embeddings, i.e., low-dimensional vector representations such as GloVe and SGNS, encode word "meaning" in the sense that distances between words' vectors correspond to their semantic proximity. This enables transfer learning of semantics for a variety of natural language processing tasks. Word embeddings are typic…