arXiv research

Geometrically transforms word embeddings into a common space for better comparison.

Recent work has demonstrated that embeddings of tree-like graphs in hyperbolic space surpass their Euclidean counterparts in performance by a large margin. Inspired by these results and scale-free structure in the word co-occurrence graph, we present an algorithm for learning word embeddings in hyperbolic space from fr…

Word embeddings generated by neural network methods such as word2vec (W2V) are well known to exhibit seemingly linear behaviour, e.g. the embeddings of analogy "woman is to queen as man is to king" approximately describe a parallelogram. This property is particularly intriguing since the embeddings are not trained to a…

LLMs learn new tasks from unstructured data, but it depends on word co-occurrence and positional information.

We propose a new method for learning word representations using hierarchical regularization in sparse coding inspired by the linguistic study of word meanings. We show an efficient learning algorithm based on stochastic proximal methods that is significantly faster than previous approaches, making it possible to perfor…

Latent topic models have been successfully applied as an unsupervised topic discovery technique in large document collections. With the proliferation of hypertext document collection such as the Internet, there has also been great interest in extending these approaches to hypertext [6, 9]. These approaches typically mo…

Bidirectional attention is shown to be equivalent to a continuous bag of words model with mixture-of-experts.

Real-valued word representations have transformed NLP applications; popular examples are word2vec and GloVe, recognized for their ability to capture linguistic regularities. In this paper, we demonstrate a {\em very simple}, and yet counter-intuitive, postprocessing technique -- eliminate the common mean vector and a f…

Semantic word embeddings represent the meaning of a word via a vector, and are created by diverse methods. Many use nonlinear operations on co-occurrence statistics, and have hand-tuned hyperparameters and reweighting methods. This paper proposes a new generative model, a dynamic version of the log-linear topic model o…

We describe a method for learning word embeddings with data-dependent dimensionality. Our Stochastic Dimensionality Skip-Gram (SD-SG) and Stochastic Dimensionality Continuous Bag-of-Words (SD-CBOW) are nonparametric analogs of Mikolov et al.'s (2013) well-known 'word2vec' models. Vector dimensionality is made dynamic b…

Geospatial analysis lacks methods like the word vector representations and pre-trained networks that significantly boost performance across a wide range of natural language and computer vision tasks. To fill this gap, we introduce Tile2Vec, an unsupervised representation learning algorithm that extends the distribution…

Word embeddings learnt from large corpora have been adopted in various applications in natural language processing and served as the general input representations to learning systems. Recently, a series of post-processing methods have been proposed to boost the performance of word embeddings on similarity comparison an…

Embedding words in a vector space has gained a lot of attention in recent years. While state-of-the-art methods provide efficient computation of word similarities via a low-dimensional matrix embedding, their motivation is often left unclear. In this paper, we argue that word embedding can be naturally viewed as a rank…

In this paper, we present Russian language datasets in the digital humanities domain for the evaluation of word embedding techniques or similar language modeling and feature learning algorithms. The datasets are split into two task types, word intrusion and word analogy, and contain 31362 task units in total. The chara…

The blind application of machine learning runs the risk of amplifying biases present in data. Such a danger is facing us with word embedding, a popular framework to represent text data as vectors which has been used in many machine learning and natural language processing tasks. We show that even word embeddings traine…

Recent progress in applying machine learning for jet physics has been built upon an analogy between calorimeters and images. In this work, we present a novel class of recursive neural networks built instead upon an analogy between QCD and natural languages. In the analogy, four-momenta are like words and the clustering…

Continuous vector representations of words and objects appear to carry surprisingly rich semantic content. In this paper, we advance both the conceptual and theoretical understanding of word embeddings in three ways. First, we ground embeddings in semantic spaces studied in cognitive-psychometric literature and introdu…

New invariants derived from random matrices for words in free groups.

Anosov representations of word hyperbolic groups into higher-rank semisimple Lie groups are representations with finite kernel and discrete image that have strong analogies with convex cocompact representations into rank-one Lie groups. However, the most naive analogy fails: generically, Anosov representations do not a…

The paper finds formulas for word lengths and conjugacy classes in surface groups.

Factor complexity $b_φ(n)$ for a vertex coloring $φ$ of a regular tree is the number of colored $n$-balls up to color-preserving automorphisms. Sturmian colorings are colorings of minimal unbounded factor complexity $b_φ(n) = n+2$. In this article, we prove an induction algorithm for Sturmian colorings using colored ba…

Proposes SNML for selecting word2vec Skip-gram dimensionality.

We develop a family of techniques to align word embeddings which are derived from different source datasets or created using different mechanisms (e.g., GloVe or word2vec). Our methods are simple and have a closed form to optimally rotate, translate, and scale to minimize root mean squared errors or maximize the averag…

In recent years, the Word2Vec model trained with the Negative Sampling loss function has shown state-of-the-art results in a number of machine learning tasks, including language modeling tasks, such as word analogy and word similarity, and in recommendation tasks, through Prod2Vec, an extension that applies to modeling…

Interventional cancer clinical trials are generally too restrictive, and some patients are often excluded on the basis of comorbidity, past or concomitant treatments, or the fact that they are over a certain age. The efficacy and safety of new treatments for patients with these characteristics are, therefore, not defin…

Machine learning algorithms are optimized to model statistical properties of the training data. If the input data reflects stereotypes and biases of the broader society, then the output of the learning algorithm also captures these stereotypes. In this paper, we initiate the study of gender stereotypes in {\em word emb…

We prove that the exponential growth rate of the regular language of penetration sequences is smaller than the growth rate of the regular language of normal form words, if the acceptor of the regular language of normal form words is strongly connected. Moreover, we show that the latter property is satisfied for all irr…

In this paper we introduce flat grafting as a deformation of quadratic differentials on a surface of finite type that is analogous to the grafting map on hyperbolic surfaces. Flat grafting maps are generic in the strata structure and preserve parallel measured foliations. We use flat grafting to construct paths connect…

The aim of this paper is to present a short introduction to supergeometry on pure odd supermanifolds. (Pseudo)differential forms, Cartan calculus (DeRham differential, Lie derivative, "inner" product), metric, inner product, Killing's vector fields, Hodge star operator, integral forms, co-differential and connection on…

Model-based reinforcement learning algorithms make decisions by building and utilizing a model of the environment. However, none of the existing algorithms attempts to infer the dynamics of any state-action pair from known state-action pairs before meeting it for sufficient times. We propose a new model-based method ca…

Let $w$ be a word in the free group on $r$ generators. The expected value of the trace of the word in $r$ independent Haar elements of $\mathrm{O}(n)$ gives a function ${\cal T}r_{w}^{\mathrm{O}}(n)$ of $n$. We show that ${\cal T}r_{w}^{\mathrm{O}}(n)$ has a convergent Laurent expansion at $n=\infty$ involving maps on …

Analyzes word2vec-like models revealing linear subspaces learned during training.

Given a function $f\colon X\to Y$ of metric spaces, its {\it asymptotic dimension} $\asdim(f)$ is the supremum of $\asdim(A)$ such that $A\subset X$ and $\asdim(f(A))=0$. Our main result is \begin{Thm} \label{ThmAInAbstract} $\asdim(X)\leq \asdim(f)+\asdim(Y)$ for any large scale uniform function $f\colon X\to Y$. \end…

Separable Non-negative Matrix Factorization (SNMF) is an important method for topic modeling, where "separable" assumes every topic contains at least one anchor word, defined as a word that has non-zero probability only on that topic. SNMF focuses on the word co-occurrence patterns to reveal topics by two steps: anchor…

This paper continues a geometric study of Harvey's Complex of Curves, whose ultimate goal is to apply the theory of hyperbolic spaces and groups to algorithmic questions for the Mapping Class Group and geometric properties of Kleinian representations. The authors' previous result that the complex is delta-hyperbolic wa…

Most existing word embedding approaches do not distinguish the same words in different contexts, therefore ignoring their contextual meanings. As a result, the learned embeddings of these words are usually a mixture of multiple meanings. In this paper, we acknowledge multiple identities of the same word in different co…

There are certain families of words and word sequences (words in the generators of a two-generator group) that arise frequently in the Teichm{ü}ller theory of hyperbolic three-manifolds and Kleinian and Fuchsian groups and in the discreteness problem for two generator matrix groups. We survey some of the families of su…

Urban2Vec combines street view imagery and POIs for better urban neighborhood embeddings.

Study on representations of four-punctured sphere group in hyperbolic spaces.

Standard methods in deep learning for natural language processing fail to capture the compositional structure of human language that allows for systematic generalization outside of the training distribution. However, human learners readily generalize in this way, e.g. by applying known grammatical rules to novel words.…

We discuss a topological approach to words introduced by the author. Words on an arbitrary alphabet are approximated by Gauss words and then studied up to natural modifications inspired by the Reidemeister moves on knot diagrams. This leads us to a notion of homotopy for words. We introduce several homotopy invariants …

In this work, three lattice-free (LF) discriminative training criteria for purely sequence-trained neural network acoustic models are compared on LVCSR tasks, namely maximum mutual information (MMI), boosted maximum mutual information (bMMI) and state-level minimum Bayes risk (sMBR). We demonstrate that, analogous to L…

If $G$ is a finite group, is a function $f:G\to\mathbb C$ determined by its sums over all cosets of cyclic subgroups of $G$? In other words, is the Radon transform on $G$ injective? This inverse problem is a discrete analogue of asking whether a function on a compact Lie group is determined by its integrals over all ge…

Continuous word representation (aka word embedding) is a basic building block in many neural network-based models used in natural language processing tasks. Although it is widely accepted that words with similar semantics should be close to each other in the embedding space, we find that word embeddings learned in seve…

The abstract explains how word and relation representations capture semantic meaning.

Paper proposes MMD-Sense-Analysis for detecting word sense shifts.

The study introduces Cayley--Abels--Rosendal graphs for Polish groups.

Bayesian algorithm improves word representations using semantic taxonomy.