Word embeddings provide point representations of words containing useful semantic information. We introduce multimodal word distributions formed from Gaussian mixtures, for multiple word meanings, entailment, and rich uncertainty information. To learn these distributions, we propose an energy-based max-margin objective…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,051 papers · 148 categories
Trend · papers per month
48 results for “word distributions”
Approaches KL divergence for learning multi-sense word distributions.
problem Capturing the polysemy and uncertainty of words in word embeddings.
method Modeling words as multi-sense Gaussian mixtures and using KL divergence for learning.
result The proposed approach effectively captures word entailment and distribution similarity.
Survey of word embedding techniques for NLP.
problem Creating effective word representations for natural language processing.
method Describes recent strategies for fixed-length, dense word embeddings.
result Word embeddings encode syntactic and semantic information and improve NLP tasks.
Unified framework for word embedding models using noise examples.
problem Improving word embedding models with negative sampling.
method Formulated a Word-Context Classification (WCC) framework that generalizes SkipGram word embedding models.
result The best noise distribution is the data distribution, improving both performance and training speed.
Top2Vec finds topic vectors from documents and words without needing stop words or custom settings.
problem Topic modeling weaknesses, including needing known topics, stop words, and custom settings.
method Joint document and word semantic embedding to find topic vectors automatically.
result Top2Vec finds more informative and representative topics than probabilistic models.
Trains word embeddings from music and text data to link music contexts.
problem Varying vocabulary size and musical relevance in word embeddings.
method Combines general text and music-specific data to train word embeddings.
result Trained embeddings better associate music contexts with compositions.
Each free homotopy class of directed closed curves on a surface with boundary can be described by a cyclic reduced word in the generators of the fundamental group and their inverses. The word length is the number of letters of the cyclic word. If the surface has a hyperbolic metric with geodesic boundary, the geometric…
This paper classifies tweets into positive and negative sentiments using distributed word and sentence representations.
problem Classifying tweets into positive and negative sentiments.
method Used distributed representations of words and sentences, and LSTM and CNN networks for classification.
result Achieved accuracies as high as 81%.
We propose a novel Wasserstein method with a distillation mechanism, yielding joint learning of word embeddings and topics. The proposed method is based on the fact that the Euclidean distance between word embeddings may be employed as the underlying distance in the Wasserstein topic model. The word distributions of to…
We describe our language-independent unsupervised word sense induction system. This system only uses topic features to cluster different word senses in their global context topic space. Using unlabeled data, this system trains a latent Dirichlet allocation (LDA) topic model then uses it to infer the topics distribution…
Zipfian Whiteningcs.CL
PCA whitening weighted by Zipfian word frequencies improves task performance.
problem Skewed word embedding spaces in neural models.
method PCA whitening weighted by empirical word frequencies following Zipf's law.
result Significantly improves task performance, surpassing baselines.
Zipf Distributions from Two-Stage Symbolic Processes: Stability Under Stochastic Lexical Filteringstat.ME
Study explains Zipf's law using geometric mechanisms from a finite alphabet.
problem Explains Zipf's law in language without relying on linguistic elements.
method Uses the Full Combinatorial Word Model (FCWM) to generate geometric distributions of word lengths.
result Supports predictions of power-law rank-frequency curves, matching various languages.
This paper proposes an efficient method to train word embeddings for large corpora without synchronization.
problem Training word embeddings for large text corpora is computationally expensive and requires synchronization.
method Partition the input space instead of the vocabulary size, using asynchronous training without parameter synchronization.
result Comparable and up to 45% performance improvement in NLP benchmarks with 1/10 the training time.
The recently introduced continuous Skip-gram model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic word relationships. In this paper we present several extensions that improve both the quality of the vectors and the traini…
We introduce BilBOWA (Bilingual Bag-of-Words without Alignments), a simple and computationally-efficient model for learning bilingual distributed representations of words which can scale to large monolingual datasets and does not require word-aligned parallel training data. Instead it trains directly on monolingual dat…
NNLMs optimize poorly for word probabilities due to embedding space structure.
problem NNLMs assign suboptimal probabilities to some words.
method Analyzed the inductive bias of NNLMs and the structure of word embeddings.
result Words on the convex hull have bounded probability, affecting others.
Geospatial analysis lacks methods like the word vector representations and pre-trained networks that significantly boost performance across a wide range of natural language and computer vision tasks. To fill this gap, we introduce Tile2Vec, an unsupervised representation learning algorithm that extends the distribution…
Traditional topic models do not account for semantic regularities in language. Recent distributional representations of words exhibit semantic consistency over directional metrics such as cosine similarity. However, neither categorical nor Gaussian observational distributions used in existing topic models are appropria…
Paper proposes a new method for sentence embeddings using weighted word vectors.
problem Improving sentence embeddings for natural language processing tasks.
method A simple sentence embedding method using weighted average of word vectors followed by soft projection.
result Demonstrates effectiveness on clinical semantic textual similarity task.
New methods for unsupervised learning of word and entity representations.
problem Learning distributed representations of words and entities from text and knowledge bases.
method MVLSA for words and NVSE for entities, both unsupervised learning methods.
result MVLSA and NVSE outperform state-of-the-art models in word and entity representation learning.
Proposes a new multi-layer model for topic distributions.
problem Leveraging deep structures for learning word distributions of topics.
method A multi-layer generative process on word distributions of topics, where each topic is drawn from a mixture of topics from the layer above.
result Discover interpretable topic hierarchies and improve topic models' accuracy and interpretability.
ETM discovers interpretable topics in large vocabularies.
problem Existing topic models fail with large, heavy-tailed vocabularies.
method Generative model combining topic models and word embeddings with variational inference.
result ETM discovers interpretable topics even with large vocabularies.
We describe a method for learning word embeddings with data-dependent dimensionality. Our Stochastic Dimensionality Skip-Gram (SD-SG) and Stochastic Dimensionality Continuous Bag-of-Words (SD-CBOW) are nonparametric analogs of Mikolov et al.'s (2013) well-known 'word2vec' models. Vector dimensionality is made dynamic b…
Model learns multilingual word representations robust to noise.
problem Learning multilingual word representations in noisy environments.
method Fit a generative latent variable model to a multilingual dictionary.
result Competitive multilingual embeddings across various tasks.
Unsupervised model learns word and context embeddings from character sequences.
problem Learning meaningful word and context embeddings from unlabeled data.
method Character-aware neural architecture that jointly learns word and context embeddings.
result Compact encoders achieve high performance in downstream tasks.
In the field of Natural Language Processing (NLP), we revisit the well-known word embedding algorithm word2vec. Word embeddings identify words by vectors such that the words' distributional similarity is captured. Unexpectedly, besides semantic similarity even relational similarity has been shown to be captured in word…
Embedding words in a vector space has gained a lot of attention in recent years. While state-of-the-art methods provide efficient computation of word similarities via a low-dimensional matrix embedding, their motivation is often left unclear. In this paper, we argue that word embedding can be naturally viewed as a rank…
Improves text clustering by incorporating sequential features and word embeddings.
problem Lack of sequential information and synonym handling in current text clustering methods.
method SiDPMM model that models documents as joint of bags of words, sequential features, and word embeddings.
result Significant improvement in performance and accurate inference of cluster numbers.
We simplify word embeddings by removing sigmoid in SGNS, revealing connections to hyperbolic spaces.
problem Improving word embeddings quality and understanding their relationship with hyperbolic spaces.
method Analyzing squashed shifted PMI matrix and its relation to graph properties and hyperbolic geometry.
result Word embeddings can be connected to hyperbolic spaces through squashed shifted PMI matrix.
CPGBN analyzes text sequences, capturing word order for better topic extraction.
problem Lack of word order in traditional text representations.
method CPFA processes words as sequences, CPGBN adds hierarchical topic modeling.
result CPGBN extracts high-quality latent representations capturing word order.
This paper improves NLP interpretability by using sentence segments instead of words.
problem Limitations of word-based sampling in explaining complex BERT models.
method Using sentence segments as elementary building blocks for NLP interpretability.
result Improved fidelity of the explainer on a benchmark classification task.
Models of bags of words typically assume topic mixing so that the words in a single bag come from a limited number of topics. We show here that many sets of bag of words exhibit a very different pattern of variation than the patterns that are efficiently captured by topic mixing. In many cases, from one bag of words to…
Enhances topic models to better handle polysemous words.
problem Lack of polysemy handling in Gaussian latent Dirichlet allocation.
method Introduces a hierarchical structure to capture polysemy in Gaussian latent Dirichlet allocation.
result Significantly improves polysemy detection and provides more parsimonious topic representations.
Social network analysis (SNA), which is a research field describing and modeling the social connection of a certain group of people, is popular among network services. Our topic words analysis project is a SNA method to visualize the topic words among emails from Obama.com to accounts registered in Columbus, Ohio. Base…
Neural network optimizes learning sequence for reading words.
problem Children struggle with learning to read words due to inconsistent spelling-sound correspondences.
method Used a neural network to structure learning trials to optimize generalization accuracy.
result Significant improvement in generalization accuracy compared to random or frequency-based sequences.
Necessary and Sufficient Conditions and a Provably Efficient Algorithm for Separable Topic Discoverycs.LG
We develop necessary and sufficient conditions and a novel provably consistent and efficient algorithm for discovering topics (latent factors) from observations (documents) that are realized from a probabilistic mixture of shared latent factors that have certain properties. Our focus is on the class of topic models in …
By representing words with probability densities rather than point vectors, probabilistic word embeddings can capture rich and interpretable semantic information and uncertainty. The uncertainty information can be particularly meaningful in capturing entailment relationships -- whereby general words such as "entity" co…
Structural correspondence learning (SCL) is an effective method for cross-lingual sentiment classification. This approach uses unlabeled documents along with a word translation oracle to automatically induce task specific, cross-lingual correspondences. It transfers knowledge through identifying important features, i.e…
The simplicial condition and other stronger conditions that imply it have recently played a central role in developing polynomial time algorithms with provable asymptotic consistency and sample complexity guarantees for topic estimation in separable topic models. Of these algorithms, those that rely solely on the simpl…
Research improves federated text models for next word prediction.
problem Training models on distributed devices efficiently and effectively.
method Employ transfer learning in federated learning for next word prediction.
result Enhancements to current baselines with pretrained embeddings and whole model pretraining.
Proposes SNML for selecting word2vec Skip-gram dimensionality.
problem Selecting optimal dimensionality for word2vec Skip-gram models.
method Information criteria (AIC, BIC, SNML) applied to SG and SG Negative Sampling models.
result SNML outperforms AIC and BIC, selecting closer optimal dimensionality.
A simple text model shows word lengths follow Zipf's law.
problem Understanding word statistics in large language models.
method A non-linguistic model of text with independent symbol draws.
result Word lengths follow a geometric distribution and Zipf's law.
A new method for measuring document similarity using hierarchical optimal transport.
problem Inability to measure semantic similarities and scalability issues in past document similarity measures.
method Model documents as distributions over topics, topics as distributions over words, solve optimal transport problem on topics.
result Hierarchical optimal transport provides better interpretability and scalability with comparable performance.
FMP improves personalized ASR models on private devices.
problem Limited accuracy and privacy in federated fine-tuning of ASR models.
method FMP estimates global and personalized marginal distributions and adjusts NNLM probabilities.
result FMP achieves modest WER reductions on ASR rescoring tasks.
Paper introduces a hyperbolic Gaussian distribution for better learning in hierarchical data.
problem Learning hierarchical data in hyperbolic space.
method Developed a novel hyperbolic distribution for gradient-based learning.
result Demonstrated improved learning on various datasets.
Word embeddings have been shown to be useful across state-of-the-art systems in many natural language processing tasks, ranging from question answering systems to dependency parsing. (Herbelot and Vecchi, 2015) explored word embeddings and their utility for modeling language semantics. In particular, they presented an …
Bidirectional attention is shown to be equivalent to a continuous bag of words model with mixture-of-experts.
problem Understanding the statistical underpinnings of bidirectional attention.
method Exploring bidirectional attention as a mixture-of-experts model and reparameterizing it.
result Bidirectional attention can be viewed as a continuous bag of words model with mixture-of-experts weights.
Ensembling word embeddings to improve distributed word representations has shown good success for natural language processing tasks in recent years. These approaches either carry out straightforward mathematical operations over a set of vectors or use unsupervised learning to find a lower-dimensional representation. Th…