Probabilistic FastText captures multiple word senses and sub-word structures.
problem Capturing multiple word senses and sub-word structures in word embeddings.
method Probabilistic FastText uses Gaussian mixture densities to represent words, sharing statistical strength across sub-word structures and capturing different word senses.
result Probabilistic FastText outperforms existing models on word-similarity benchmarks and discerning different meanings.
Paper finds better words for topic models by reranking top words.
problem Top words in topic models are not always representative.
method Reranking words by considering marginal probability over every topic.
result Reranked top words are more representative of topics.
End-to-end ASR model combines word and character representation for improved performance.
problem Difficulty in training with word-level supervision due to sparsity of examples.
method Multi-task learning framework combining word and character representations.
result Improved word-error rate (WER) by interpolating between word-level and character-level models.
FRAGE learns word embeddings without frequency bias, improving performance across NLP tasks.
problem Word embeddings are biased towards word frequency, affecting performance for rare words.
method Adversarial training to learn Frequency-Agnostic word Embedding (FRAGE).
result FRAGE achieves higher performance than baselines in all four NLP tasks.
New method improves ASR word confidence for diverse applications.
problem Mitigating ASR errors and improving word error rate.
method Heterogeneous Word Confusion Network (HWCN) with score calibration.
result Word sequence with best overall confidence is more accurate than 1-best result.
Word2vec improved but lacks multi-meaning words; ConEc creates new embeddings.
problem Lack of meaningful embeddings for words with multiple meanings and OOV words.
method Context encoders (ConEc) extend word2vec by multiplying embeddings with context vectors.
result ConEc creates embeddings for OOV words and words with multiple meanings based on local contexts.
Bayesian algorithm improves word representations using semantic taxonomy.
problem Improving word representations in semantic taxonomy.
method Bayesian Hierarchical Words Representation (BHWR) learning algorithm combining Variational Bayes and semantic taxonomy modeling.
result BHWR produces better representations for rare words.
Interactive topic models with user-defined anchor words.
problem Building interpretable topic models.
method Protocol allowing users to interactively define anchor words for topic modeling.
result Validation of the usefulness of interactive topic models.
A new method selects anchor words for better topic discovery in text corpora.
problem Selecting anchor words for improved topic modeling in text corpora.
method Proposes a new greedy method to find a minimum edge-weight anchor clique in a word similarity graph.
result The proposed method outperforms existing methods on topic quality and is faster.
New method uses word subspaces and term-frequency to improve text classification.
problem Lack of semantic meaning in bag-of-words features.
method Proposes word subspaces and term-frequency weighted word subspaces for text classification.
result Improved text classification performance compared to state-of-the-art algorithms.
This paper closes the accuracy gap between A2W models and sub-word models using English conversational speech data.
problem Closing the accuracy gap between direct acoustics-to-word models and sub-word models.
method Training an A2W model with orders of magnitude more data, optimizing model initialization, training data order, and regularization.
result Achieved word error rates of 8.8%/13.9% on Hub5-2000 Switchboard/CallHome test sets.
Estimator Vectors learns OOV word embeddings using subword and context clues.
problem Lack of OOV word representations in neural network models.
method Jointly learns word, subword, and context clue representations.
result Strong estimates for OOV words via combined subword and context clue embeddings.
Direct acoustics-to-word models improve speech recognition without LMs.
problem Improving speech recognition without requiring a Language Model (LM).
method Direct acoustics-to-word CTC models trained on public benchmark tasks.
result CTC word model achieves 13.0%/18.8% word error rate compared to 9.6%/16.0% for phone-based CTC with a 4-gram LM.
Sparse activations in neural models correlate with frequent words, suggesting sparsity is natural.
problem Interpretability and resource efficiency in neural language models.
method Used the Taxi-Euclidean norm to measure sparsity and analyzed gradients and activations of frequent words.
result Frequent input words are associated with sparse activations, while frequent target words are associated with dispersed activations.
The abstract explains how word and relation representations capture semantic meaning.
problem Understanding how word and relation representations capture semantic meaning.
method Theoretical justification and extension of geometric relationships between word embeddings and knowledge graph representations.
result The geometric relationships between word embeddings correspond to semantic relations between words and entities in knowledge graphs.
Paper learns identity-sensitive word embeddings from text corpora.
problem Lack of context-aware word embeddings.
method Constructs a heterogeneous network of words and identities, then embeds into a low-dimensional space.
result Identity-sensitive word embeddings capture different meanings of words.
PCA whitening weighted by Zipfian word frequencies improves task performance.
problem Skewed word embedding spaces in neural models.
method PCA whitening weighted by empirical word frequencies following Zipf's law.
result Significantly improves task performance, surpassing baselines.
Paper analyzes word embedding composition using tensor decomposition.
problem Given vector representations of two words, compute a vector for the entire phrase.
method Generative model with low rank Tucker decomposition of word embedding correlations.
result Word embeddings and a core tensor can be derived from the Tucker decomposition.
ETM discovers interpretable topics in large vocabularies.
problem Existing topic models fail with large, heavy-tailed vocabularies.
method Generative model combining topic models and word embeddings with variational inference.
result ETM discovers interpretable topics even with large vocabularies.
Unified framework for word embedding models using noise examples.
problem Improving word embedding models with negative sampling.
method Formulated a Word-Context Classification (WCC) framework that generalizes SkipGram word embedding models.
result The best noise distribution is the data distribution, improving both performance and training speed.
Proposes MorphMine for unsupervised morpheme segmentation to improve word embeddings.
problem Lack of semantic information in word-level analysis for infrequent and out-of-vocabulary words.
method MorphMine applies a parsimony criterion to hierarchically segment words into the fewest number of morphemes.
result MorphMine segments words into human-verified morphemes and improves word embedding quality.
This work learns shared word embeddings for acoustic and phonetic sequences.
problem Mapping variable-length acoustic and phonetic sequences to fixed-dimensional vectors.
method Weak supervision and binary classification task to predict word similarity.
result Best model achieves an F1 score of 0.95 for binary classification.
Top2Vec finds topic vectors from documents and words without needing stop words or custom settings.
problem Topic modeling weaknesses, including needing known topics, stop words, and custom settings.
method Joint document and word semantic embedding to find topic vectors automatically.
result Top2Vec finds more informative and representative topics than probabilistic models.
The paper shows context vectors are half the dimensions of word vectors.
problem Understanding the relationship between word and context embeddings.
method Starting from probabilistic assumptions, the paper shows context vectors are reflections of word vectors in half the dimensions.
result Context vectors are reflections of word vectors in approximately half the dimensions.
GROVER improves word representations by gradually adding random noises during training.
problem Improving word representations for better model performance.
method Gradually adding random noises to word embeddings during training.
result GROVER improves model performances on most text classification datasets.
The paper develops dynamic word embeddings to capture evolving language structures.
problem Capturing the evolving meanings and associations of words over time.
method Develops a dynamic statistical model to learn time-aware word vector representation, solving the alignment problem.
result The model reliably captures the evolution of language over time and outperforms state-of-the-art approaches.
Paper proposes a method to align word embedding models in a joint latent space.
problem Challenges in aligning variations of word embedding models.
method Generative process using synthetic data points based on linguistic relationships.
result Substantial improvements in recovering embeddings of local neighborhoods.
A novel method learns word embeddings and topics using Wasserstein distance.
problem Learning word embeddings and topics from text data.
method Distilled Wasserstein learning framework for joint word embedding and topic modeling.
result Superior performance on disease network construction, mortality prediction, and procedure recommendation.
Paper proposes a new word embedding method optimizing word similarity.
problem Optimizing word similarity in embedding space.
method Two-step random walks between words via topics to learn an optimal embedding simplex.
result Our method outperforms existing approaches in various queries.
We create interpretable word embeddings through sparse coding.
problem Difficult to interpret word embeddings in natural language processing.
method Transform pretrained dense word embeddings into sparse embeddings through sparse coding.
result Sparse embeddings are more interpretable and achieve good performance.
Simple model classifies traffic tweets from Twitter using word embeddings.
problem Sparsity and curse of dimensionality in bag-of-words representation for traffic tweets.
method Proposes a word embedding-based framework to classify traffic-related tweets from non-traffic ones.
result State-of-the-art test accuracy of 95.9% achieved with a simple model.
Approaches KL divergence for learning multi-sense word distributions.
problem Capturing the polysemy and uncertainty of words in word embeddings.
method Modeling words as multi-sense Gaussian mixtures and using KL divergence for learning.
result The proposed approach effectively captures word entailment and distribution similarity.
Paper proposes a method to learn word translations bidirectionally.
problem Word translation between languages.
method Jointly learns translations in both directions with minimal supervision.
result Improves accuracy of translations over previous methods.
Recent work on learning multilingual word representations usually relies on the use of word-level alignements (e.g. infered with the help of GIZA++) between translated sentences, in order to align the word embeddings in different languages. In this workshop paper, we investigate an autoencoder model for learning multil…
A new framework ties word vectors and classifiers for better language modeling.
problem Inefficiencies in learning and parameter usage in conventional language models.
method Introduces a novel loss framework that ties input embeddings and output projections, reducing parameters.
result State-of-the-art performance on Penn Treebank with various network models.
Improves text clustering by incorporating sequential features and word embeddings.
problem Lack of sequential information and synonym handling in current text clustering methods.
method SiDPMM model that models documents as joint of bags of words, sequential features, and word embeddings.
result Significant improvement in performance and accurate inference of cluster numbers.
Dynamic model tracks word meanings over time.
problem Capturing semantic evolution of words over time.
method Latent diffusion process, variational inference algorithms.
result Higher predictive likelihoods and interpretable word trajectories.
Models of bags of words typically assume topic mixing so that the words in a single bag come from a limited number of topics. We show here that many sets of bag of words exhibit a very different pattern of variation than the patterns that are efficiently captured by topic mixing. In many cases, from one bag of words to…
Corpus poisoning can manipulate word meanings in word embeddings, affecting natural language processing tasks.
problem Controlling word meanings via corpus modifications.
method Developed an explicit expression over corpus features to control word embeddings.
result Demonstrated the ability to manipulate word meanings in word embeddings, affecting various downstream tasks.
Paper extends topic models using neighborhood aggregation for better performance.
problem Extending topic models with pre-trained word embeddings and nonlinear output functions.
method Network view of topic models, neighborhood aggregation algorithm.
result Approach outperforms state-of-the-art supervised Latent Dirichlet Allocation.
Social network analysis (SNA), which is a research field describing and modeling the social connection of a certain group of people, is popular among network services. Our topic words analysis project is a SNA method to visualize the topic words among emails from Obama.com to accounts registered in Columbus, Ohio. Base…
The study examines how character and word-level representations improve sentence-level tasks.
problem Improving the quality of word and sentence representations using character-level information.
method Feature-wise sigmoid gating mechanism for combining character and word-level representations.
result Modeling characters improves final word and sentence representations, especially for less frequent words.
Paper evaluates word embedding techniques on Russian digital humanities datasets.
problem Evaluating word embedding techniques on Russian language datasets.
method Created and evaluated word intrusion and word analogy tasks on manually curated datasets.
result Baseline evaluations with word embedding models show differences in performance for Russian and English datasets.
Improved SGNS model resolves ambiguity in word vector learning.
problem Ambiguity in word vectors under SGNS model.
method Rectified SGNS model with quadratic regularization.
result Simple modification structures the solution effectively.
Model learns multilingual word representations robust to noise.
problem Learning multilingual word representations in noisy environments.
method Fit a generative latent variable model to a multilingual dictionary.
result Competitive multilingual embeddings across various tasks.
Enhanced word embeddings boost multiclass text classification accuracy.
problem Improving multiclass text classification accuracy using pre-trained embeddings.
method Proposed word-class embeddings (WCEs) to enhance pre-trained word embeddings.
result WCEs significantly improve multiclass text classification accuracy.
Semantic word embeddings represent the meaning of a word via a vector, and are created by diverse methods. Many use nonlinear operations on co-occurrence statistics, and have hand-tuned hyperparameters and reweighting methods. This paper proposes a new generative model, a dynamic version of the log-linear topic model o…
This paper assesses biases in contextualized word representations.
problem Analyzing biases in contextualized word representations.
method Proposes assessing bias at the contextual word level, capturing contextual effects of bias.
result Demonstrates evidence of bias in contextual word models, including racial bias and exacerbated effects for intersectional minorities.