FRAGE learns word embeddings without frequency bias, improving performance across NLP tasks.
problem Word embeddings are biased towards word frequency, affecting performance for rare words.
method Adversarial training to learn Frequency-Agnostic word Embedding (FRAGE).
result FRAGE achieves higher performance than baselines in all four NLP tasks.
Estimator Vectors learns OOV word embeddings using subword and context clues.
problem Lack of OOV word representations in neural network models.
method Jointly learns word, subword, and context clue representations.
result Strong estimates for OOV words via combined subword and context clue embeddings.
We find words embed in Lie groups and propose new neural networks.
problem Understanding word embeddings and RNNs in natural language processing.
method Hyperparameter tuning and analysis of GRU and word embeddings.
result Words naturally embed in Lie groups and RNNs form nonlinear representations.
Unsupervised model learns word and context embeddings from character sequences.
problem Learning meaningful word and context embeddings from unlabeled data.
method Character-aware neural architecture that jointly learns word and context embeddings.
result Compact encoders achieve high performance in downstream tasks.
This paper reviews different word embeddings for sentiment classification using deep learning.
problem Handling large textual data with simple ML algorithms.
method Word embedding strategies implemented on an Amazon Review Dataset.
result Different word embeddings improve accuracy in sentiment classification.
NNLMs optimize poorly for word probabilities due to embedding space structure.
problem NNLMs assign suboptimal probabilities to some words.
method Analyzed the inductive bias of NNLMs and the structure of word embeddings.
result Words on the convex hull have bounded probability, affecting others.
Paper distills word embeddings to reduce dimensionality without sacrificing accuracy.
problem Reducing neural model size for practical deployment.
method Teacher-student model-based embedding distillation with ensemble learning.
result Significant reduction in model size (80x faster and lighter) with minimal accuracy loss.
New insights into word embeddings reveal linear relationships behind analogy phenomena.
problem Understanding the linear behavior of word embeddings in analogy tasks.
method Derive a probabilistic definition of paraphrasing, interpret as word transformation, and prove linear relationships.
result Existence of linear relationships between W2V-type embeddings underpinning analogy phenomena.
Enhanced word embeddings boost multiclass text classification accuracy.
problem Improving multiclass text classification accuracy using pre-trained embeddings.
method Proposed word-class embeddings (WCEs) to enhance pre-trained word embeddings.
result WCEs significantly improve multiclass text classification accuracy.
Paper debiases multiple word embedding biases simultaneously.
problem Reduction of multiple biases in word embeddings.
method Joint multiclass debiasing approach using Word Embeddings Association Test (WEAT).
result Demonstrates reduction or complete elimination of bias in word embeddings.
New method reduces word embedding storage space by 100x.
problem Large space required for storing word embeddings.
method Inspired by quantum computing, proposes word2ket and word2ketXS methods.
result Achieves a hundred-fold reduction in space required for word embeddings.
Neural model improves text normalization for non-English languages.
problem Improving text normalization in non-English languages with limited data.
method Sequence-to-sequence model with character and word embeddings, using pre-trained word embeddings with subword information.
result Achieved state-of-the-art F1 score on Arabic language correction dataset.
CNNs and biomedical word embeddings detect ADRs in literature.
problem Detecting Adverse Drug Reactions (ADRs) in biomedical literature is time-consuming.
method Used token-level convolutions and biomedical word embeddings in CNN architectures.
result CNNs outperform traditional models and word embeddings improve performance.
PCA whitening weighted by Zipfian word frequencies improves task performance.
problem Skewed word embedding spaces in neural models.
method PCA whitening weighted by empirical word frequencies following Zipf's law.
result Significantly improves task performance, surpassing baselines.
With a simple architecture and the ability to learn meaningful word embeddings efficiently from texts containing billions of words, word2vec remains one of the most popular neural language models used today. However, as only a single embedding is learned for every word in the vocabulary, the model fails to optimally re…
A new neural network model extends word embedding vectors with MeSH concepts for biomedical semantic similarity.
problem Eliciting semantic similarity between biomedical concepts remains challenging.
method Proposes a MeSH-gram neural network model that extends skip-gram by using MeSH descriptors.
result MeSH-gram outperforms skip-gram and is comparable to best methods but requires more computation and external resources.
Improved SGNS model resolves ambiguity in word vector learning.
problem Ambiguity in word vectors under SGNS model.
method Rectified SGNS model with quadratic regularization.
result Simple modification structures the solution effectively.
A new neural network for text classification reduces parameters with improved accuracy.
problem Reducing the number of parameters in text classification models.
method Compositional coding, capsule network, k-means routing algorithm.
result The proposed method achieves competitive accuracy with significantly fewer parameters.
Proposes a new feature-based evaluation method for explaining Deep Learning models in text classification.
problem Lack of consideration for linguistic dependencies in existing attribution-based explanations.
method Investigates perturbations based on embedded features removal from intermediate layers of Convolutional Neural Networks.
result Visualization tool assists analysts in understanding model predictions better.
Fruit fly brain network learns word embeddings using sparse binary codes.
problem Learning semantic word representations from text.
method Inspired by mushroom body neural network, sparse binary hash codes.
result Fruit fly network achieves comparable NLP performance with reduced resources.
Paper uses neural word embeddings to analyze UN speeches for policy preferences and voting behavior.
problem Analyzing policy preferences and paradigm shifts in international politics.
method Applied neural word embeddings (Word2vec) to UN General Debate speeches.
result Found statistical relation between speech semantic content and voting behavior, contrary to hypothesis.
Neural topic models improve topic discovery in large corpora.
problem Topic discovery in large text corpora.
method Neural auto-encoder based document embedding with PLSA assumptions.
result Neural topic models outperform LDA in capturing relevant topics and perplexity.
D-ETM models document topics over time using embeddings and variational inference.
problem Capturing evolving topic patterns in sequential documents.
method Combines D-LDA and word embeddings, using random walk priors and variational inference.
result D-ETM outperforms D-LDA on document completion tasks, learning more diverse and coherent topics.
Paper develops neural network for Mandarin polyphone disambiguation.
problem Homograph problem in Mandarin Chinese text-to-speech.
method Bidirectional RNN for context, prediction network for mapping embeddings to pronunciations.
result Achieves 94.69% accuracy on polyphonic character dataset.
Most existing word embedding approaches do not distinguish the same words in different contexts, therefore ignoring their contextual meanings. As a result, the learned embeddings of these words are usually a mixture of multiple meanings. In this paper, we acknowledge multiple identities of the same word in different co…
Geometrically transforms word embeddings into a common space for better comparison.
problem Comparing embeddings from different sources is challenging.
method Applies orthogonal rotations and Mahalanobis scaling to transform embeddings into a shared latent space.
result The method improves word similarity and analogy tasks.
Paper proposes a new word embedding method optimizing word similarity.
problem Optimizing word similarity in embedding space.
method Two-step random walks between words via topics to learn an optimal embedding simplex.
result Our method outperforms existing approaches in various queries.
Neural Machine Translation (MT) has reached state-of-the-art results. However, one of the main challenges that neural MT still faces is dealing with very large vocabularies and morphologically rich languages. In this paper, we propose a neural MT system using character-based embeddings in combination with convolutional…
Most existing word embedding methods can be categorized into Neural Embedding Models and Matrix Factorization (MF)-based methods. However some models are opaque to probabilistic interpretation, and MF-based methods, typically solved using Singular Value Decomposition (SVD), may incur loss of corpus information. In addi…
Paper introduces score embedding for Twitter sentiment analysis of health care issues.
problem Analyzing public opinion on health care plans using social media.
method Score embedding, a neural network model for word representations.
result Score embedding effectively captures sentiment and outperforms existing methods.
Word embeddings are a powerful approach for capturing semantic similarity among terms in a vocabulary. In this paper, we develop exponential family embeddings, a class of methods that extends the idea of word embeddings to other types of high-dimensional data. As examples, we studied neural data with real-valued observ…
BERT-based word embeddings improve active learning for text datasets.
problem Efficiently labelling large text datasets for machine learning.
method Evaluation of text representation mechanisms (BERT vs. bag of words) in active learning.
result BERT-based word embeddings significantly improve active learning performance.
SWESA learns word embeddings with document labels for sentiment analysis.
problem Sentiment analysis using limited text data.
method SWESA uses supervised learning to optimize word embeddings and classifier performance.
result SWESA outperforms existing methods in sentiment analysis.
Textual data predicts electricity consumption and weather.
problem Lack of textual data in time series prediction models.
method Used TF-IDF and neural word embeddings to predict time series from text.
result Textual data can predict time series with sufficient accuracy.
Paper explains word embedding dimensionality and proposes a new metric.
problem Understanding the optimal dimensionality of word embeddings.
method Proposed Pairwise Inner Product (PIP) loss to measure dissimilarity.
result Revealed a bias-variance trade-off in dimensionality selection.
We create interpretable word embeddings through sparse coding.
problem Difficult to interpret word embeddings in natural language processing.
method Transform pretrained dense word embeddings into sparse embeddings through sparse coding.
result Sparse embeddings are more interpretable and achieve good performance.
Paper uses JIVE to decompose word embeddings, improving sentiment analysis performance.
problem Improving sentiment analysis performance on word embeddings.
method Joint and individual variance explained (JIVE) method for decomposition.
result Mapping word embeddings into joint components improves sentiment analysis performance.
New methods for unsupervised learning of word and entity representations.
problem Learning distributed representations of words and entities from text and knowledge bases.
method MVLSA for words and NVSE for entities, both unsupervised learning methods.
result MVLSA and NVSE outperform state-of-the-art models in word and entity representation learning.
This paper tackles rare word problem in low-resource language pairs using NMT.
problem Rare word problem in neural machine translation, especially for low-resource languages.
method Three solutions: enhanced source context, morphology learning, and wordnet synonyms.
result Significant improvements in BLEU scores (+1.0 points) on English-Vietnamese and Japanese-Vietnamese.
WME generates document embeddings from word embeddings, outperforming state-of-the-art techniques.
problem Lack of unsupervised document embeddings from pre-trained word embeddings.
method Word Mover's Embedding (WME) approach.
result WME consistently matches or outperforms state-of-the-art techniques on various text classification and similarity tasks.
Locally-contextual CRFs improve sequence labeling performance.
problem Improving sequence labeling with contextual embeddings.
method Locally-contextual nonlinear CRFs using deep neural networks.
result Consistently outperforms linear chain CRF and previous state of the art.
Paper proposes FOFE for efficient WSD.
problem Word sense disambiguation (WSD) problem.
method Fixed-size ordinally forgetting encoding (FOFE) combined with FFNN.
result FOFE-based FFNN achieves comparable performance to state-of-the-art at lower cost.
Paper analyzes word embedding composition using tensor decomposition.
problem Given vector representations of two words, compute a vector for the entire phrase.
method Generative model with low rank Tucker decomposition of word embedding correlations.
result Word embeddings and a core tensor can be derived from the Tucker decomposition.
Proposes MorphMine for unsupervised morpheme segmentation to improve word embeddings.
problem Lack of semantic information in word-level analysis for infrequent and out-of-vocabulary words.
method MorphMine applies a parsimony criterion to hierarchically segment words into the fewest number of morphemes.
result MorphMine segments words into human-verified morphemes and improves word embedding quality.
Improved NER on Turkish tweets using semi-supervised learning and word embeddings.
problem Named Entity Recognition on informal Turkish text types.
method Semi-supervised learning with neural networks and word embeddings.
result Achieved better F-score performances than previous Turkish NER systems.
GroupReduce compresses neural language models by reducing embedding and softmax matrices.
problem Large vocabulary size in neural language models leads to high model size and memory usage.
method Vocabulary-partition based low-rank matrix approximation and frequency distribution of tokens.
result Significantly outperforms traditional compression methods with 6.6 times compression rate for embedding and softmax matrices.
Word embeddings in hyperbolic space outperform Euclidean ones.
problem Improving word embeddings for better performance.
method Learning word embeddings in hyperbolic space using skip-gram architecture and hyperbolic distance objective function.
result Hyperbolic word embeddings show potential, especially in low dimensions, but not clear superiority over Euclidean embeddings.
Unified framework for word embedding models using noise examples.
problem Improving word embedding models with negative sampling.
method Formulated a Word-Context Classification (WCC) framework that generalizes SkipGram word embedding models.
result The best noise distribution is the data distribution, improving both performance and training speed.