Improves NMT by sampling context from predicted sequence during training.
problem Error accumulation and overcorrection in NMT due to mismatched training and inference contexts.
method Samples context words from both ground truth and predicted sequences during training.
result Significant improvements on multiple datasets, including Chinese->English and WMT'14 English->German.
The abstract explains how word and relation representations capture semantic meaning.
problem Understanding how word and relation representations capture semantic meaning.
method Theoretical justification and extension of geometric relationships between word embeddings and knowledge graph representations.
result The geometric relationships between word embeddings correspond to semantic relations between words and entities in knowledge graphs.
We show that the predictability of letters in written English texts depends strongly on their position in the word. The first letters are usually the least easy to predict. This agrees with the intuitive notion that words are well defined subunits in written languages, with much weaker correlations across these units t…
Research improves federated text models for next word prediction.
problem Training models on distributed devices efficiently and effectively.
method Employ transfer learning in federated learning for next word prediction.
result Enhancements to current baselines with pretrained embeddings and whole model pretraining.
A novel method learns word embeddings and topics using Wasserstein distance.
problem Learning word embeddings and topics from text data.
method Distilled Wasserstein learning framework for joint word embedding and topic modeling.
result Superior performance on disease network construction, mortality prediction, and procedure recommendation.
Proposes a new feature-based evaluation method for explaining Deep Learning models in text classification.
problem Lack of consideration for linguistic dependencies in existing attribution-based explanations.
method Investigates perturbations based on embedded features removal from intermediate layers of Convolutional Neural Networks.
result Visualization tool assists analysts in understanding model predictions better.
CD interprets LSTM predictions by identifying word interactions.
problem LSTMs are black boxes; understanding their internal workings is difficult.
method Contextual decomposition (CD) to interpret LSTM predictions.
result CD reliably identifies word interactions and sentiment combinations.
Amobee wins WASSA 2018 emotion prediction with language models and LSTM.
problem Predicting emotions from tweets without explicit mentions.
method Ensemble system of language models and LSTM with CNN attention.
result 1st place with macro F1 score of 0.7145.
Improved stock prediction using news features and RNN.
problem Predicting stock prices with high accuracy.
method Extracted news features, optimized seed words, calculated positive polar, constructed news features, proposed RNN model.
result Our method improves stock prediction accuracy by over 5%.
ChemBoost predicts protein-ligand binding affinity using SMILES syntax.
problem Predicting high affinity drug-target interactions from sequence similarity alone.
method ChemBoost uses SMILES syntax to represent ligands as documents and proteins as sequences or ligand-centric features. It learns chemical word embeddings and predicts affinities using eXtreme Gradient Boosting.
result ChemBoost outperforms state-of-the-art systems in predicting protein-ligand affinities.
A mixture of experts model predicts brain activation from word stimuli.
problem Classical encoding models ignore connections among brain regions.
method Mixture of experts capturing ROI-specific brain activity patterns.
result Model predicts entire brain activation with high spatial accuracy.
Model learns multi-sense word embeddings using bilingual and monolingual data.
problem Learning multi-sense word embeddings with crosslingual information.
method Discrete autoencoder with encoder and decoder components.
result Bilingual word representations outperform monolingual ones across tasks.
This work learns shared word embeddings for acoustic and phonetic sequences.
problem Mapping variable-length acoustic and phonetic sequences to fixed-dimensional vectors.
method Weak supervision and binary classification task to predict word similarity.
result Best model achieves an F1 score of 0.95 for binary classification.
End-to-end ASR model combines word and character representation for improved performance.
problem Difficulty in training with word-level supervision due to sparsity of examples.
method Multi-task learning framework combining word and character representations.
result Improved word-error rate (WER) by interpolating between word-level and character-level models.
A new neural network models discourse relations with latent variables.
problem Jointly modeling discourse relations and word sequences.
method Latent variable recurrent neural network for discourse relations.
result Model outperforms state-of-the-art alternatives on discourse classification tasks.
Textual data predicts electricity consumption and weather.
problem Lack of textual data in time series prediction models.
method Used TF-IDF and neural word embeddings to predict time series from text.
result Textual data can predict time series with sufficient accuracy.
Word2vec improved but lacks multi-meaning words; ConEc creates new embeddings.
problem Lack of meaningful embeddings for words with multiple meanings and OOV words.
method Context encoders (ConEc) extend word2vec by multiplying embeddings with context vectors.
result ConEc creates embeddings for OOV words and words with multiple meanings based on local contexts.
Humor in word embeddings reveals patterns across different groups.
problem Understanding humor in natural language processing.
method Analyzed humor ratings and word embeddings to identify humor features.
result Different demographic groups have distinct humor preferences.
Paper speeds up neural language model inference by 20x for top-k word prediction.
problem Slow inference speed of neural language models on mobile devices.
method Introduced a screening model using Gumbel softmax to approximate softmax layer.
result Achieved 20.4x speedup with 98.9% precision@1 and 99.3% precision@5 for German to English translation.
ETM discovers interpretable topics in large vocabularies.
problem Existing topic models fail with large, heavy-tailed vocabularies.
method Generative model combining topic models and word embeddings with variational inference.
result ETM discovers interpretable topics even with large vocabularies.
The paper explores how word embeddings affect the stability of downstream NLP models.
problem Small changes in training data can cause significant changes in model predictions.
method Empirical and theoretical analysis of embedding instability, including the introduction of eigenspace instability measure.
result Increasing embedding memory can reduce the disagreement in predictions by 5% to 37%.
Machine learning explains text document categorization decisions.
problem Understanding how text documents are categorized by machine learning models.
method Layer-wise relevance propagation (LRP) to trace predictions back to individual words.
result Word-based ML models can be made more comprehensible through LRP.
Parallel decoding improves machine translation efficiency and accuracy.
problem Efficiently generating translations from left to right.
method Conditional masked language modeling for parallel decoding.
result Improves translation performance by over 4 BLEU points.
The study categorizes knowledge graph relations and explains their embedding representations.
problem Understanding how knowledge graph relation representations capture semantic information.
method Categorizing knowledge graph relations into three types and deriving explicit requirements for their representations.
result Empirical properties of relation representations and the performance of methods are justified by the analysis.
Hierarchical density embeddings capture word relationships with uncertainty.
problem Capturing semantic relationships and uncertainty in word embeddings.
method Learn hierarchical representations through probability density encapsulation, using simple loss functions and distance metrics.
result State-of-the-art performance on WordNet and Hyperlex datasets.
Dynamic model tracks word meanings over time.
problem Capturing semantic evolution of words over time.
method Latent diffusion process, variational inference algorithms.
result Higher predictive likelihoods and interpretable word trajectories.
Model predicts enactment probability of U.S. bills using word vectors and context.
problem Predicting the enactment probability of U.S. bills.
method Ensemble model combining text and context variables, using word vectors.
result Combining text and context variables improves prediction accuracy.
NNLMs optimize poorly for word probabilities due to embedding space structure.
problem NNLMs assign suboptimal probabilities to some words.
method Analyzed the inductive bias of NNLMs and the structure of word embeddings.
result Words on the convex hull have bounded probability, affecting others.
Supervised topic models improve clinical diagnostics by interpreting cooccurence patterns in count data.
problem Standard supervised Latent Dirichlet Allocation (sLDA) struggles with documents having many more words than labels and lacks effective use of supervised labels.
method Investigates penalized optimization methods to train interpretable sLDA models using recognition networks for faster inference.
result Preliminary results show improved predictions on heldout data for predicting anti-depressant medication success based on patient history.
A new framework ties word vectors and classifiers for better language modeling.
problem Inefficiencies in learning and parameter usage in conventional language models.
method Introduces a novel loss framework that ties input embeddings and output projections, reducing parameters.
result State-of-the-art performance on Penn Treebank with various network models.
TeLeS improves ASR confidence estimation by considering temporal alignment and lexical errors.
problem Inaccurate confidence scores from E2E ASR models, especially for overconfident predictions.
method Proposes TeLeS, a novel confidence score that considers temporal alignment and lexical errors, and uses shrinkage loss to handle data imbalance.
result TeLeS generalizes well across different languages and ASR models, leading to significant WER reduction.
Analyzes how word meaning is captured by co-occurrence features.
problem Understanding the theoretical basis of word representation by co-occurrences.
method Theoretical analysis of word representation methods using co-occurrences.
result Using multiple context features improves word prediction scores.
This paper tackles rare word problem in low-resource language pairs using NMT.
problem Rare word problem in neural machine translation, especially for low-resource languages.
method Three solutions: enhanced source context, morphology learning, and wordnet synonyms.
result Significant improvements in BLEU scores (+1.0 points) on English-Vietnamese and Japanese-Vietnamese.
Language models help text classification tasks by predicting next words.
problem Lack of theoretical understanding of why language models perform well on downstream tasks.
method Mathematical study of the connection between next word prediction and text classification, formalizing it and quantifying the benefit.
result Language models that are ε-optimal in cross-entropy learn features that can solve classification tasks with linear approximation.
Models of bags of words typically assume topic mixing so that the words in a single bag come from a limited number of topics. We show here that many sets of bag of words exhibit a very different pattern of variation than the patterns that are efficiently captured by topic mixing. In many cases, from one bag of words to…
Locally-contextual CRFs improve sequence labeling performance.
problem Improving sequence labeling with contextual embeddings.
method Locally-contextual nonlinear CRFs using deep neural networks.
result Consistently outperforms linear chain CRF and previous state of the art.
The paper improves part-of-speech tagging with multi-task learning and character-level word representations.
problem Improving part-of-speech tagging accuracy.
method Developed a new character-level word representation using feedforward neural network, pretraining with existing word vectors, and an additional prediction of neighbour labels as an auxiliary loss.
result The methods significantly improved POS tagging performance on English and Russian languages.
Paper introduces a benchmark for predicting bankruptcy from text data.
problem Lack of a common benchmark dataset and evaluation strategy for unstructured data in bankruptcy prediction.
method Describes and evaluates several baseline models, including a bag-of-words model.
result A lightweight bag-of-words model performs surprisingly well, especially when considering data from multiple years.
Develops S-EFE for analyzing grouped data, improving word usage interpretation.
problem Analyzing how words are used differently across related groups of data.
method Structured exponential family embeddings (S-EFE) with hierarchical modeling and amortization.
result S-EFE enables group-specific interpretation of word usage and outperforms EFE.
Study explains Zipf's law using geometric mechanisms from a finite alphabet.
problem Explains Zipf's law in language without relying on linguistic elements.
method Uses the Full Combinatorial Word Model (FCWM) to generate geometric distributions of word lengths.
result Supports predictions of power-law rank-frequency curves, matching various languages.
WideDTA predicts drug-target binding affinity using text-based information.
problem Predicting drug-target binding affinity is a major challenge in drug discovery.
method WideDTA uses chemical and biological textual sequence information, including protein sequence, ligand SMILES, protein domains and motifs, and maximum common substructure words.
result WideDTA outperformed DeepDTA on the KIBA dataset, indicating the word-based sequence representation is a promising alternative.
New method predicts ICU stay for pancreatitis patients.
problem Predicting ICU stay for pancreatitis patients.
method Survival-supervised topic modeling with elastic-net regularized Cox model and anchor words.
result Our method is as accurate as best baselines but more interpretable.
Proposes a new model to predict antidepressants from EHRs, improving accuracy and interpretability.
problem Improving prediction accuracy and interpretability in topic models for clinical data.
method Prediction-constrained latent Dirichlet allocation framework balancing generative likelihood and prediction accuracy.
result Improved prediction of depression medications from EHRs compared to previous methods.
Improved emotion prediction using autoencoder codebooks.
problem Continuous emotion prediction from audio data.
method Bag-of-Words model based on autoencoder codebook.
result Improved CCC scores for emotion dimensions.
Researchers use LRP to explain CNN predictions in NLP tasks.
problem Explaining predictions of complex non-linear classifiers in NLP.
method Layer-wise relevance propagation (LRP) applied to a CNN for topic categorization.
result LRP highlights relevant words for CNN predictions, validating its suitability for NLP.
Neural approach for Chinese word segmentation using lexicon and unlabeled data.
problem Efficiently segment Chinese words with limited labeled data.
method Posterior regularization algorithm with indirect supervision from lexicon and unlabeled data.
result Validation of approach on multiple benchmark datasets in in-domain and cross-domain scenarios.
Attention layers are sensitive to single words, improving generalization over random features.
problem Understanding why attention layers are effective in NLP tasks.
method Study of word sensitivity in random features using BERT-Base word embeddings.
result Attention layers have high word sensitivity, improving generalization over random features.
Generative model learns from multiple sources to embed words and relationships.
problem Lack of diverse training data in specialized domains.
method Integrates evidence from diverse data sources using affine transformations on semantic vector spaces.
result Outperforms recent models on link prediction tasks and partially observed data.