New method uses word subspaces and term-frequency to improve text classification.
problem Lack of semantic meaning in bag-of-words features.
method Proposes word subspaces and term-frequency weighted word subspaces for text classification.
result Improved text classification performance compared to state-of-the-art algorithms.
Vector representations of words have heralded a transformational approach to classical problems in NLP; the most popular example is word2vec. However, a single vector does not suffice to model the polysemous nature of many (frequent) words, i.e., words with multiple meanings. In this paper, we propose a three-fold appr…
BERT captures linguistic features in separate semantic and syntactic subspaces.
problem Understanding how transformer models like BERT represent linguistic features internally.
method Qualitative and quantitative investigations of BERT's internal representations.
result Evidence of a fine-grained geometric representation of word senses and syntactic representations.
Analyzes word2vec-like models revealing linear subspaces learned during training.
problem Understanding representation learning in word embeddings.
method Analytical solution of word2vec loss dynamics and final embeddings.
result Models learn orthogonal linear subspaces incrementally, representing interpretable concepts.
We prove a rigidity theorem for the geometry of the unit ball in random subspaces of the scl norm in B_1^H of a free group. In a free group F of rank k, a random word w of length n (conditioned to lie in [F,F]) has scl(w)=log(2k-1)n/6log(n) + o(n/log(n)) with high probability, and the unit ball in a subspace spanned by…
New method neutralizes gender bias in word embeddings without losing semantic information.
problem Gender biases in word embeddings trained on human-generated corpora.
method Latent Disentanglement and Counterfactual Generation with siamese auto-encoder and gradient reversal layer.
result Our method outperforms existing debiasing methods in preserving semantic information and neutralizing gender biases.
Clarifies metric properties on group power sets.
problem Properties of metric on group power sets.
method Analysis of Hausdorff-like metric induced from word length norm.
result Results on quasi-isometries between subspaces of power sets.
GLIMPS tackles abundant outlier detection in matched subspace detection.
problem Detecting matched subspaces in high-dimensional data with a high proportion of outliers.
method Two-stage approach combining greedy algorithm and mixed integer programming.
result GLIMPS can tolerate over 80% outliers, significantly outperforming state-of-the-art methods.
The problem of clustering noisy and incompletely observed high-dimensional data points into a union of low-dimensional subspaces and a set of outliers is considered. The number of subspaces, their dimensions, and their orientations are assumed unknown. We propose a simple low-complexity subspace clustering algorithm, w…
New algorithm clusters sparse, high-dimensional texts efficiently.
problem Clustering very short texts with high dimensions and sparsity.
method Linear algebra-based subspace clustering algorithm.
result Algorithm performs competitively on text categorization tasks.
Proposes methods to compute optimal transport maps via subspace projections.
problem Computing optimal transport in high dimensions is challenging due to the curse of dimensionality.
method Develops two methods to extrapolate optimal transport plans from subspace projections to the full space.
result The best optimal transport plan is a generalization of the Knothe-Rosenblatt transport.
Transformer++ improves neural machine translation BLEU scores.
problem Handling long-range dependencies in sentences.
method Proposes a new multi-head attention mechanism with context learning.
result Achieves new state-of-the-art BLEU scores on English-to-German and English-to-French translation tasks.
We propose to model the acoustic space of deep neural network (DNN) class-conditional posterior probabilities as a union of low-dimensional subspaces. To that end, the training posteriors are used for dictionary learning and sparse coding. Sparse representation of the test posteriors using this dictionary enables proje…
New hyperbolic 3-pseudomanifolds with unique properties.
problem Understanding cubulable groups in hyperbolic 3-manifolds.
method Constructing compact hyperbolic 3-manifolds with specific boundary conditions.
result Found groups that are word hyperbolic but not cubulable.
This article deals with two topics: the first, which has a general character, is a variation formula for the the determinant line bundle in non-Kählerian geometry. This formula, which is a consequence of the non-Kählerian version of the Grothendieck-Riemann Roch theorem proved recently by Bismut, gives the variation of…
Method captures shared information across many views robustly.
problem Modeling hundreds of views per event and learning robust embeddings without view knowledge.
method View bootstrapping using multi-view correlation and matrix concentration theory.
result View bootstrapping captures shared information across many views robustly.
How can we correlate neural activity in the human brain as it responds to words, with behavioral data expressed as answers to questions about these same words? In short, we want to find latent variables, that explain both the brain activity, as well as the behavioral responses. We show that this is an instance of the C…
The paper combinatorizes spaces of nondegenerate spherical curves.
problem Understanding the homotopy type of spaces of nondegenerate spherical curves.
method Orthogonalization of Frenet frames, decomposition into Schubert cells, and construction of cell complexes.
result Spaces of nondegenerate curves are contractible topological submanifolds.
Gradient-based framework for optimizing text prompts in diffusion models.
problem Efficiently optimizing prompts in text-to-image diffusion models with large domain space and non-differentiable embeddings.
method Formulated as discrete optimization over language space, designed compact subspaces, and introduced shortcut text gradient.
result Empirically discovered prompts that enhance or destroy image faithfulness.
A new method selects anchor words for better topic discovery in text corpora.
problem Selecting anchor words for improved topic modeling in text corpora.
method Proposes a new greedy method to find a minimum edge-weight anchor clique in a word similarity graph.
result The proposed method outperforms existing methods on topic quality and is faster.
Most existing word embedding approaches do not distinguish the same words in different contexts, therefore ignoring their contextual meanings. As a result, the learned embeddings of these words are usually a mixture of multiple meanings. In this paper, we acknowledge multiple identities of the same word in different co…
There are certain families of words and word sequences (words in the generators of a two-generator group) that arise frequently in the Teichm{ü}ller theory of hyperbolic three-manifolds and Kleinian and Fuchsian groups and in the discreteness problem for two generator matrix groups. We survey some of the families of su…
Probabilistic FastText captures multiple word senses and sub-word structures.
problem Capturing multiple word senses and sub-word structures in word embeddings.
method Probabilistic FastText uses Gaussian mixture densities to represent words, sharing statistical strength across sub-word structures and capturing different word senses.
result Probabilistic FastText outperforms existing models on word-similarity benchmarks and discerning different meanings.
FRAGE learns word embeddings without frequency bias, improving performance across NLP tasks.
problem Word embeddings are biased towards word frequency, affecting performance for rare words.
method Adversarial training to learn Frequency-Agnostic word Embedding (FRAGE).
result FRAGE achieves higher performance than baselines in all four NLP tasks.
Geometrically transforms word embeddings into a common space for better comparison.
problem Comparing embeddings from different sources is challenging.
method Applies orthogonal rotations and Mahalanobis scaling to transform embeddings into a shared latent space.
result The method improves word similarity and analogy tasks.
We discuss a topological approach to words introduced by the author. Words on an arbitrary alphabet are approximated by Gauss words and then studied up to natural modifications inspired by the Reidemeister moves on knot diagrams. This leads us to a notion of homotopy for words. We introduce several homotopy invariants …
We fully describe the horofunction boundary ∂hL2 with the word metric associated with the generating set {t,at} (i.e the metric arising in the Diestel-Leader graph DL(2,2)). The visual boundary ∂∞L2 with this metric is a subset of ∂hL2. Although $\partial_\infty L_2…
Paper proposes a new word embedding method optimizing word similarity.
problem Optimizing word similarity in embedding space.
method Two-step random walks between words via topics to learn an optimal embedding simplex.
result Our method outperforms existing approaches in various queries.
Paper finds better words for topic models by reranking top words.
problem Top words in topic models are not always representative.
method Reranking words by considering marginal probability over every topic.
result Reranked top words are more representative of topics.
The abstract explains how word and relation representations capture semantic meaning.
problem Understanding how word and relation representations capture semantic meaning.
method Theoretical justification and extension of geometric relationships between word embeddings and knowledge graph representations.
result The geometric relationships between word embeddings correspond to semantic relations between words and entities in knowledge graphs.
Approaches KL divergence for learning multi-sense word distributions.
problem Capturing the polysemy and uncertainty of words in word embeddings.
method Modeling words as multi-sense Gaussian mixtures and using KL divergence for learning.
result The proposed approach effectively captures word entailment and distribution similarity.
Paper proposes MMD-Sense-Analysis for detecting word sense shifts.
problem Detecting and interpreting shifts in word meanings over time.
method Leverages Maximum Mean Discrepancy (MMD) to identify and explain word sense changes.
result Demonstrates effectiveness of MMD-Sense-Analysis through empirical results.
Paper uses random projection to preserve subspace structure for efficient data analysis.
problem Efficiently analyzing data with low-dimensional structure.
method Compressed Subspace Learning (CSL) framework based on Johnson-Lindenstrauss property.
result Random projection preserves the UoS structure of data, enabling efficient analysis.
Bayesian algorithm improves word representations using semantic taxonomy.
problem Improving word representations in semantic taxonomy.
method Bayesian Hierarchical Words Representation (BHWR) learning algorithm combining Variational Bayes and semantic taxonomy modeling.
result BHWR produces better representations for rare words.
New method improves ASR word confidence for diverse applications.
problem Mitigating ASR errors and improving word error rate.
method Heterogeneous Word Confusion Network (HWCN) with score calibration.
result Word sequence with best overall confidence is more accurate than 1-best result.
Detects missing tensor signals in a KS subspace with high probability.
problem Detecting tensor signals with many missing entities in a KS subspace.
method Projecting the signal onto the KS subspace and bounding residual energy.
result Reliable detection is possible if the missing signal cardinality exceeds KS subspace dimensions.
Recent work on learning multilingual word representations usually relies on the use of word-level alignements (e.g. infered with the help of GIZA++) between translated sentences, in order to align the word embeddings in different languages. In this workshop paper, we investigate an autoencoder model for learning multil…
Proposes MorphMine for unsupervised morpheme segmentation to improve word embeddings.
problem Lack of semantic information in word-level analysis for infrequent and out-of-vocabulary words.
method MorphMine applies a parsimony criterion to hierarchically segment words into the fewest number of morphemes.
result MorphMine segments words into human-verified morphemes and improves word embedding quality.
Estimator Vectors learns OOV word embeddings using subword and context clues.
problem Lack of OOV word representations in neural network models.
method Jointly learns word, subword, and context clue representations.
result Strong estimates for OOV words via combined subword and context clue embeddings.
End-to-end ASR model combines word and character representation for improved performance.
problem Difficulty in training with word-level supervision due to sparsity of examples.
method Multi-task learning framework combining word and character representations.
result Improved word-error rate (WER) by interpolating between word-level and character-level models.
Paper analyzes word embedding composition using tensor decomposition.
problem Given vector representations of two words, compute a vector for the entire phrase.
method Generative model with low rank Tucker decomposition of word embedding correlations.
result Word embeddings and a core tensor can be derived from the Tucker decomposition.
The study examines how character and word-level representations improve sentence-level tasks.
problem Improving the quality of word and sentence representations using character-level information.
method Feature-wise sigmoid gating mechanism for combining character and word-level representations.
result Modeling characters improves final word and sentence representations, especially for less frequent words.
By defining combinatorial moves, we can define an equivalence relation on Gauss words called homotopy. In this paper we define a homotopy invariant of Gauss words. We use this to show that there exist Gauss words that are not homotopically equivalent to the empty Gauss word, disproving a conjecture by Turaev. In fact, …
Word embeddings provide point representations of words containing useful semantic information. We introduce multimodal word distributions formed from Gaussian mixtures, for multiple word meanings, entailment, and rich uncertainty information. To learn these distributions, we propose an energy-based max-margin objective…
We introduce a topological approach to words. Words are approximated by Gauss words and then studied up to natural modifications inspired by homotopy transformations of curves on the plane.
A longstanding question of Gromov asks whether every one-ended word-hyperbolic group contains a subgroup isomorphic to the fundamental group of a closed hyperbolic surface. An infinite family of word-hyperbolic groups can be obtained by taking doubles of free groups amalgamated along words that are not proper powers. W…
Paper bounds subspace estimator error from noisy projections.
problem Estimating subspaces from noisy data.
method Derives perturbation bound on optimal subspace estimator.
result Fundamental result with implications in matrix completion and clustering.
This paper stratifies spaces of locally convex curves by their itineraries.
problem Homotopy type of spaces of locally convex curves with prescribed initial and final jets.
method Defining itineraries and proving submanifolds of finite codimension.
result Stratification of the space of curves by their itineraries.