arXiv research

Framework learns portable representations for diverse tasks.

Study shows how transformers classify symbols without naming them, proving a margin-versus-collision criterion.

We present a novel, scalable and Bayesian approach to modelling the occurrence of pairs of symbols (i,j) drawn from a large vocabulary. Observed pairs are assumed to be generated by a simple popularity based selection process followed by censoring using a preference function. By basing inference on the well-founded pri…

Conventional embedding methods directly associate each symbol with a continuous embedding vector, which is equivalent to applying a linear transformation based on a "one-hot" encoding of the discrete symbols. Despite its simplicity, such approach yields the number of parameters that grows linearly with the vocabulary s…

Paper proposes a bijective approach for signal/symbol translation using variational auto-encoders.

This paper shows that scientific discovery can be efficiently learned via compositional function trees, reducing the sample complexity.

Embedding methods such as word embedding have become pillars for many applications containing discrete structures. Conventional embedding methods directly associate each symbol with a continuous embedding vector, which is equivalent to applying linear transformation based on "one-hot" encoding of the discrete symbols. …

More than 50 years ago Bongard introduced 100 visual concept learning problems as a testbed for intelligent vision systems. These problems are now known as Bongard problems. Although they are well known in the cognitive science and AI communities only moderate progress has been made towards building systems that can so…

Typilus predicts types for Python programs using neural networks.

A simple text model shows word lengths follow Zipf's law.

Study finds optimal vocabulary size for neural machine translation.

Paper reduces vocabulary losslessly for language model cooperation.

This study analyzes data science vocabulary changes over 13 years.

Paper explores EEG-based speech recognition using transformers, showing faster training and better performance for smaller vocabularies.

With the rise of social media, people can now form relationships and communities easily regardless of location, race, ethnicity, or gender. However, the power of social media simultaneously enables harmful online behavior such as harassment and bullying. Cyberbullying is a serious social problem, making it an important…

Despite significant progress in object categorization, in recent years, a number of important challenges remain, mainly, ability to learn from limited labeled data and ability to recognize object classes within large, potentially open, set of labels. Zero-shot learning is one way of addressing these challenges, but it …

Paper shows continuous speech recognition with EEG features, no speech input.

New analysis shows optimal embedding learning rate depends on vocabulary size, not just model width.

ETM discovers interpretable topics in large vocabularies.

Effective multilingual search with instance-based transfer learning.

Dynamic Vocabulary Pruning stabilizes LLM training by removing low-probability tokens.

The novel unseen classes can be formulated as the extreme values of known classes. This inspired the recent works on open-set recognition \cite{Scheirer_2013_TPAMI,Scheirer_2014_TPAMIb,EVM}, which however can have no way of naming the novel unseen classes. To solve this problem, we propose the Extreme Value Learning (E…

ANT learns sparse embeddings for large vocabularies efficiently.

Training-free source selection for LLM families with shared vocabularies

In latent Dirichlet allocation (LDA), topics are multinomial distributions over the entire vocabulary. However, the vocabulary usually contains many words that are not relevant in forming the topics. We adopt a variable selection method widely used in statistical modeling as a dimension reduction tool and combine it wi…

Machine learning models that take computer program source code as input typically use Natural Language Processing (NLP) techniques. However, a major challenge is that code is written using an open, rapidly changing vocabulary due to, e.g., the coinage of new variable and method names. Reasoning over such a vocabulary i…

Scientific publications have evolved several features for mitigating vocabulary mismatch when indexing, retrieving, and computing similarity between articles. These mitigation strategies range from simply focusing on high-value article sections, such as titles and abstracts, to assigning keywords, often from controlled…

VaSST uses soft symbolic trees for probabilistic symbolic regression.

NIS learns optimal embedding sizes for recommendation models.

With a simple architecture and the ability to learn meaningful word embeddings efficiently from texts containing billions of words, word2vec remains one of the most popular neural language models used today. However, as only a single embedding is learned for every word in the vocabulary, the model fails to optimally re…

Enhanced word embedding creates new consumer-friendly health terms.

We describe dimensionally constrained symbolic regression which has been developed for mass measurement in certain classes of events in high-energy physics (HEP). With symbolic regression, we can derive equations that are well known in HEP. However, in problems with large number of variables, we find that by constraini…

In natural language processing, a lot of the tasks are successfully solved with recurrent neural networks, but such models have a huge number of parameters. The majority of these parameters are often concentrated in the embedding layer, which size grows proportionally to the vocabulary length. We propose a Bayesian spa…

Paper closes neural-symbolic learning loop with grammar model and back-search algorithm.

Reinforcement learning and symbolic planning have both been used to build intelligent autonomous agents. Reinforcement learning relies on learning from interactions with real world, which often requires an unfeasibly large amount of experience. Symbolic planning relies on manually crafted symbolic knowledge, which may …

For an arbitrary Riemannian manifold $X$ and Hermitian vector bundles $E$ and $F$ over $X$ we define the notion of the normal symbol of a pseudodifferential operator $P$ from $E$ to $F$. The normal symbol of $P$ is a certain smooth function from the cotangent bundle $T^*X$ to the homomorphism bundle $Hom (E,F)$ and dep…

Based on the ideas of Optimal Control, we introduce the new basic characteristic of a bracket generating distribution, the Jacobi symbol. In contrast to the classical Tanaka symbol, the set of Jacobi symbols is discrete and classifiable. We give an explicit and unified algebraic procedure for the construction of the ca…

The Softmax function is used in the final layer of nearly all existing sequence-to-sequence models for language generation. However, it is usually the slowest layer to compute which limits the vocabulary size to a subset of most frequent types; and it has a large memory footprint. We propose a general technique for rep…

We propose BlackOut, an approximation algorithm to efficiently train massive recurrent neural network language models (RNNLMs) with million word vocabularies. BlackOut is motivated by using a discriminative loss, and we describe a new sampling strategy which significantly reduces computation while improving stability, …

Improves slot key and value prediction for unseen entities.

Neural Machine Translation (MT) has reached state-of-the-art results. However, one of the main challenges that neural MT still faces is dealing with very large vocabularies and morphologically rich languages. In this paper, we propose a neural MT system using character-based embeddings in combination with convolutional…

Symbolic knowledge in neural models can inadvertently make them more vulnerable to adversarial attacks.

We introduce mod 3 triple Milnor invariants and triple cubic residue symbols for certain primes of the Eisenstein number field $\mathbb{Q}(\sqrt{-3})$, following the analogies between knots and primes. Our triple symbol generalizes both the cubic residue symbol and Rédei's triple symbol, and describes the decomposition…

A formula for Rademacher symbols in triangle groups is provided.

The symbolic dynamics technique is well-known for low-dimensional dynamical systems and chaotic maps, and lies at the roots of the thermodynamic formalism of dynamical systems. Here we show that this technique can also be successfully applied to time series generated by complex systems of much higher dimensionality. Ou…

Complex $b$-$6j$ symbols relate to hyperbolic tetrahedron volumes and determinants.

The paper classifies symbols of differential operators on vector bundles.

NeSS combines neural and symbolic approaches for better compositional generalization.