Bayesian technique compresses RNNs by 100x without tuning.
problem Large parameter size in RNNs, especially embeddings.
method Bayesian sparsification for RNNs and vocabulary filtering.
result RNNs can be compressed dozens or hundreds of times.
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,051 papers · 148 categories
Trend · papers per month
48 results for “vocabulary filtering”
A Bloom filter approach combined with Transformer models improves accuracy for machine learning tasks on opaque IDs.
problem Improving accuracy for machine learning tasks on opaque IDs with large vocabulary sizes.
method Applying hash functions to map opaque IDs to multiple hash tokens, similar to a Bloom filter, and using a multi-layer Transformer to process these digests.
result Models outperform those without hashing and sampled softmax, achieving high accuracy with a smaller computational budget.
Study finds optimal vocabulary size for neural machine translation.
problem Imbalanced class distribution in language data affects NMT performance.
method Casts NMT as a classification task, analyzes vocabulary sizes, and tests multiple languages.
result Certain vocabulary sizes outperform others, explaining NMT performance.
Paper reduces vocabulary losslessly for language model cooperation.
problem Language models struggle to cooperate with different tokenizations.
method Established a theoretical framework for lossless vocabulary reduction.
result Efficiently converts models with different tokenizations to cooperate with maximal common vocabulary.
The evolving of Data Science and the Saudi Arabia case. How much have we changed in 13 years?stat.AP
This study analyzes data science vocabulary changes over 13 years.
problem Understanding evolution of data science terms over time.
method Exploratory Data Analysis, Latent Semantic Analysis, Latent Dirichlet Analysis, N-grams Analysis.
result Identified new vocabulary and its incorporation into scientific literature.
Paper explores EEG-based speech recognition using transformers, showing faster training and better performance for smaller vocabularies.
problem Continuous speech recognition using EEG features.
method Transformer-based ASR model compared to RNN-based models.
result Transformer models perform better for smaller vocabularies but RNN models outperform them for larger vocabularies.
With the rise of social media, people can now form relationships and communities easily regardless of location, race, ethnicity, or gender. However, the power of social media simultaneously enables harmful online behavior such as harassment and bullying. Cyberbullying is a serious social problem, making it an important…
Despite significant progress in object categorization, in recent years, a number of important challenges remain, mainly, ability to learn from limited labeled data and ability to recognize object classes within large, potentially open, set of labels. Zero-shot learning is one way of addressing these challenges, but it …
Paper shows continuous speech recognition with EEG features, no speech input.
problem Continuous speech recognition with limited vocabulary and noisy/no speech input.
method Connectionist temporal classification (CTC) model, EEG features, new deep learning architecture.
result Continuous speech recognition achieved on limited vocabulary with noisy/no speech input.
New analysis shows optimal embedding learning rate depends on vocabulary size, not just model width.
problem Optimal learning rate for language model embeddings is not well understood, especially with large vocabularies.
method Theoretical analysis of training dynamics, interpolation between P and LV regimes.
result Optimal embedding learning rate scales as in the LV regime, not as P predicts.
ETM discovers interpretable topics in large vocabularies.
problem Existing topic models fail with large, heavy-tailed vocabularies.
method Generative model combining topic models and word embeddings with variational inference.
result ETM discovers interpretable topics even with large vocabularies.
Effective multilingual search with instance-based transfer learning.
problem Search in multilingual setting, especially next-sentence prediction and inverse cloze.
method Instance-based transfer learning, analyzing vocabulary overlaps and transitive overlaps.
result Positive transfer on all 35 target languages and two tasks, even with no vocabulary overlap.
Dynamic Vocabulary Pruning stabilizes LLM training by removing low-probability tokens.
problem Training Large Language Models (LLMs) with Reinforcement Learning (RL) causes numerical divergence between inference and training.
method Dynamic Vocabulary Pruning (DVP) constrains the RL objective to a safe vocabulary that excludes low-probability tokens.
result DVP stabilizes training by reducing systematic bias introduced by the extreme tail of the token distribution.
The novel unseen classes can be formulated as the extreme values of known classes. This inspired the recent works on open-set recognition \cite{Scheirer_2013_TPAMI,Scheirer_2014_TPAMIb,EVM}, which however can have no way of naming the novel unseen classes. To solve this problem, we propose the Extreme Value Learning (E…
ANT learns sparse embeddings for large vocabularies efficiently.
problem Lack of scalable methods for embedding large vocabularies in neural networks.
method Anchor & Transform (ANT) algorithm that learns a small set of anchor embeddings and a sparse transformation matrix.
result ANT achieves stronger performance with fewer parameters (up to 40x compression) compared to existing methods.
Framework learns portable representations for diverse tasks.
problem Creating task-independent abstract representations for diverse environments.
method Autonomously learns portable representations in egocentric space.
result Portable representations enable task-independent planning and transfer.
Training-free source selection for LLM families with shared vocabularies
problem Source selection for LLM families with shared vocabularies
method Fisher alignment at vocabulary scale
result Fisher alignment is a cosine between kernel mean embeddings in the joint activation-error space
In latent Dirichlet allocation (LDA), topics are multinomial distributions over the entire vocabulary. However, the vocabulary usually contains many words that are not relevant in forming the topics. We adopt a variable selection method widely used in statistical modeling as a dimension reduction tool and combine it wi…
Multilingual Topic Modelsstat.ML
Scientific publications have evolved several features for mitigating vocabulary mismatch when indexing, retrieving, and computing similarity between articles. These mitigation strategies range from simply focusing on high-value article sections, such as titles and abstracts, to assigning keywords, often from controlled…
A scalable algorithm compresses large categorical feature vocabularies while maximizing mutual information.
problem Learning from categorical features with very large vocabularies.
method Submodular optimization to maximize mutual information between compressed feature and target labels.
result The algorithm achieves a 63% approximation of the global optimal solution for mutual information.
NIS learns optimal embedding sizes for recommendation models.
problem Finding optimal embedding sizes for large-scale recommendation models.
method Neural Input Search (NIS) uses reinforcement learning to automatically find optimal vocabulary sizes and embedding dimensions.
result NIS improves prediction accuracy by 6.8% on Recall@1 and 1.8% on ROC-AUC.
Graph-Structured Cache improves code completion and variable naming tasks.
problem Learning open vocabulary in source code.
method Graph-Structured Cache for handling new words in code.
result Improves code completion and variable naming tasks by over 100%.
With a simple architecture and the ability to learn meaningful word embeddings efficiently from texts containing billions of words, word2vec remains one of the most popular neural language models used today. However, as only a single embedding is learned for every word in the vocabulary, the model fails to optimally re…
Enhanced word embedding creates new consumer-friendly health terms.
problem Laymen's health terms are often jargon and hard to understand.
method Developed an enhanced GloVe word embedding technique to generate new consumer-friendly terms.
result New CHV terms generated from consumer-generated text.
We propose BlackOut, an approximation algorithm to efficiently train massive recurrent neural network language models (RNNLMs) with million word vocabularies. BlackOut is motivated by using a discriminative loss, and we describe a new sampling strategy which significantly reduces computation while improving stability, …
Improves slot key and value prediction for unseen entities.
problem Dealing with unseen slot keys and values in real-world dialogue systems.
method Leverages external knowledge bases to project slots into an attribute space and generate candidate keys and values.
result Significant improvements in F1 score and accuracy (57.7% and 82.7%, respectively) over a previous approach.
Neural Machine Translation (MT) has reached state-of-the-art results. However, one of the main challenges that neural MT still faces is dealing with very large vocabularies and morphologically rich languages. In this paper, we propose a neural MT system using character-based embeddings in combination with convolutional…
Estimates unknown population sizes using the hypergeometric distribution.
problem Estimating discrete distributions with unknown population sizes and category sizes.
method Proposes a novel solution using the hypergeometric likelihood, accounting for a data generating process with a latent variable.
result Empirically demonstrates superior performance in estimating population sizes and learning latent spaces compared to other methods.
A new loss function speeds up sequence-to-sequence models for continuous outputs.
problem Slow and memory-intensive softmax layer limits vocabulary size and translation quality.
method Proposes a probabilistic loss and continuous embedding layer training/inference procedure.
result Models achieve up to 2.5x speed-up in training time with similar translation quality.
Traditional disease surveillance can be augmented with a wide variety of real-time sources such as, news and social media. However, these sources are in general unstructured and, construction of surveillance tools such as taxonomical correlations and trace mapping involves considerable human supervision. In this paper,…
EdgeSpeechNets improve speech recognition on mobile devices.
problem Deploying deep learning for speech recognition on edge devices is challenging.
method Human-machine collaboration for designing efficient DNN architectures.
result EdgeSpeechNets achieve higher accuracy with smaller network size and lower computational cost.
Model compression is essential for serving large deep neural nets on devices with limited resources or applications that require real-time responses. As a case study, a state-of-the-art neural language model usually consists of one or more recurrent layers sandwiched between an embedding layer used for representing inp…
Improves E2E ASR performance on numeric sequences with additional training data and denormalization.
problem Challenges in recognizing numeric sequences out-of-vocabulary in ASR systems.
method Uses text-to-speech for additional numeric training data and a small-footprint neural network for denormalization.
result Reduction of WER by up to a factor of 8 in the longest numeric sequences.
Team QCRI-MIT detects hyperpartisan news with 72.9% accuracy.
problem Detecting hyperpartisan news from biased political content.
method Logistic regression model using engineered features from propaganda detection.
result Significant performance improvements with better feature pre-processing.
Automatically classifying the tissues types of Region of Interest (ROI) in medical imaging has been an important application in Computer-Aided Diagnosis (CAD), such as classification of breast parenchymal tissue in the mammogram, classify lung disease patterns in High-Resolution Computed Tomography (HRCT) etc. Recently…
Most modern neural machine translation (NMT) systems rely on presegmented inputs. Segmentation granularity importantly determines the input and output sequence lengths, hence the modeling depth, and source and target vocabularies, which in turn determine model size, computational costs of softmax normalization, and han…
Paper introduces TrufLL for language model training without labeled data.
problem Training language models from scratch with reinforcement learning.
method Dynamic vocabulary truncation in RL for language generation.
result TrufLL successfully learns language generation policies from scratch.
Neural machine translation aims at building a single large neural network that can be trained to maximize translation performance. The encoder-decoder architecture with an attention mechanism achieves a translation performance comparable to the existing state-of-the-art phrase-based systems on the task of English-to-Fr…
Combines BERT and GCN for better text classification.
problem Limited global information capture by BERT.
method Integrates BERT with VGCN for improved text classification.
result VGCN-BERT outperforms BERT and GCN alone.
Long Short-Term Memory (LSTM) is a recurrent neural network (RNN) architecture that has been designed to address the vanishing and exploding gradient problems of conventional RNNs. Unlike feedforward neural networks, RNNs have cyclic connections making them powerful for modeling sequences. They have been successfully u…
Unified framework for discrete diffusion modeling with flexible noising processes.
problem Efficient modeling of large discrete state spaces with arbitrary corruption dynamics.
method Generalized Discrete Diffusion from Snapshots (GDDS) framework that supports uniformization for fast noising and snapshot-based ELBO for reverse process.
result GDDS outperforms existing discrete diffusion methods in training efficiency and generation quality.
The restricted Boltzmann machine (RBM) is a flexible tool for modeling complex data, however there have been significant computational difficulties in using RBMs to model high-dimensional multinomial observations. In natural language processing applications, words are naturally modeled by K-ary discrete distributions, …
New neural network layer handles OOV words in NLP tasks without pre-training.
problem Handling out-of-vocabulary words in natural language processing.
method Contextual-compositional neural network layer that attends to character sequence and context.
result Improves performance on 23 languages in joint tagging tasks.
Current recommender systems exploit user and item similarities by collaborative filtering. Some advanced methods also consider the temporal evolution of item ratings as a global background process. However, all prior methods disregard the individual evolution of a user's experience level and how this is expressed in th…
Convolutional neural network summarizes code comments across multiple languages.
problem Insufficient or missing comments in source code.
method Language-agnostic encoder-decoder model with open vocabulary.
result Comparable results to state-of-the-art on single-language data; first results on multi-language data.
More than 50 years ago Bongard introduced 100 visual concept learning problems as a testbed for intelligent vision systems. These problems are now known as Bongard problems. Although they are well known in the cognitive science and AI communities only moderate progress has been made towards building systems that can so…
Unlocking Noisy Real-World Corpora for Foundation Model Pre-Training via Quality-Aware Tokenizationcs.AI
QA-Token improves tokenization for noisy data, boosting model performance.
problem Tokenization ignores data quality, limiting model effectiveness on noisy corpora.
method QA-Token combines signal quality with vocabulary construction through bilevel optimization and reinforcement learning.
result QA-Token achieves state-of-the-art performance on genomic and financial datasets.
We study how to leverage off-the-shelf visual and linguistic data to cope with out-of-vocabulary answers in visual question answering task. Existing large-scale visual datasets with annotations such as image class labels, bounding boxes and region descriptions are good sources for learning rich and diverse visual conce…