arXiv research

Dynamic Vocabulary Pruning stabilizes LLM training by removing low-probability tokens.

New analysis shows optimal embedding learning rate depends on vocabulary size, not just model width.

Dynamic pruning during training reduces deep network complexity without significant accuracy loss.

New pruning methods improve dynamic sparse training performance.

Dynamic sample pruning speeds up spatio-temporal forecasting models.

Model compression is essential for serving large deep neural nets on devices with limited resources or applications that require real-time responses. As a case study, a state-of-the-art neural language model usually consists of one or more recurrent layers sandwiched between an embedding layer used for representing inp…

Proposes dynamic channel pruning during neural network training.

NTK-SAP improves neural network pruning by aligning training dynamics.

A new framework explains why early pruning works well.

Study examines effects of pruning techniques on deep learning models.

Dynamic Sparse Training finds efficient sparse networks from scratch.

New method grows deep networks efficiently by dynamically pruning and growing layers.

PAC-MCTS addresses biased search in LLM-guided planning by dynamically pruning.

Dynamic model pruning improves performance on deep neural networks without retraining.

To deal with various datasets over different complexity, this paper presents an self-adaptive learning model that combines the proposed Dynamic Connected Neural Decision Networks (DNDN) and a new pruning method--Dynamic Soft Pruning (DSP). DNDN is a combination of random forests and deep neural networks that enjoys bot…

Deep neural networks (DNNs) although achieving human-level performance in many domains, have very large model size that hinders their broader applications on edge computing devices. Extensive research work have been conducted on DNN model compression or pruning. However, most of the previous work took heuristic approac…

Geometric pruning rules improve change point detection in multiple time series.

Study finds optimal vocabulary size for neural machine translation.

This paper introduces blind adversarial pruning to balance accuracy, efficiency, and robustness in neural networks.

Paper reduces vocabulary losslessly for language model cooperation.

Unified framework for accelerating DNNs on resource-limited platforms.

This study analyzes data science vocabulary changes over 13 years.

We introduce Delay Pruning, a simple yet powerful technique to regularize dynamic Boltzmann machines (DyBM). The recently introduced DyBM provides a particularly structured Boltzmann machine, as a generative model of a multi-dimensional time-series. This Boltzmann machine can have infinitely many layers of units but al…

Optimizes pruning masks for neural networks using probabilistic fine-tuning and PAC-Bayes bounds.

DASH simplifies neural networks for gene regulatory dynamics using domain knowledge.

To improve the execution speed and efficiency of neural networks in embedded systems, it is crucial to decrease the model size and computational complexity. In addition to conventional compression techniques, e.g., weight pruning and quantization, removing unimportant activations can reduce the amount of data communica…

With the rise of social media, people can now form relationships and communities easily regardless of location, race, ethnicity, or gender. However, the power of social media simultaneously enables harmful online behavior such as harassment and bullying. Cyberbullying is a serious social problem, making it an important…

Despite significant progress in object categorization, in recent years, a number of important challenges remain, mainly, ability to learn from limited labeled data and ability to recognize object classes within large, potentially open, set of labels. Zero-shot learning is one way of addressing these challenges, but it …

In this paper we investigate continuous speech recognition using electroencephalography (EEG) features using recently introduced end-to-end transformer based automatic speech recognition (ASR) model. Our results demonstrate that transformer based model demonstrate faster training compared to recurrent neural network (R…

Most modern neural machine translation (NMT) systems rely on presegmented inputs. Segmentation granularity importantly determines the input and output sequence lengths, hence the modeling depth, and source and target vocabularies, which in turn determine model size, computational costs of softmax normalization, and han…

In this work, dynamic Bayesian multinets are introduced where a Markov chain state at time t determines conditional independence patterns between random variables lying within a local time window surrounding t. It is shown how information-theoretic criterion functions can be used to induce sparse, discriminative, and c…

Paper introduces TrufLL for language model training without labeled data.

Survey on reducing deep neural network training complexity.

The novel unseen classes can be formulated as the extreme values of known classes. This inspired the recent works on open-set recognition \cite{Scheirer_2013_TPAMI,Scheirer_2014_TPAMIb,EVM}, which however can have no way of naming the novel unseen classes. To solve this problem, we propose the Extreme Value Learning (E…

In this paper we demonstrate continuous noisy speech recognition using connectionist temporal classification (CTC) model on limited Chinese vocabulary using electroencephalography (EEG) features with no speech signal as input and we further demonstrate single CTC model based continuous noisy speech recognition on limit…

Unified framework for discrete diffusion modeling with flexible noising processes.

As a result of the growing size of Deep Neural Networks (DNNs), the gap to hardware capabilities in terms of memory and compute increases. To effectively compress DNNs, quantization and connection pruning are usually considered. However, unconstrained pruning usually leads to unstructured parallelism, which maps poorly…

Sparsification is an efficient approach to accelerate CNN inference, but it is challenging to take advantage of sparsity in training procedure because the involved gradients are dynamically changed. Actually, an important observation shows that most of the activation gradients in back-propagation are very close to zero…

Lookahead pruning extends single-layer optimization to multi-layer, outperforming magnitude-based pruning.

We focus on the problem of search in the multilingual setting. Examining the problems of next-sentence prediction and inverse cloze, we show that at large scale, instance-based transfer learning is surprisingly effective in the multilingual setting, leading to positive transfer on all of the 35 target languages and two…

Recent pruning methods at initialization fall short of random pruning's accuracy.

ANT learns sparse embeddings for large vocabularies efficiently.

ICE-Pruning accelerates deep neural network pruning by 9.61x.

DSA efficiently allocates sparsity across layers for budgeted pruning.

Training-free source selection for LLM families with shared vocabularies

Pruning neural network parameters is often viewed as a means to compress models, but pruning has also been motivated by the desire to prevent overfitting. This motivation is particularly relevant given the perhaps surprising observation that a wide variety of pruning approaches increase test accuracy despite sometimes …

The study reveals flaws in pruning criteria and proposes a new assumption for better filter selection.

Deep neural networks have dramatically achieved great success on a variety of challenging tasks. However, most successful DNNs have an extremely complex structure, leading to extensive research on model compression.As a significant area of progress in model compression, traditional gradual pruning approaches involve an…