arXiv research

SpanishTinyRoBERTa distills large Spanish models into efficient question-answering models.

Paper improves natural language understanding with less data using a new training method.

SFAVEL distills features from language models for fact verification without annotations.

Self-distillation improves constrained language generation by aligning models with target distributions.

Survey examines distillation methods for large language models.

The paper establishes theoretical foundations for low-rank knowledge distillation in LLMs.

This paper investigates teacher hacking during language model distillation and proposes methods to mitigate it.

Since deep learning became a key player in natural language processing (NLP), many deep learning models have been showing remarkable performances in a variety of NLP tasks, and in some cases, they are even outperforming humans. Such high performance can be explained by efficient knowledge representation of deep learnin…

The paper studies how search and distillation improve reasoning in large language models.

The paper distills a weighted automaton from RNNs for language modeling.

DLM-One speeds up language generation by 500x with continuous models.

Recent advances in pre-training huge models on large amounts of text through self supervision have obtained state-of-the-art results in various natural language processing tasks. However, these huge and expensive models are difficult to use in practise for downstream tasks. Some recent efforts use knowledge distillatio…

Paper proposes a method to speed up discrete diffusion models by distilling many steps into few.

This paper improves speech recognition by distilling knowledge from acoustic models.

MixKD improves large-scale language model compression and generalization.

Compressed Federated Distillation reduces communication in federated learning.

Enhances model compression with multi-teacher knowledge distillation.

Extracting a curriculum from a teacher network improves distillation efficiency.

DOSFL reduces federated learning communication by one round, preserving model performance.

Techniques such as ensembling and distillation promise model quality improvements when paired with almost any base model. However, due to increased test-time cost (for ensembles) and increased complexity of the training pipeline (for distillation), these techniques are challenging to use in industrial settings. In this…

Deep Neural Networks have achieved huge success at a wide spectrum of applications from language modeling, computer vision to speech recognition. However, nowadays, good performance alone is not sufficient to satisfy the needs of practical deployment where interpretability is demanded for cases involving ethics and mis…

Significant advances have been made in Natural Language Processing (NLP) modelling since the beginning of 2018. The new approaches allow for accurate results, even when there is little labelled data, because these NLP models can benefit from training on both task-agnostic and task-specific unlabelled data. However, the…

This paper shows ARMs and EBMs are equivalent, revealing ARM lookahead capabilities.

New findings explain why online methods outperform offline methods in noisy expert feedback settings.

Logit distance bounds representational similarity of models.

Faster WIND accelerates iterative BOND for LLM alignment.

Overparameterized transformer networks have obtained state of the art results in various natural language processing tasks, such as machine translation, language modeling, and question answering. These models contain hundreds of millions of parameters, necessitating a large amount of computation and making them prone t…

CoDistill-GRPO improves small models in GRPO by distilling knowledge from a larger model.

Study bandwidth-limited training and inference of language models.

Adversarial examples are important for understanding the behavior of neural models, and can improve their robustness through adversarial training. Recent work in natural language processing generated adversarial examples by assuming white-box access to the attacked model, and optimizing the input directly against it (E…

Recent state-of-the-art natural language understanding models, such as BERT and XLNet, score a pair of sentences (A and B) using multiple cross-attention operations - a process in which each word in sentence A attends to all words in sentence B and vice versa. As a result, computing the similarity between a query sente…

This study improves knowledge distillation for RNN-T models with noisy labels.

Extensive research has recently shown that recurrent neural language models are able to process a wide range of grammatical phenomena. How these models are able to perform these remarkable feats so well, however, is still an open question. To gain more insight into what information LSTMs base their decisions on, we pro…

Study uses LLMs to simplify financial regulation interpretation.

Recent variants improve knowledge distillation performance.

Knowledge Distillation (KD) consists of transferring “knowledge” from one machine learning model (the teacher) to another (the student). Commonly, the teacher is a high-capacity model with formidable performance, while the student is more compact. By transferring knowledge, one hopes to benefit from the student’s…

Survey of knowledge distillation for resource-limited devices.

New method distills discrete diffusion models, maintaining quality and diversity.

Distillation affects some classes more than others, impacting fairness and bias.

Real-time policy distillation speeds up and improves reinforcement learning.

Labels distilled from images improve model training efficiency and flexibility.

Dataset distillation is a method for reducing dataset sizes by learning a small number of synthetic samples containing all the information of a large dataset. This has several benefits like speeding up model training, reducing energy consumption, and reducing required storage space. Currently, each synthetic sample is …

Estimates model performance from compute budget for distillation.

Repeated self-distillation improves model performance significantly.

Paper tackles treatment leakage in text-based causal inference, proposing methods to mitigate bias.

Study on ensemble, distillation, and self-distillation in deep learning models.

Consistency distillation reduces memorization in diffusion models without harming sample quality.

RID framework quantifies and regularizes task-relevant knowledge in distillation.