Faster WIND accelerates iterative BOND for LLM alignment.
problem Iterative BOND is inefficient in practice due to sample and computation inefficiency.
method Unified game-theoretic connection to self-play alignment, WIND framework with efficient algorithms.
result WIND variant achieves superior sample efficiency and faster computation.
Improved inference-time alignment using Best-of-N and smoothing.
problem Reward overoptimization in Best-of-N (BoN) due to poor proxy reward models.
method Introduced Soft Best-of-N (SBoN) and analyzed its performance through KL divergence and regret analysis.
result Smoothing helps SBoN mitigate reward overoptimization, especially when proxy reward quality is low.
Best-of-N sampling reveals reward targets from preference data, influencing N and base distribution choices.
problem Understanding reward extraction from Best-of-N preference data and optimal N and base distribution choices.
method Specialized analysis of preference data via induced conditional distribution, deriving reward targets and design principles.
result Reward targets are explicit functions of N and base distribution, and bounded-class minimizers approach these targets as N grows.
Two methods for model adaptation compared; fine-tuning outperforms Best-of-N in realizable settings.
problem Comparing methods for adapting large language models to new tasks.
method Supervised fine-tuning vs. Best-of-N approach.
result Supervised fine-tuning outperforms Best-of-N in realizable settings.
GSI improves efficiency of large language model inference.
problem Efficiently guiding test-time alignment in large language models.
method Combines soft best-of-n scaling with a reward model and speculative samples. result Achieves higher accuracy and reduced latency compared to standard methods.
New method optimizes language model performance for test-time strategies.
problem Mismatch between training objectives and test-time deployment of large language models.
method Tail-Extrapolated estimators to approximate best-of-N performance from limited training rollouts.
result Improved performance of best-of-N deployment across various models and datasets.
SWIFT learns intrinsic rewards from LLM hidden states for efficient best-of-N sampling.
problem Efficiency and scalability of reward models for LLMs.
method SWIFT (Simple Weighted Intrinsic Feedback Technique) learns a reward function directly from LLM hidden states.
result SWIFT outperforms existing baselines by 12.7% on MATH dataset while using less than 0.005% of their parameters.
Improved selection of best outputs from LLMs for better accuracy.
problem BoN fails to reliably find the correct answer with imperfect rewards.
method Majority-of-the-Bests (MoB) selects the mode of a bootstrapped output distribution of BoN.
result MoB consistently improves over BoN in 25 out of 30 setups.
The paper analyzes the reward improvement of aligned policies in large language models.
problem Optimizing policies in large language models while staying close to a reference policy.
method Information-theoretic analysis and reduction to exponential order statistics.
result Information-theoretic upper bounds on reward improvement are derived.
New algorithm improves inference-time alignment without reward hacking.
problem Improving quality of responses from language models with limited compute.
method Inference-time alignment, focusing on extttInferenceTimePessimism algorithm. result Optimal performance and scaling-monotonicity of extttInferenceTimePessimism. The study examines how verifier imperfections impact test-time scaling techniques.
problem Understanding how verifier imperfections affect test-time scaling methods.
method Proves the instance-level accuracy of Best-of-N and Rejection Sampling methods using the geometry of the verifier's ROC curve.
result RS outperforms BoN for fixed compute, but both converge to the same accuracy in the infinite-compute limit.
Recent variants improve knowledge distillation performance.
problem Improving the performance of knowledge distillation.
method Introducing additional components or changing the learning process.
result These variants have shown promising results.
Paper characterizes optimal language model alignment methods.
problem Aligning language models to maximize reward while keeping them close to the original model.
method KL-constrained reinforcement learning and best-of-N methods.
result Optimal KL-constrained RL solution has a large deviation principle rate function.
This work explores test-time scaling strategies for LLMs, improving sample efficiency and expressiveness.
problem Understanding the sample efficiency and expressiveness of test-time scaling strategies for LLMs.
method Established separation and expressiveness results for self-consistency, best-of-n, and self-correction strategies. result Self-correction enables Transformers to simulate online learning over multiple tasks without prior knowledge.
This study improves knowledge distillation for RNN-T models with noisy labels.
problem Challenges in distilling knowledge from RNN-T models with variable quality teachers.
method Full-sum distillation and sequence-level knowledge distillation.
result Full-sum distillation outperforms other methods for RNN-T models, especially for bad teachers.
Improved dataset distillation for images and texts boosts model accuracy.
problem Reducing dataset size for faster and more energy-efficient model training.
method Simultaneous distillation of images and soft labels, extending to text datasets.
result 2-4% increase in accuracy for image classification tasks, 20% reduction in distilled samples.
Labels distilled from images improve model training efficiency and flexibility.
problem Creating synthetic labels for a small set of real images to train models effectively.
method Introduce a more robust and flexible meta-learning algorithm for distillation and an effective first-order strategy based on convex optimization layers.
result Label distillation leads to improved results and greater flexibility in neural architectures.
New method distills discrete diffusion models, maintaining quality and diversity.
problem Difficult to distill discrete diffusion models.
method Discrete Moment Matching Distillation (D-MMD)
result Maintains high quality and diversity in distilled models.
Estimates model performance from compute budget for distillation.
problem Risk mitigation in large-scale distillation.
method Distillation scaling law based on compute budget allocation.
result Maximizes student performance with compute-optimal allocation.
Real-time policy distillation speeds up and improves reinforcement learning.
problem Slow and inefficient policy distillation in reinforcement learning.
method Simultaneous training and distillation of a teacher and student model.
result Significantly reduced distillation time and improved small model performance.
Distillation affects some classes more than others, impacting fairness and bias.
problem Distillation affects some classes more than others, impacting fairness and bias.
method Examined class-wise accuracy and fairness metrics (DPD, EOD) on models trained with different datasets.
result Increasing the distillation temperature improves the distilled student model's fairness and individual fairness.
Survey of knowledge distillation for resource-limited devices.
problem Deploying large deep learning models on resource-limited devices.
method Knowledge distillation using a smaller model trained with information from a larger model.
result A new metric (distillation metric) for comparing different knowledge distillation algorithms.
Study on ensemble, distillation, and self-distillation in deep learning models.
problem Improving test accuracy in deep learning models using ensemble and distillation methods.
method Formal study of ensemble and distillation, considering multi-view data structure.
result Proven that ensemble and distillation can improve test accuracy in deep learning models, and the superior performance can be distilled into a single model.
A new distillation method transfers channel information from teacher to student.
problem Transfer knowledge from teacher to student with fewer parameters and calculations.
method Channel Distillation (CD) and Guided Knowledge Distillation (GKD) with loss decay.
result Achieved 27.68% top-1 error on ImageNet with ResNet18, outperforming state-of-the-art methods.
Knowledge distillation is effective for producing small, high-performance neural networks for classification, but these small networks are vulnerable to adversarial attacks. This paper studies how adversarial robustness transfers from teacher to student during knowledge distillation. We find that a large amount of robu…
Repeated self-distillation improves model performance significantly.
problem How much gain is possible by applying multiple steps of self-distillation?
method Investigated linear regression tasks, applied multiple steps of self-distillation, analyzed excess risk reduction.
result Multi-step self-distillation reduces excess risk by a factor as large as d, where d is the input dimension. OKDDip uses diverse peers to improve online knowledge distillation.
problem Early saturation in group-based distillation.
method Two-level distillation with multiple auxiliary peers and a group leader, using attention-based aggregation weights.
result OKDDip consistently gives better performance than state-of-the-art approaches.
RID framework quantifies and regularizes task-relevant knowledge in distillation.
problem Distilling irrelevant information can hinder student model performance.
method Partial Information Decomposition to quantify and regularize task-relevant knowledge.
result RID framework leads to more resilient distillation under nuisance teachers.
Distiller simplifies DNN compression research with a Python package.
problem Efficiently compressing deep neural networks.
method Open-source Python package with DNN compression algorithms.
result Facilitates new research and learning tasks in DNN compression.
A new method for training GNNs without a teacher model.
problem Training over-parameterized GNN models is difficult and inefficient.
method GNN Self-Distillation (GNN-SD) with NDR and ADR.
result Improves GNN performance with less training cost and better generalization.
FasterVoiceGrad speeds up VC by 6-7x with novel distillation.
problem Slow iterative sampling in diffusion-based VC models.
method Adversarial diffusion conversion distillation (ADCD) to create a faster one-step model.
result 6.6-6.9 and 1.8x faster on GPU and CPU, respectively.
This paper improves speech recognition by distilling knowledge from acoustic models.
problem Improving speech recognition accuracy using ensemble models.
method Proposes multi-teacher distillation strategies for joint CTC-attention end-to-end ASR systems, integrating error rate metric for optimization.
result Reports state-of-the-art error rates on various datasets and languages.
A framework preserves uncertainty in ensemble distillation.
problem Preserving uncertainty decomposition in ensemble distillation.
method General framework for distilling both regression and classification ensembles, preserving natural uncertainty decomposition.
result Framework maintains decomposition of predictive uncertainty.
Novel framework learns efficient student models from teacher networks.
problem Model capacity gap between teacher and student networks.
method Neural architecture search and oracle knowledge distillation.
result Searched student models often outperform teacher models.
CAKD framework optimizes knowledge transfer by focusing on influential components of distillation.
problem Balancing and optimizing knowledge transfer in distillation models.
method Decouple KL divergence into BCD, SCD, and WCD; prioritize influential components.
result CAKD framework consistently outperforms baseline across diverse models and datasets.
New method predicts and optimizes test-time scaling for LLMs.
problem Lack of principled guidance on scaling LLMs efficiently.
method Tail-guided search to predict and allocate compute.
result SLG Search achieves higher rewards with less compute.
Consistency distillation reduces memorization in diffusion models without harming sample quality.
problem Understanding how distillation affects memorization in diffusion models.
method Analysis of consistency distillation in diffusion models using a random feature neural network model.
result Consistency distillation reduces memorization in diffusion models without harming sample quality.
Self-distillation improves model performance but can lead to underfitting.
problem Understanding why self-distillation improves model performance and its limitations.
method Theoretical analysis of self-distillation in Hilbert space with ℓ2 regularization. result Self-distillation modifies regularization by limiting the number of basis functions, potentially leading to underfitting.
A new method distills datasets more efficiently and effectively.
problem Achieving competitive performance on test data with a small synthetic dataset.
method Tackles dataset distillation as a bilevel optimization problem, introduces RaT-BPTT to stabilize gradients and speed up optimization.
result Establishes new state-of-the-art performance across various benchmarks.
The transfer of knowledge from one policy to another is an important tool in Deep Reinforcement Learning. This process, referred to as distillation, has been used to great success, for example, by enhancing the optimisation of agents, leading to stronger performance faster, on harder domains [26, 32, 5, 8]. Despite the…
Improved generalization with iterative self-distillation using weighted ground-truth targets.
problem Improving generalization accuracy in neural networks.
method Iterative kernel regression with weighted ground-truth targets and ℓ2 regularization. result Closed-form solution for optimal weighting parameter and efficient estimation.
The paper explains knowledge distillation by analyzing visual concepts in DNNs.
problem Understanding how knowledge distillation affects the learning of visual concepts in deep neural networks.
method The paper proposes three hypotheses and designs mathematical metrics to evaluate feature representations of DNNs.
result The hypotheses were verified through experiments on various DNNs.
This work proposes splitting deep neural networks into smaller sub-networks for faster and more efficient distillation.
problem Challenges in training deep neural networks, including local optima, gradient issues, and computational demands.
method Proposes a non-end-to-end distillation approach by splitting networks into smaller, independent sub-networks (neighbourhoods).
result Independent training of smaller sub-networks can speed up distillation and improve efficiency in various applications.
SiD distills pretrained diffusion models into a fast one-step generator.
problem Efficiently distilling pretrained diffusion models into a fast generator.
method Reformulates forward diffusion processes as semi-implicit distributions and uses three score-related identities to create a loss mechanism.
result Achieves high FID performance and significantly reduces generation time.
Compressed Federated Distillation reduces communication in federated learning.
problem Communication constraints in Federated Learning.
method Compressed Federated Distillation (CFD) leverages soft labels and quantization techniques.
result Reduces communication by more than 4 orders of magnitude compared to Federated Averaging.
Extracting a curriculum from a teacher network improves distillation efficiency.
problem Efficiently training a small network using a large teacher network's output.
method Random projection of teacher network's hidden representations to progressively train the student network.
result Extracted curriculum significantly outperforms one-shot distillation and achieves similar performance to progressive distillation.
New method improves knowledge transfer from large to small networks.
problem Improving knowledge transfer from large to small neural networks.
method Contrastive learning to capture more structural knowledge.
result Contrastive learning outperforms knowledge distillation on various transfer tasks.
Improved self-distillation reduces label noise and enhances model accuracy.
problem Label noise in multi-class classification.
method Label averaging and refined partial labels.
result Single-round self-distillation achieves comparable performance to multi-round distillation.