In this work we aim to obtain computationally-efficient uncertainty estimates with deep networks. For this, we propose a modified knowledge distillation procedure that achieves state-of-the-art uncertainty estimates both for in and out-of-distribution samples. Our contributions include a) demonstrating and adapting to …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,694 papers · 148 categories
Trend · papers per month
48 results for “Distribution Distillation”
Two approaches extend knowledge distillation to Gaussian Processes, showing relationships to existing methods.
problem Applying knowledge distillation to Gaussian Processes for regression and classification.
method Data-centric and distribution-centric approaches to extend distillation to GPR and GPC.
result Distribution-centric approach for GPC approximately corresponds to data duplication and scaling.
This paper distills Bayesian posterior expectations for deep neural networks.
problem Improving deep neural network performance and uncertainty quantification.
method Develops a framework for distilling expectations from Bayesian posterior distributions using Monte Carlo samples.
result The framework successfully distills posterior predictive distribution and expected entropy.
EM Distillation simplifies diffusion models to one-step generators.
problem Efficient sampling from complex diffusion models with minimal loss of quality.
method EM Distillation, a maximum likelihood approach based on Expectation-Maximization.
result EM Distillation outperforms existing one-step generative methods in FID scores.
KDM unifies feature matching in neural networks.
problem Matching neural network activations for knowledge transfer.
method Formalizes distribution matching strategy.
result New theoretical results for KD.
Distributional Correlation--Aware Knowledge Distillation for Stock Trading Volume Predictionq-fin.TR
A new distillation framework predicts stock trading volumes more accurately with less model size.
problem Predicting stock trading volumes using regression models without class correlations.
method Transformed regression model into a probabilistic forecasting model, matching distributions and correlational relationships.
result Framework achieves superior prediction accuracy with significantly smaller model size.
Distilled models often fail to match teacher models, despite improving generalization.
problem The discrepancy between teacher and student predictive distributions remains large.
method Investigated the optimization difficulties and dataset details affecting student performance.
result Optimizing for matching the teacher does not always lead to better generalization.
This study improves knowledge distillation for RNN-T models with noisy labels.
problem Challenges in distilling knowledge from RNN-T models with variable quality teachers.
method Full-sum distillation and sequence-level knowledge distillation.
result Full-sum distillation outperforms other methods for RNN-T models, especially for bad teachers.
Distillation is an effective knowledge-transfer technique that uses predicted distributions of a powerful teacher model as soft targets to train a less-parameterized student model. A pre-trained high capacity teacher, however, is not always available. Recently proposed online variants use the aggregated intermediate pr…
S2D efficiently trains models to estimate uncertainty without increasing resource costs.
problem Efficiently estimating uncertainty in deep learning models for safety-critical applications.
method Self-distribution distillation (S2D) approach to train a single model for uncertainty estimation.
result S2D models outperform standard models and Monte-Carlo dropout in uncertainty estimation.
Dataset distillation is a method for reducing dataset sizes by learning a small number of synthetic samples containing all the information of a large dataset. This has several benefits like speeding up model training, reducing energy consumption, and reducing required storage space. Currently, each synthetic sample is …
Distilling Model Knowledgestat.ML
Top-performing machine learning systems, such as deep neural networks, large ensembles and complex probabilistic graphical models, can be expensive to store, slow to evaluate and hard to integrate into larger systems. Ideally, we would like to replace such cumbersome models with simpler models that perform equally well…
Ensembles of models often yield improvements in system performance. These ensemble approaches have also been empirically shown to yield robust measures of uncertainty, and are capable of distinguishing between different \emph{forms} of uncertainty. However, ensembles come at a computational and memory cost which may be…
Extracurricular learning closes the accuracy gap in knowledge distillation.
problem Accuracy gap between teacher and student models after knowledge distillation.
method Modeling student and teacher output distributions, sampling from an extended data distribution, and matching over this set.
result Extracurricular learning reduces the accuracy gap by 46% to 68%.
A framework preserves uncertainty in ensemble distillation.
problem Preserving uncertainty decomposition in ensemble distillation.
method General framework for distilling both regression and classification ensembles, preserving natural uncertainty decomposition.
result Framework maintains decomposition of predictive uncertainty.
New method regularizes deep networks by distilling self-knowledge.
problem Overfitting in deep neural networks.
method Self-knowledge distillation to regularize class-wise predictions.
result Significant improvement in generalization and calibration.
Proposes LsrKD and MrKD to improve neural network training performance.
problem Improving neural network training performance, especially on deep networks.
method Extends Label Smoothing Regularization with Self-Knowledge Distillation, introducing LsrKD and MrKD.
result LsrKD and MrKD significantly improve training performance on deep neural networks.
This paper investigates teacher hacking during language model distillation and proposes methods to mitigate it.
problem Teacher hacking during language model distillation, leading to suboptimal performance.
method A controlled experimental setup involving an oracle LM, teacher LM, and student LM, using fixed offline or online data generation techniques.
result Data diversity is the key factor in preventing teacher hacking during distillation.
A new distillation method transfers channel information from teacher to student.
problem Transfer knowledge from teacher to student with fewer parameters and calculations.
method Channel Distillation (CD) and Guided Knowledge Distillation (GKD) with loss decay.
result Achieved 27.68% top-1 error on ImageNet with ResNet18, outperforming state-of-the-art methods.
Distillation improves simple models by approximating complex labels.
problem Why does distillation improve simple models?
method Statistical perspective on distillation, connecting to extreme multiclass retrieval.
result Distillation helps by approximating underlying class-probabilities, reducing bias and variance.
Self-distillation improves constrained language generation by aligning models with target distributions.
problem Sparse and uninformative reward signals in constrained generation settings.
method Iteratively refining the base model through self-distillation, incorporating learned twist functions and proposals.
result Substantial gains in generation quality through improved model alignment with target distributions.
New method improves ensemble inference for high-class tasks.
problem High inference costs for ensemble models.
method Proxy-Dirichlet target to minimize reverse KL-divergence.
result Resolves gradient issues for large-scale classification tasks.
New method distills cloud models into edge-friendly ones.
problem Cloud-to-edge model compression with limited data exchange.
method Two-step workflow of deprivatization and distillation.
result Outperforms previous state-of-the-art approaches on various benchmarks.
Dataset Distillationcs.LG
Model distillation aims to distill the knowledge of a complex model into a simpler one. In this paper, we consider an alternative formulation called dataset distillation: we keep the model fixed and instead attempt to distill the knowledge from a large training dataset into a small one. The idea is to synthesize a smal…
TL-ANDI distills context from source data to improve transfer learning for TFMs.
problem Limited transfer learning due to context-size constraints and distribution shifts.
method TL-ANDI uses posterior-aware distillation to construct a compact source context and locally distills labels.
result Improves transfer performance by addressing context-size and distribution shifts.
SiD distills pretrained diffusion models into a fast one-step generator.
problem Efficiently distilling pretrained diffusion models into a fast generator.
method Reformulates forward diffusion processes as semi-implicit distributions and uses three score-related identities to create a loss mechanism.
result Achieves high FID performance and significantly reduces generation time.
A new algorithm improves federated learning by combining knowledge distillation and weighted combination loss.
problem Non-IID client data in federated learning leads to model drift and poor generalization.
method pFedKD-WCL integrates knowledge distillation with bi-level optimization to address non-IID challenges.
result pFedKD-WCL outperforms state-of-the-art algorithms in accuracy and convergence speed.
Distillation speeds up classifier training and provides insights into its success.
problem Empirical success of knowledge distillation without theoretical explanation.
method Study of linear and deep linear classifiers, proving a generalization bound.
result Three key factors for distillation success: data geometry, optimization bias, strong monotonicity.
DOSFL reduces federated learning communication by one round, preserving model performance.
problem High communication costs in federated learning with poorly distributed data.
method Clients distill their private data into synthetic data, sending only this to the server for training.
result Total communication cost up to 3 orders of magnitude less than FedAvg while maintaining comparable performance.
Optimal quantization improves dataset distillation for faster training.
problem Efficiently train models with synthetic data.
method Reformulate disentangled methods as optimal quantization problems.
result Better performance and generalization on ImageNet-1K and subsets.
Regression Prior Networks improve ensemble performance on regression tasks.
problem Improving ensemble performance on regression tasks.
method Extending Prior Networks and Ensemble Distribution Distillation (EnD) to regression tasks using the Normal-Wishart distribution.
result Regression Prior Networks yield performance competitive with ensemble approaches on regression tasks.
New method for decentralized learning from diverse clients.
problem Achieving high-performance global model from diverse client models.
method Decentralized Learning via Adaptive Distillation (DLAD).
result Demonstrates effectiveness of DLAD on multiple public datasets.
SMT trains generative models by estimating mixture scores, outperforming existing methods.
problem Training one-step generative models efficiently and effectively.
method Score-of-Mixture Training (SMT) estimates the score of mixture distributions between real and fake samples.
result SMT/SMD outperform existing methods on CIFAR-10 and ImageNet 64x64 datasets.
Current approaches for Knowledge Distillation (KD) either directly use training data or sample from the training data distribution. In this paper, we demonstrate effectiveness of 'mismatched' unlabeled stimulus to perform KD for image classification networks. For illustration, we consider scenarios where this is a comp…
Knowledge distillation is an effective technique that transfers knowledge from a large teacher model to a shallow student. However, just like massive classification, large scale knowledge distillation also imposes heavy computational costs on training models of deep neural networks, as the softmax activations at the la…
FED distills ensemble predictions efficiently, improving accuracy and uncertainty.
problem Efficiently distilling ensemble predictions from limited data.
method Proposes Functional Ensemble Distillation (FED) using mixup augmentation.
result Significantly boosts performance in accuracy and uncertainty estimation.
Ensembles of models have been empirically shown to improve predictive performance and to yield robust measures of uncertainty. However, they are expensive in computation and memory. Therefore, recent research has focused on distilling ensembles into a single compact model, reducing the computational and memory burden o…
Bayesian neural networks (BNNs) allow us to reason about uncertainty in a principled way. Stochastic Gradient Langevin Dynamics (SGLD) enables efficient BNN learning by drawing samples from the BNN posterior using mini-batches. However, SGLD and its extensions require storage of many copies of the model parameters, a p…
Recent variants improve knowledge distillation performance.
problem Improving the performance of knowledge distillation.
method Introducing additional components or changing the learning process.
result These variants have shown promising results.
High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Lawsstat.ML
Sharp analysis of knowledge distillation for high-dimensional regression.
problem Characterizing the risk of target models in high-dimensional settings.
method Sharp non-asymptotic bounds for ridgeless regression under model and distribution shifts.
result Identifies optimal surrogate models and reveals benefits and limitations of discarding weak features.
Paper tackles adapting multiple domains to a target domain using distillation and dictionary learning.
problem Adapting multiple heterogeneous labeled source domains to an unlabeled target domain.
method Combines Multi-Source Domain Adaptation and Dataset Distillation with Dataset Dictionary Learning.
result Achieves state-of-the-art adaptation performance even with minimal labeled data.
Paper proposes a method to speed up discrete diffusion models by distilling many steps into few.
problem Challenges in capturing dependencies between elements in discrete diffusion models.
method Proposes 'mixture' models and loss functions to distill many sampling steps into few.
result Effective in distilling pretrained discrete diffusion models across image and language domains.
Knowledge distillation is a widely used technique for model compression. We posit that the teacher model used in a distillation setup, captures relationships between classes, that extend beyond the original dataset. We empirically show that a teacher model can transfer this knowledge to a student model even on an {\it …
ReOPD uses pre-collected teacher trajectories to distill knowledge from multi-turn interactions.
problem The cost of fully online on-policy distillation for multi-turn interactions.
method ReOPD, an off-environment alternative that reuses pre-collected teacher trajectories as replayed prefixes, addressing the prefix trap and distribution shift.
result ReOPD preserves or improves OPD-level accuracy, uses zero tool calls, and is at least 4 faster per training step.
We propose a novel Wasserstein method with a distillation mechanism, yielding joint learning of word embeddings and topics. The proposed method is based on the fact that the Euclidean distance between word embeddings may be employed as the underlying distance in the Wasserstein topic model. The word distributions of to…
Federated learning framework with uncertainty estimation.
problem Uncertainty estimation in federated learning models.
method Infers and distills posterior predictive distributions from clients to server.
result Improved accuracy and reliability in predictions and tasks like OOD detection.
Few-step distillation improves T2I models without real images or CFG trade-offs.
problem Challenges in accelerating T2I models with high-resolution and CFG.
method Score identity distillation (SiD) for few-step generation, with adversarial loss and new guidance strategies.
result State-of-the-art performance on SDXL at 1024x1024 resolution, robust to real images absence.
Real-Time Uncertainty Estimation in Computer Vision via Uncertainty-Aware Distribution Distillationcs.CV
Real-time uncertainty estimation for computer vision tasks.
problem Real-time inference of uncertainty in deep learning models.
method Uncertainty-Aware Distribution Distillation method for fast inference.
result Significantly reduced inference time with improved uncertainty and predictive performance.