Dataset distillation is a method for reducing dataset sizes by learning a small number of synthetic samples containing all the information of a large dataset. This has several benefits like speeding up model training, reducing energy consumption, and reducing required storage space. Currently, each synthetic sample is …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “dataset distillation”
Labels distilled from images improve model training efficiency and flexibility.
problem Creating synthetic labels for a small set of real images to train models effectively.
method Introduce a more robust and flexible meta-learning algorithm for distillation and an effective first-order strategy based on convex optimization layers.
result Label distillation leads to improved results and greater flexibility in neural architectures.
A new method distills datasets more efficiently and effectively.
problem Achieving competitive performance on test data with a small synthetic dataset.
method Tackles dataset distillation as a bilevel optimization problem, introduces RaT-BPTT to stabilize gradients and speed up optimization.
result Establishes new state-of-the-art performance across various benchmarks.
Optimal quantization improves dataset distillation for faster training.
problem Efficiently train models with synthetic data.
method Reformulate disentangled methods as optimal quantization problems.
result Better performance and generalization on ImageNet-1K and subsets.
DGKIP extends KIP for dataset distillation without bi-level optimization.
problem Efficiently distill datasets for various loss functions.
method Leverages duality theory to avoid bi-level optimization.
result DGKIP supports a wider range of loss functions.
New algorithm improves dataset distillation with 108% improvement on ImageNet.
problem Improving dataset distillation performance.
method Reparameterization and convexification of implicit gradients (RCIG).
result Establishes new state-of-the-art on various dataset distillation tasks.
Paper analyzes dataset distillation for efficient encoding of task-relevant information.
problem Efficiently encoding task-relevant information from gradient-based learning of non-linear tasks.
method Theoretical analysis of dataset distillation applied to two-layer neural networks with gradient-based training.
result Low-dimensional structure of the problem is efficiently encoded into distilled data, reproducing a model with high generalization ability.
Class-Imbalanced-Aware Adaptive Dataset Distillation for Scalable Pretrained Model on Credit Scoringcs.LG
This paper improves credit scoring models using a novel dataset distillation technique.
problem Limited scalability of pretrained models for tabular credit scoring datasets.
method Integrates class imbalance-aware dataset distillation with pretrained models.
result Improved AUC by 2.5% in financial datasets.
Dataset Distillationcs.LG
Model distillation aims to distill the knowledge of a complex model into a simpler one. In this paper, we consider an alternative formulation called dataset distillation: we keep the model fixed and instead attempt to distill the knowledge from a large training dataset into a small one. The idea is to synthesize a smal…
Knowledge distillation (KD) is a technique to derive optimal performance from a small student network (SN) by distilling knowledge of a large teacher network (TN) and transferring the distilled knowledge to the small SN. Since a role of convolutional neural network (CNN) in KD is to embed a dataset so as to perform a g…
Survey examines distillation methods for large language models.
problem Efficiently compress large language models while preserving their capabilities.
method Knowledge Distillation and Dataset Distillation techniques.
result Integrating KD and DD can produce more effective and scalable compression strategies.
Distillation affects some classes more than others, impacting fairness and bias.
problem Distillation affects some classes more than others, impacting fairness and bias.
method Examined class-wise accuracy and fairness metrics (DPD, EOD) on models trained with different datasets.
result Increasing the distillation temperature improves the distilled student model's fairness and individual fairness.
New method distills discrete diffusion models, maintaining quality and diversity.
problem Difficult to distill discrete diffusion models.
method Discrete Moment Matching Distillation (D-MMD)
result Maintains high quality and diversity in distilled models.
RFAD uses random features to speed up dataset distillation.
problem Efficiently compress large datasets for reduced storage and computation.
method Random feature approximation of the Neural Network Gaussian Process kernel.
result At least 100-fold speedup over KIP with competitive accuracy.
This paper improves speech recognition by distilling knowledge from acoustic models.
problem Improving speech recognition accuracy using ensemble models.
method Proposes multi-teacher distillation strategies for joint CTC-attention end-to-end ASR systems, integrating error rate metric for optimization.
result Reports state-of-the-art error rates on various datasets and languages.
Distilled models often fail to match teacher models, despite improving generalization.
problem The discrepancy between teacher and student predictive distributions remains large.
method Investigated the optimization difficulties and dataset details affecting student performance.
result Optimizing for matching the teacher does not always lead to better generalization.
Paper tackles adapting multiple domains to a target domain using distillation and dictionary learning.
problem Adapting multiple heterogeneous labeled source domains to an unlabeled target domain.
method Combines Multi-Source Domain Adaptation and Dataset Distillation with Dataset Dictionary Learning.
result Achieves state-of-the-art adaptation performance even with minimal labeled data.
This study improves knowledge distillation for RNN-T models with noisy labels.
problem Challenges in distilling knowledge from RNN-T models with variable quality teachers.
method Full-sum distillation and sequence-level knowledge distillation.
result Full-sum distillation outperforms other methods for RNN-T models, especially for bad teachers.
Knowledge distillation deals with the problem of training a smaller model (Student) from a high capacity source model (Teacher) so as to retain most of its performance. Existing approaches use either the training data or meta-data extracted from it in order to train the Student. However, accessing the dataset on which …
Knowledge distillation is a widely used technique for model compression. We posit that the teacher model used in a distillation setup, captures relationships between classes, that extend beyond the original dataset. We empirically show that a teacher model can transfer this knowledge to a student model even on an {\it …
Develops efficient algorithm for distilling supervised and offline RL datasets.
problem Creating synthetic datasets for supervised and offline RL models to match training data performance.
method An efficient algorithm based on matching losses without model training, using randomly sampled regressors.
result Synthetic datasets derived with sampled regressors match MSE loss of bounded linear models on training data.
DOSFL reduces federated learning communication by one round, preserving model performance.
problem High communication costs in federated learning with poorly distributed data.
method Clients distill their private data into synthetic data, sending only this to the server for training.
result Total communication cost up to 3 orders of magnitude less than FedAvg while maintaining comparable performance.
Learning to Generate Synthetic Training Data using Gradient Matching and Implicit Differentiationcs.LG
New methods distill data for deep networks efficiently.
problem Reduction of training data cost and inconvenience.
method Generative teaching networks, gradient matching, Implicit Function Theorem.
result New methods are computationally more efficient and improve model performance.
Extracurricular learning closes the accuracy gap in knowledge distillation.
problem Accuracy gap between teacher and student models after knowledge distillation.
method Modeling student and teacher output distributions, sampling from an extended data distribution, and matching over this set.
result Extracurricular learning reduces the accuracy gap by 46% to 68%.
A new method for privacy-preserving data distillation using wavelet features from ScatterNet.
problem Privacy-preserving data distillation for small datasets.
method Differentially private kernel inducing points using wavelet features from ScatterNet.
result Achieves privacy-preserving data distillation with manageable computational cost.
CAKD framework optimizes knowledge transfer by focusing on influential components of distillation.
problem Balancing and optimizing knowledge transfer in distillation models.
method Decouple KL divergence into BCD, SCD, and WCD; prioritize influential components.
result CAKD framework consistently outperforms baseline across diverse models and datasets.
Techniques such as ensembling and distillation promise model quality improvements when paired with almost any base model. However, due to increased test-time cost (for ensembles) and increased complexity of the training pipeline (for distillation), these techniques are challenging to use in industrial settings. In this…
This work improves understanding of neural network reconstruction attacks and distillation.
problem Understanding and mitigating reconstruction attacks on neural networks.
method Developed a stronger dataset reconstruction attack and studied its characteristics.
result Reconstruction attacks can recover entire training sets in the infinite width regime.
A modular framework for knowledge distillation simplifies experiments and reproducibility.
problem Difficulty in reproducing knowledge distillation studies due to lack of generalized frameworks.
method A configuration-driven PyTorch framework for knowledge distillation studies.
result Demonstrates efficient training strategies and various knowledge distillation methods.
A new algorithm improves federated learning by combining knowledge distillation and weighted combination loss.
problem Non-IID client data in federated learning leads to model drift and poor generalization.
method pFedKD-WCL integrates knowledge distillation with bi-level optimization to address non-IID challenges.
result pFedKD-WCL outperforms state-of-the-art algorithms in accuracy and convergence speed.
We present a novel framework of knowledge distillation that is capable of learning powerful and efficient student models from ensemble teacher networks. Our approach addresses the inherent model capacity issue between teacher and student and aims to maximize benefit from teacher models during distillation by reducing t…
SiD distills pretrained diffusion models into a fast one-step generator.
problem Efficiently distilling pretrained diffusion models into a fast generator.
method Reformulates forward diffusion processes as semi-implicit distributions and uses three score-related identities to create a loss mechanism.
result Achieves high FID performance and significantly reduces generation time.
Gaussian processes (GPs) are flexible models that can capture complex structure in large-scale dataset due to their non-parametric nature. However, the usage of GPs in real-world application is limited due to their high computational cost at inference time. In this paper, we introduce a new framework, \textit{kernel di…
Paper tackles hypothesis transfer learning for black-box models.
problem Difficult to build universal machine learning models across different institutions.
method Dynamic Knowledge Distillation (dkdHTL) with instance-wise weighting.
result Empirical results show the effectiveness of dkdHTL.
Model compression is eminently suited for deploying deep learning on IoT-devices. However, existing model compression techniques rely on access to the original or some alternate dataset. In this paper, we address the model compression problem when no real data is available, e.g., when data is private. To this end, we p…
FAST-DAD distills complex ensemble models into faster, more accurate individual models.
problem Deploying complex AutoML ensemble predictors on tabular data is slow, large, and opaque.
method Data augmentation strategy based on Gibbs sampling from a self-attention pseudolikelihood estimator.
result FAST-DAD distillation produces significantly better individual models than standard training.
Distillation works even with hard labels from overparameterized teacher, leading to better performance.
problem Improving model performance with hard labels from overparameterized teacher.
method Training a student model on a large held-out dataset labeled by a highly overparameterized teacher.
result Student model outperforms traditional approaches due to double descent phenomenon.
New method for decentralized learning from diverse clients.
problem Achieving high-performance global model from diverse client models.
method Decentralized Learning via Adaptive Distillation (DLAD).
result Demonstrates effectiveness of DLAD on multiple public datasets.
Paper shows distillation can maintain low model churn.
problem Maintaining low model churn in real-world systems.
method Equivalence between distillation and explicit churn constraints.
result Distillation significantly reduces model churn with minimal accuracy loss.
Current approaches for Knowledge Distillation (KD) either directly use training data or sample from the training data distribution. In this paper, we demonstrate effectiveness of 'mismatched' unlabeled stimulus to perform KD for image classification networks. For illustration, we consider scenarios where this is a comp…
Knowledge distillation boosts simple models in banking without complexity.
problem Complexity and performance constraints in banking models.
method Soft Targets, Sample Selection, Data Augmentation.
result Improved model performance without altering model simplicity.
SARD improves deep learning clinical prediction performance.
problem Deep learning models struggle to match linear models in healthcare predictions.
method Reverse Distillation to initialize deep models, combined with contextual and temporal embeddings.
result SARD outperforms state-of-the-art methods on clinical prediction outcomes.
SAIL improves graph node representation learning by distilling knowledge between graphs.
problem Improving graph node representation learning with GNNs in unsupervised scenarios.
method SAIL framework with intra- and inter-graph knowledge distillation.
result SAIL consistently outperforms state-of-the-art baselines on various benchmark datasets.
Knowledge distillation is an effective technique that transfers knowledge from a large teacher model to a shallow student. However, just like massive classification, large scale knowledge distillation also imposes heavy computational costs on training models of deep neural networks, as the softmax activations at the la…
Many engineers wish to deploy modern neural networks in memory-limited settings; but the development of flexible methods for reducing memory use is in its infancy, and there is little knowledge of the resulting cost-benefit. We propose structural model distillation for memory reduction using a strategy that produces a …
KDM unifies feature matching in neural networks.
problem Matching neural network activations for knowledge transfer.
method Formalizes distribution matching strategy.
result New theoretical results for KD.
Deep Neural Networks have achieved huge success at a wide spectrum of applications from language modeling, computer vision to speech recognition. However, nowadays, good performance alone is not sufficient to satisfy the needs of practical deployment where interpretability is demanded for cases involving ethics and mis…
Paper proposes G-CRD to improve GNNs by preserving global graph topology.
problem Improving lightweight GNNs for robust performance on large-scale real-world graphs.
method Introduces Graph Contrastive Representation Distillation (G-CRD) using contrastive learning.
result G-CRD consistently boosts GNN performance and robustness, outperforming existing methods.