We study the use of knowledge distillation to compress the U-net architecture. We show that, while standard distillation is not sufficient to reliably train a compressed U-net, introducing other regularization methods, such as batch normalization and class re-weighting, in knowledge distillation significantly improves …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,932 papers · 148 categories
Trend · papers per month
48 results for “Knowledge Compression”
Survey of knowledge distillation for model compression.
problem Deploying large deep models on resource-limited devices.
method Knowledge distillation from large teacher models to small student models.
result Effective model compression and acceleration.
Weight Squeezing transfers knowledge from large models to smaller ones, improving performance and speed.
problem Transfer learning and model compression for faster and more efficient training.
method Reparameterization of weights from a large model to a smaller one.
result Weight Squeezing outperforms other methods on GLUE benchmark with faster training.
RKD improves model compression by distilling residual knowledge from a deep teacher model.
problem Performance degradation due to the gap between student and teacher models.
method Introduces an assistant model to distill residual knowledge from the teacher model.
result RKD achieves better results on popular classification datasets than state-of-the-art methods.
Paper analyzes consistency between neural networks at different levels.
problem Understanding knowledge consistency between neural networks.
method Generic definition and task-agnostic method to disentangle consistent knowledge.
result Knowledge consistency provides new insights and can improve network performance.
New theory controls compression change probability without prior knowledge.
problem Controlling compression change probability without prior knowledge.
method New theory on compression function and statistical risk.
result Cardinality of compressed set is a consistent estimator of probability of change of compression.
Few-shot network compression improves accuracy with minimal data.
problem High estimation errors from original network during inference.
method Cross distillation, layer-wise knowledge distillation approach.
result Cross distillation significantly improves student network's accuracy with few training instances.
Distilled teacher model transfers knowledge to student model on new datasets.
problem Improving model quality using unlabeled data.
method Knowledge distillation with an unlabeled teacher model.
result Teacher model knowledge transfers to student model on out-of-distribution datasets.
A new teacher-class network method compresses DNNs by distributing knowledge to multiple student networks.
problem Overwhelming size of Deep Neural Networks (DNNs).
method Single teacher with multiple student networks, transferring knowledge to each student.
result The combined knowledge of the class of students achieves better performance and reduces parameters.
Method finds motifs in knowledge graphs, revealing their structure.
problem Identifying meaningful subunits in knowledge graphs.
method Inspired by simple graphs, the approach uses compression techniques to find motifs.
result Motifs found reflect the basic structure of the graph.
This work considers an estimation task in compressive sensing, where the goal is to estimate an unknown signal from compressive measurements that are corrupted by additive pre-measurement noise (interference, or clutter) as well as post-measurement noise, in the specific setting where some (perhaps limited) prior knowl…
Video analytics requires operating with large amounts of data. Compressive sensing allows to reduce the number of measurements required to represent the video using the prior knowledge of sparsity of the original signal, but it imposes certain conditions on the design matrix. The Bayesian compressive sensing approach r…
Enhances model compression with multi-teacher knowledge distillation.
problem Uncertainty evaluation and diverse teacher expertise in model deployment.
method Bayesian inference and teacher-informed prior with entropy-based weighting.
result Improved predictive accuracy and robust uncertainty quantification.
This work automates CNN model compression for mobile devices.
problem Deploying trained CNNs to mobile devices requires balancing speed, memory, and accuracy.
method Reinforcement learning with architecture search and knowledge distillation.
result An automated model compression algorithm improves the trade-off between speed, memory, and accuracy.
Study explores various KD methods to improve model performance.
problem Improving model performance with limited data.
method Nine different KD methods covering various aspects of the teacher.
result KD framework provides generalization gains over standard training.
HOPE uses Hilbert space to deconstruct deep network representations.
problem Deconstructing learned representations in deep networks is challenging.
method Introduces Hilbert Operator for Progressive Encoding (HOPE) to deconstruct network weights.
result HOPE provides an unbiased approach to network compression and fine-tuning.
Proposes a method to compress and sparsify neural networks using KD and VI.
problem Porting deep neural networks to embedded platforms with minimal accuracy loss.
method Combines knowledge distillation and variational inference to create a sparse student network.
result Significant memory footprint reduction and improved sparsity without accuracy loss.
Knowledge distillation improves model accuracy by mimicking teacher model probabilities.
problem Improving model accuracy through model compression.
method Casting knowledge distillation as a semiparametric inference problem, deriving new guarantees, and developing enhancements.
result Enhancements improve student performance by mitigating teacher overfitting and underfitting.
Paper compresses neural network weight-updates for image artifacts removal.
problem Efficiently compressing neural network weight-updates for image artifacts removal.
method Fine-tuning a pre-trained artifact removal network on target data with a compression objective that encourages sparse and quantized weight-updates.
result Achieves reconstruction quality comparable to traditional codecs at comparable bitrates.
Bayesian Dark Knowledge fails to perform well with high posterior uncertainty.
problem Performance degradation of Bayesian Dark Knowledge with high posterior uncertainty.
method Compresses posterior predictive distribution into a single network, using a student network matching the teacher ensemble architecture.
result Using a matching student network architecture does not guarantee acceptable performance with high posterior uncertainty.
Generative Adversarial Networks (GANs) have been used in several machine learning tasks such as domain transfer, super resolution, and synthetic data generation. State-of-the-art GANs often use tens of millions of parameters, making them expensive to deploy for applications in low SWAP (size, weight, and power) hardwar…
Adjoined Networks trains both base and compressed networks together for efficient model compression.
problem Efficiently compressing deep neural networks while maintaining accuracy.
method Adjoined Networks (AN) trains both a base network and a smaller compressed network simultaneously, sharing parameters.
result AN achieves 71.8% top-1 accuracy with 1.8M parameters and 1.6 GFLOPs on ImageNet.
Extracurricular learning closes the accuracy gap in knowledge distillation.
problem Accuracy gap between teacher and student models after knowledge distillation.
method Modeling student and teacher output distributions, sampling from an extended data distribution, and matching over this set.
result Extracurricular learning reduces the accuracy gap by 46% to 68%.
This paper improves understanding of knowledge distillation and its effects.
problem Improving model quality with a fixed capacity budget.
method Categorizing teacher's knowledge into three levels and studying their effects on distillation.
result The three hierarchical levels of teacher's knowledge (universe, domain, instance) significantly impact knowledge distillation.
How to obtain a model with good interpretability and performance has always been an important research topic. In this paper, we propose rectified decision trees (ReDT), a knowledge distillation based decision trees rectification with high interpretability, small model size, and empirical soundness. Specifically, we ext…
FEDS distills LIC model knowledge into a lightweight student for efficient compression.
problem Efficiently compress images with high performance and low resources.
method FEDS combines feature alignment and entropy-based loss for lightweight compression.
result Student model matches teacher's performance while reducing parameters and speeding up.
We give an algorithmically efficient version of the learner-to-compression scheme conversion in Moran and Yehudayoff (2016). In extending this technique to real-valued hypotheses, we also obtain an efficient regression-to-bounded sample compression converter. To our knowledge, this is the first general compressed regre…
Compressed imitation learning uses simplicity priors for efficient expert behavior copying.
problem Efficiently learn expert behaviors with minimal data.
method Utilizes policy simplicity as a prior for sample-efficient imitation learning.
result Significantly higher scores achieved with limited expert demonstrations.
The soaring demand for intelligent mobile applications calls for deploying powerful deep neural networks (DNNs) on mobile devices. However, the outstanding performance of DNNs notoriously relies on increasingly complex models, which in turn is associated with an increase in computational expense far surpassing mobile d…
IDF uses integer flows for lossless compression of discrete data.
problem Lossless compression of discrete data with minimal reconstruction errors.
method Integer Discrete Flows (IDF) using integer maps and integer discrete coupling layers.
result IDFs achieve state-of-the-art lossless compression rates on various datasets.
We introduce a conceptually simple and scalable framework for continual learning domains where tasks are learned sequentially. Our method is constant in the number of parameters and is designed to preserve performance on previously encountered tasks while accelerating learning progress on subsequent problems. This is a…
Collaborative filtering often suffers from sparsity and cold start problems in real recommendation scenarios, therefore, researchers and engineers usually use side information to address the issues and improve the performance of recommender systems. In this paper, we consider knowledge graphs as the source of side info…
SpanishTinyRoBERTa distills large Spanish models into efficient question-answering models.
problem Efficient Spanish question-answering models for resource-constrained environments.
method Knowledge distillation from large Spanish language models onto a smaller model.
result SpanishTinyRoBERTa achieves comparable performance to large models with faster inference.
Distributed sensors compress and send features to a fusion center for linear regression.
problem Efficiently compress and transmit features from distributed sensors to a fusion center under varying communication constraints.
method Designs a distributed and adaptive feature compression scheme using optimal quantizers and simple adaptive strategies.
result Demonstrates improved inference performance through simulated experiments.
Deep neural network compression techniques such as pruning and weight tensor decomposition usually require fine-tuning to recover the prediction accuracy when the compression ratio is high. However, conventional fine-tuning suffers from the requirement of a large training set and the time-consuming training procedure. …
This paper speeds up SVC clustering by compressing data while preserving key properties.
problem Efficiently clustering large-scale real-world data sets.
method Spectrum-preserving data compression for fast support vector clustering.
result Achieved 100X and 115X speedups on real-world data sets while maintaining clustering quality.
Improves compression of neural networks for embedded systems.
problem High computational cost and data labeling issues in DNNs.
method Domain Adaptation Regularization for Spectral Pruning.
result Our method outperforms existing methods by a large margin for high compression rates.
Paper proposes a method to compress deep learning models using PU setting and cloud data.
problem Compression and acceleration of deep learning models on portable devices.
method Positive-Unlabeled (PU) setting, attention-based multi-scale feature extractor, robust knowledge distillation.
result An efficient model can be obtained using only 8% of ImageNet data.
Deep network compression has been achieved notable progress via knowledge distillation, where a teacher-student learning manner is adopted by using predetermined loss. Recently, more focuses have been transferred to employ the adversarial training to minimize the discrepancy between distributions of output from two net…
Proposes using equivariant generative models for compressed sensing with unknown orientations.
problem Recovering signals with unknown orientations from underdetermined systems of linear measurements.
method Equivariant variational autoencoder as a generative prior for compressed sensing.
result Signals with unknown orientations can be recovered using iterative gradient descent on the latent space of equivariant models.
This paper proposes a method to compress and adapt CNNs for real-world applications.
problem Differences in data distributions and high computational costs limit CNN adoption.
method Joint optimization of CNNs for unsupervised domain adaptation and knowledge distillation.
result The proposed method achieves the highest accuracy with comparable or lower time complexity.
Interpreting black box classifiers, such as deep networks, allows an analyst to validate a classifier before it is deployed in a high-stakes setting. A natural idea is to visualize the deep network's representations, so as to "see what the network sees". In this paper, we demonstrate that standard dimension reduction m…
New methods compress models without real data, reducing accuracy loss.
problem Compression requires real data, which is often unavailable or sensitive.
method Synthetic data generation from trained models for calibration and fine-tuning.
result Best method shows negligible accuracy loss compared to original training set.
Distilling Model Knowledgestat.ML
Top-performing machine learning systems, such as deep neural networks, large ensembles and complex probabilistic graphical models, can be expensive to store, slow to evaluate and hard to integrate into larger systems. Ideally, we would like to replace such cumbersome models with simpler models that perform equally well…
Electron Cryo-Tomography (ECT) enables 3D visualization of macromolecule structure inside single cells. Macromolecule classification approaches based on convolutional neural networks (CNN) were developed to separate millions of macromolecules captured from ECT systematically. However, given the fast accumulation of ECT…
This paper proposes a simple adaptive sensing and group testing algorithm for sparse signal recovery. The algorithm, termed Compressive Adaptive Sense and Search (CASS), is shown to be near-optimal in that it succeeds at the lowest possible signal-to-noise-ratio (SNR) levels, improving on previous work in adaptive comp…
PLD distills knowledge using choice-theoretic Plackett-Luce model.
problem Model compression and knowledge transfer between large and small networks.
method PLD uses a weighted list-wise ranking loss based on the Plackett-Luce model.
result PLD achieves consistent gains across diverse architectures and distillation methods.
AntMan compresses RNNs for faster inference with minimal accuracy loss.
problem Inference performance, cost, and memory requirements of complex RNN models.
method Structured sparsity combined with low-rank decomposition.
result Up to 100x computation reduction with less than 1pt accuracy drop.