RID framework quantifies and regularizes task-relevant knowledge in distillation.
problem Distilling irrelevant information can hinder student model performance.
method Partial Information Decomposition to quantify and regularize task-relevant knowledge.
result RID framework leads to more resilient distillation under nuisance teachers.
Framework detects brain tumors robustly from MRI images.
problem Low clinical incidence of brain tumor cases makes diagnosis challenging.
method YOLOv8n for detection, DeiT for classification, PTP metric for evaluation.
result F1-score of 0.92 achieved with reduced computational resources.
Deep Neural Networks are built to generalize outside of training set in mind by using techniques such as regularization, early stopping and dropout. But considerations to make them more resilient to adversarial examples are rarely taken. As deep neural networks become more prevalent in mission-critical and real-time sy…
Time series classification models have been garnering significant importance in the research community. However, not much research has been done on generating adversarial samples for these models. These adversarial samples can become a security concern. In this paper, we propose utilizing an adversarial transformation …
Paper introduces a new index to measure financial and workplace resilience of firms.
problem Corporate resilience and its types in turbulent markets.
method Quantitative analysis of earnings expectations and implied discount rates.
result Evidence of workplace resilience amplification by financial status in the COVID-19 era.
We provide a methodology, resilient feature engineering, for creating adversarially resilient classifiers. According to existing work, adversarial attacks identify weakly correlated or non-predictive features learned by the classifier during training and design the adversarial noise to utilize these features. Therefore…
Measures financial resilience using BSDEs and their properties.
problem Measuring financial resilience in dynamic risk environments.
method Developed stochastic calculus for BSDEs with jumps, revealing resilience rate as expectation of generator.
result Resilience rate can be represented as expectation of BSDE generator, revealing properties of dynamic risk measures.
Recent variants improve knowledge distillation performance.
problem Improving the performance of knowledge distillation.
method Introducing additional components or changing the learning process.
result These variants have shown promising results.
Python tool assesses European agricultural production resilience.
problem Estimating resilience of European agricultural production systems.
method Python software to compute annual production resilience indicator.
result Demonstrates resilience of European agricultural production.
This paper measures financial market resilience in China and identifies key uncertainties.
problem Measuring financial market resilience in China.
method Quantitative analysis of total financial market and sub-markets, Diebold-Yilmaz connectedness approach.
result Financial market resilience in China is event-driven and influenced by geopolitical risks, economic and trade policy uncertainty, and U.S.-China tensions.
Paper proposes transforming ATN to attack multivariate time series models.
problem Generating adversarial samples for multivariate time series classification models.
method Proposes using a distilled model as a surrogate to mimic attacked models and applies 1-NN DTW and FCN attacks.
result Both models were susceptible to attacks on all 18 datasets.
The resilience of low-degree Rademacher chaos is studied, providing probabilistic lower bounds.
problem Understanding how much a Rademacher chaos can withstand adversarial sign-flips without significant probability changes.
method Probabilistic lower-bound guarantees for the resilience of Rademacher chaos of arbitrary degree.
result Probabilistic lower-bound guarantees for the resilience of Rademacher chaos of arbitrary degree, especially meaningful for constant degree.
New model shows negative resilience can improve trading efficiency.
problem Optimal trade execution in limit order books with negative resilience.
method Stochastic order book model with negative resilience.
result Negative resilience can lead to more efficient trading.
Proposes resilience metrics for large blackout costs with logarithmic resilience.
problem Large variations in blackout costs make estimating risk impractical.
method Uses mean of log of large blackout costs, tail slope index, and frequency.
result Solves problems of heavy tail and large variations in blackout costs.
This study improves knowledge distillation for RNN-T models with noisy labels.
problem Challenges in distilling knowledge from RNN-T models with variable quality teachers.
method Full-sum distillation and sequence-level knowledge distillation.
result Full-sum distillation outperforms other methods for RNN-T models, especially for bad teachers.
Labels distilled from images improve model training efficiency and flexibility.
problem Creating synthetic labels for a small set of real images to train models effectively.
method Introduce a more robust and flexible meta-learning algorithm for distillation and an effective first-order strategy based on convex optimization layers.
result Label distillation leads to improved results and greater flexibility in neural architectures.
Stocks of more resilient firms outperformed during the pandemic, reflecting disaster risk.
problem The impact of social distancing on firms' operations and stock performance.
method Cross-sectional analysis of firms' resilience and stock performance, controlling for risk factors.
result Stocks of more resilient firms are expected to yield significantly lower returns than less resilient ones, reflecting disaster risk.
New method distills discrete diffusion models, maintaining quality and diversity.
problem Difficult to distill discrete diffusion models.
method Discrete Moment Matching Distillation (D-MMD)
result Maintains high quality and diversity in distilled models.
Dataset distillation is a method for reducing dataset sizes by learning a small number of synthetic samples containing all the information of a large dataset. This has several benefits like speeding up model training, reducing energy consumption, and reducing required storage space. Currently, each synthetic sample is …
Estimates model performance from compute budget for distillation.
problem Risk mitigation in large-scale distillation.
method Distillation scaling law based on compute budget allocation.
result Maximizes student performance with compute-optimal allocation.
Distillation affects some classes more than others, impacting fairness and bias.
problem Distillation affects some classes more than others, impacting fairness and bias.
method Examined class-wise accuracy and fairness metrics (DPD, EOD) on models trained with different datasets.
result Increasing the distillation temperature improves the distilled student model's fairness and individual fairness.
Survey of knowledge distillation for resource-limited devices.
problem Deploying large deep learning models on resource-limited devices.
method Knowledge distillation using a smaller model trained with information from a larger model.
result A new metric (distillation metric) for comparing different knowledge distillation algorithms.
Financial markets can be seen as complex systems that are constantly evolving and sensitive to external disturbance, such as systemic risks and economic instabilities. Analysis of resilient market performance, therefore, becomes useful for investors. From a systems perspective, this paper proposes a novel function-base…
New method quantifies resilience of electric distribution systems from historical data.
problem Large blackouts caused by extreme winds have significant costs and impacts.
method Formulate large event risk from utility outage data, quantify resilience improvements through investments.
result Investments in wind hardening and faster restoration can reduce the probability of large cost events.
Policy distillation in deep reinforcement learning provides an effective way to transfer control policies from a larger network to a smaller untrained network without a significant degradation in performance. However, policy distillation is underexplored in deep reinforcement learning, and existing approaches are compu…
Study on ensemble, distillation, and self-distillation in deep learning models.
problem Improving test accuracy in deep learning models using ensemble and distillation methods.
method Formal study of ensemble and distillation, considering multi-view data structure.
result Proven that ensemble and distillation can improve test accuracy in deep learning models, and the superior performance can be distilled into a single model.
A new distillation method transfers channel information from teacher to student.
problem Transfer knowledge from teacher to student with fewer parameters and calculations.
method Channel Distillation (CD) and Guided Knowledge Distillation (GKD) with loss decay.
result Achieved 27.68% top-1 error on ImageNet with ResNet18, outperforming state-of-the-art methods.
Knowledge distillation is effective for producing small, high-performance neural networks for classification, but these small networks are vulnerable to adversarial attacks. This paper studies how adversarial robustness transfers from teacher to student during knowledge distillation. We find that a large amount of robu…
Repeated self-distillation improves model performance significantly.
problem How much gain is possible by applying multiple steps of self-distillation?
method Investigated linear regression tasks, applied multiple steps of self-distillation, analyzed excess risk reduction.
result Multi-step self-distillation reduces excess risk by a factor as large as d, where d is the input dimension. New framework for resilient bi-criteria optimization under noisy feedback.
problem Bi-criteria combinatorial optimization with noisy function evaluations.
method Introducing (α,β,δ,extttN)-resilience and developing a black-box framework. result Achieves sublinear regret and constraint violation for bi-criteria bandit problems.
A new method for training GNNs without a teacher model.
problem Training over-parameterized GNN models is difficult and inefficient.
method GNN Self-Distillation (GNN-SD) with NDR and ADR.
result Improves GNN performance with less training cost and better generalization.
FasterVoiceGrad speeds up VC by 6-7x with novel distillation.
problem Slow iterative sampling in diffusion-based VC models.
method Adversarial diffusion conversion distillation (ADCD) to create a faster one-step model.
result 6.6-6.9 and 1.8x faster on GPU and CPU, respectively.
This paper improves speech recognition by distilling knowledge from acoustic models.
problem Improving speech recognition accuracy using ensemble models.
method Proposes multi-teacher distillation strategies for joint CTC-attention end-to-end ASR systems, integrating error rate metric for optimization.
result Reports state-of-the-art error rates on various datasets and languages.
Paper tackles Byzantine resilience in distributed multi-task learning.
problem Resilience of distributed algorithms in the presence of Byzantine agents.
method Online weight assignment rule based on accumulated loss and filtering.
result Aggregation with proposed weight assignment rule improves expected regret.
Federated Learning aims to train distributed deep models without sharing the raw data with the centralized server. Similarly, in distributed inference of neural networks, by partitioning the network and distributing it across several physical nodes, activations and gradients are exchanged between physical nodes, rather…
This paper presents the philosophy, design and feature-set of Neural Network Distiller, an open-source Python package for DNN compression research. Distiller is a library of DNN compression algorithms implementations, with tools, tutorials and sample applications for various learning tasks. Its target users are both en…
CAKD framework optimizes knowledge transfer by focusing on influential components of distillation.
problem Balancing and optimizing knowledge transfer in distillation models.
method Decouple KL divergence into BCD, SCD, and WCD; prioritize influential components.
result CAKD framework consistently outperforms baseline across diverse models and datasets.
Consistency distillation reduces memorization in diffusion models without harming sample quality.
problem Understanding how distillation affects memorization in diffusion models.
method Analysis of consistency distillation in diffusion models using a random feature neural network model.
result Consistency distillation reduces memorization in diffusion models without harming sample quality.
A new method distills datasets more efficiently and effectively.
problem Achieving competitive performance on test data with a small synthetic dataset.
method Tackles dataset distillation as a bilevel optimization problem, introduces RaT-BPTT to stabilize gradients and speed up optimization.
result Establishes new state-of-the-art performance across various benchmarks.
This paper uses robust optimization to analyze supply chain resilience.
problem Supply chain resilience analysis of multi-modal logistics networks.
method Robust optimization with budget-of-uncertainty.
result Interactive effects of network size, disruption scale, and degree on resilience.
The transfer of knowledge from one policy to another is an important tool in Deep Reinforcement Learning. This process, referred to as distillation, has been used to great success, for example, by enhancing the optimisation of agents, leading to stronger performance faster, on harder domains [26, 32, 5, 8]. Despite the…
Improved generalization with iterative self-distillation using weighted ground-truth targets.
problem Improving generalization accuracy in neural networks.
method Iterative kernel regression with weighted ground-truth targets and ℓ2 regularization. result Closed-form solution for optimal weighting parameter and efficient estimation.
The paper explains knowledge distillation by analyzing visual concepts in DNNs.
problem Understanding how knowledge distillation affects the learning of visual concepts in deep neural networks.
method The paper proposes three hypotheses and designs mathematical metrics to evaluate feature representations of DNNs.
result The hypotheses were verified through experiments on various DNNs.
This work proposes splitting deep neural networks into smaller sub-networks for faster and more efficient distillation.
problem Challenges in training deep neural networks, including local optima, gradient issues, and computational demands.
method Proposes a non-end-to-end distillation approach by splitting networks into smaller, independent sub-networks (neighbourhoods).
result Independent training of smaller sub-networks can speed up distillation and improve efficiency in various applications.
SiD distills pretrained diffusion models into a fast one-step generator.
problem Efficiently distilling pretrained diffusion models into a fast generator.
method Reformulates forward diffusion processes as semi-implicit distributions and uses three score-related identities to create a loss mechanism.
result Achieves high FID performance and significantly reduces generation time.
Compressed Federated Distillation reduces communication in federated learning.
problem Communication constraints in Federated Learning.
method Compressed Federated Distillation (CFD) leverages soft labels and quantization techniques.
result Reduces communication by more than 4 orders of magnitude compared to Federated Averaging.
Extracting a curriculum from a teacher network improves distillation efficiency.
problem Efficiently training a small network using a large teacher network's output.
method Random projection of teacher network's hidden representations to progressively train the student network.
result Extracted curriculum significantly outperforms one-shot distillation and achieves similar performance to progressive distillation.
Improved self-distillation reduces label noise and enhances model accuracy.
problem Label noise in multi-class classification.
method Label averaging and refined partial labels.
result Single-round self-distillation achieves comparable performance to multi-round distillation.