Study on neural networks' performance in sequential task learning.
problem Understanding the performance of neural networks in sequential task learning.
method Theoretical analysis of generalization performance in continual learning using statistical mechanical analysis of kernel ridge-less regression.
result Characteristic transitions from positive to negative transfer observed in neural networks.
New method regularizes deep networks by distilling self-knowledge.
problem Overfitting in deep neural networks.
method Self-knowledge distillation to regularize class-wise predictions.
result Significant improvement in generalization and calibration.
This paper improves ASR performance by aligning frames more accurately.
problem Disagreement between teacher-student models in frame-level alignment.
method Introduces self-knowledge distillation (SKD) to guide frame-level alignment.
result Improves both resource efficiency and performance in ASR.
Since deep learning became a key player in natural language processing (NLP), many deep learning models have been showing remarkable performances in a variety of NLP tasks, and in some cases, they are even outperforming humans. Such high performance can be explained by efficient knowledge representation of deep learnin…
SPEQ improves quantized neural networks by stochastic precision sharing and cosine similarity loss.
problem Improving quantized deep neural networks for edge devices.
method SPEQ combines stochastic precision sharing and cosine similarity loss for knowledge distillation.
result SPEQ outperforms existing methods in various tasks.
PS-KD distills a model's own knowledge to soften hard targets during training.
problem Improving generalization of deep neural networks by softening hard targets.
method Progressive self-knowledge distillation (PS-KD) that progressively distills a model's own knowledge to soften hard targets.
result PS-KD improves accuracy and provides high quality of confidence estimates in terms of calibration and ordinal ranking.
ProSelfLC improves robustness of deep neural networks by automatically deciding trust in predictions.
problem Training robust deep neural networks requires addressing issues like label noise and low entropy predictions.
method ProSelfLC progressively increases trust in predicted labels over time, considering entropy and learning time.
result ProSelfLC demonstrates improved robustness in both clean and noisy settings through empirical validation.
Transfer learning improves portfolio optimization by identifying transfer risk.
problem Financial portfolio optimization problem.
method Introduces transfer risk concept within transfer learning framework.
result Transfer risk is a significant indicator of transferability and enhances portfolio management efficiency.
Paper analyzes transfer risk in transfer learning for finance.
problem Evaluate transferability of transfer learning in finance.
method Proposes transfer risk concept and applies to stock return prediction and portfolio optimization.
result Transfer risk correlates with transfer learning performance and identifies appropriate source tasks.
This paper explores the connection between adversarial and knowledge transferability.
problem Understanding the factors affecting knowledge transferability.
method Theoretical analysis and practical metrics for adversarial transferability.
result Adversarial transferability and knowledge transferability are closely related.
Proposes LsrKD and MrKD to improve neural network training performance.
problem Improving neural network training performance, especially on deep networks.
method Extends Label Smoothing Regularization with Self-Knowledge Distillation, introducing LsrKD and MrKD.
result LsrKD and MrKD significantly improve training performance on deep neural networks.
Mathematical framework for transfer learning feasibility and transfer risk.
problem Theoretical analysis of transfer learning.
method Reformulated transfer learning as an optimization problem, introduced transfer risk concept.
result Demonstrated the potential and benefits of incorporating transfer risk in transfer learning evaluation.
Transfer learning borrows knowledge from a source domain to facilitate learning in a target domain. Two primary issues to be addressed in transfer learning are what and how to transfer. For a pair of domains, adopting different transfer learning algorithms results in different knowledge transferred between them. To dis…
Survey connects and systematizes transfer learning research.
problem Reduce dependence on target domain data for target learners.
method Systematic review of 40+ transfer learning approaches.
result Importance of choosing appropriate transfer learning models.
Study measures impact of data and neural net similarity on transferability in restaurant sales data.
problem Identify indicators for successful transferability of neural nets across different data sets.
method Empirical study on sales data from six restaurants, calculating indicators based on data and neural net similarities.
result Negative correlations between transferability and indicators, allowing better model performance and fewer transfers.
The paper analyzes phase transitions in transfer learning for perceptrons.
problem Understanding when transfer learning from a source task to a target task is beneficial.
method Theoretical analysis of a pair of related perceptron learning tasks.
result Reveals a phase transition from negative to positive transfer as task similarity changes.
Adaptive source selection for positive transfer in linear models improves target dataset performance.
problem Limited task-specific labeled data in business settings.
method Greedily decides from which sources and how many samples to incorporate into the target dataset using an accept/reject rule based on a data-dependent estimate of the transfer gain.
result Consistent gains over classical and recent strong baselines while avoiding negative transfer.
New research on limits of transfer learning, proving key selection and dependence requirements.
problem Insufficient theoretical foundation for transfer learning.
method Proved novel results on transfer learning, emphasizing selection of information and dependence between domains.
result Upper bound on improvement possible with transfer learning, highlighting the need for careful selection.
Proposes a transfer learning method for high-dimensional quantile regression.
problem Inadequate handling of heterogeneity and heavy tails in transfer learning.
method High-dimensional quantile regression framework with double transfer learning estimator.
result Established error bounds and valid confidence intervals for high-dimensional quantile regression coefficients.
Localized transfer learning improves nonparametric regression performance.
problem Improving nonparametric regression performance on target tasks.
method Localized transfer learning framework that models heterogeneity and partition covariate space into cells.
result Sharp minimax rates show local transfer mitigates the curse of dimensionality.
Transfer learning aims at building robust prediction models by transferring knowledge gained from one problem to another. In the semantic Web, learning tasks are enhanced with semantic representations. We exploit their semantics to augment transfer learning by dealing with when to transfer with semantic measurements an…
This work transfers causal knowledge between tasks for Individual Treatment Effect estimation.
problem Estimating Individual Treatment Effects (ITE) requires a large amount of data, making it challenging.
method The authors introduce a practical framework for efficient transfer of causal knowledge between tasks, using a Causal Inference Task Affinity (CITA) measure.
result ITE knowledge transfer can significantly reduce the amount of data needed for ITE estimation.
Investigates transfer learning in spatial statistics.
problem Applying transfer learning to spatial statistics.
method Simple MLP models for spatial data.
result Potential of transfer learning in spatial statistics.
With the help of transfer entropy, we analyze information flows between communities of complex networks. We show that the transfer entropy provides a coherent description of interactions between communities, including non-linear interactions. To put some flesh on the bare bones, we analyze transfer entropies between co…
Simple methods improve regression transferability estimation.
problem Estimating how well regression models transfer between tasks.
method Two simple, computationally efficient approaches based on negative regularized mean squared error.
result Significantly outperform existing methods in accuracy and efficiency.
This paper defines and quantifies transferability in domain generalization.
problem Understanding and quantifying transferability between domains.
method Formal definition and estimation of transferability, upper bound for target error.
result Many algorithms do not learn transferable features, proposing a new algorithm.
Training a source model optimally for its own task is suboptimal for downstream transfer.
problem The optimality of a source model for its own task hinders downstream transfer performance.
method Analyzes L2-SP ridge regression, characterizes transfer-optimal source penalty, and identifies alignment-dependent effects.
result Transfer benefits from stronger source regularization when aligned imperfectly, and from weaker regularization when aligned perfectly.
Transfer learning does not improve character recognition performance.
problem Improving character recognition performance using transfer learning.
method Performed experiments with varying levels of similarity between source and target tasks, transferring both parameters and features.
result No significant advantage gained by transfer learning over traditional machine learning.
AdaTrans adapts to feature and sample transfer in high-dimensional regression.
problem High-dimensional linear regression with more features than samples.
method F-AdaTrans and S-AdaTrans methods using fused-penalties and adaptive weights.
result AdaTrans achieves convergence rates close to oracle estimators and near-minimax optimal rates.
When labeled data is scarce for a specific target task, transfer learning often offers an effective solution by utilizing data from a related source task. However, when transferring knowledge from a less related source, it may inversely hurt the target performance, a phenomenon known as negative transfer. Despite its p…
Adversarial perturbations fool wearable sensor systems, showing transferability across different systems.
problem Adversarial examples fool wearable sensor systems, showing transferability across different systems.
method Study of adversarial transferability in wearable sensor systems from four perspectives: systems, subjects, sensor body locations, and datasets.
result Strong untargeted transferability in most cases, targeted attacks less successful.
Paper tackles continuous transfer learning with evolving target domains.
problem Challenges of negative transfer in evolving target domains.
method Proposes label-informed C-divergence for measuring distribution shift and negative transfer.
result Demonstrates effectiveness of TransLATE framework in minimizing classification error and C-divergence.
We observe standard transfer learning can improve prediction accuracies of target tasks at the cost of lowering their prediction fairness -- a phenomenon we named discriminatory transfer. We examine prediction fairness of a standard hypothesis transfer algorithm and a standard multi-task learning algorithm, and show th…
Paper introduces a new method for improving reinforcement learning performance using transfer learning.
problem Improving reinforcement learning performance with limited sample sizes in dynamic decision-making scenarios.
method Developed a novel ``re-weighted targeting procedure'' and ``transfer deep Q∗-learning'' approach. result Demonstrated improved reinforcement learning performance through strategic sample construction.
LEEP measures transferability of learned representations efficiently.
problem Evaluating the transferability of learned representations in machine learning.
method LEEP: Log Expected Empirical Prediction, a simple measure requiring one pass through the target data set.
result LEEP predicts transfer and meta-transfer learning performance and convergence speed, outperforming existing measures.
This study investigates how much knowledge from natural images can be transferred to pathology images.
problem Quantifying how much knowledge from natural images can be transferred to pathology images.
method Proposes a framework to quantify knowledge gain by a particular layer, conducts empirical investigation in pathology image centered transfer learning.
result Early layers of deep models can transfer knowledge to pathology image classification tasks.
We analyze a monetary system of random money transfer on the basis of double entry bookkeeping. Without boundary conditions, we do not reach a price equilibrium and violate text-book formulas of economists quantity theory (MV=PQ). To match the resulting quantity of money with the model assumption of a constant price, w…
Paper proposes a statistical test for transfer learning in linear regression.
problem Theoretical framework for parameter transfer in linear regression.
method Developed a statistical test to predict transfer quality.
result The test can predict if a fine-tuned model has lower prediction risk.
New framework explains fast transfer of hyperparameters across model scales.
problem Understanding and optimizing hyperparameters for large-scale models.
method Developed a conceptual framework for HP transfer across scale, showing fast transfer is equivalent to useful transfer for compute-optimal grid search.
result Fast transfer of hyperparameters is equivalent to useful transfer for compute-optimal grid search, offering asymptotic computational advantage.
Bayesian method mitigates negative transfer in unknown source data.
problem Negative transfer in transfer learning where target performance worsens after source data consideration.
method Proxy-informed robust method for probabilistic transfer learning (PROMPT).
result Negative transfer can be mitigated without prior knowledge of source data.
Paper tackles robust transfer learning with unreliable source data.
problem Challenges in robust transfer learning stemming from ambiguity in Bayes classifiers and weak transferable signals.
method Introduces ambiguity level, proposes Transfer Around Boundary (TAB) model, establishes general theorem.
result Demonstrates efficiency and robustness of TAB model improving classification while avoiding negative transfer.
ART improves transfer learning performance with robust theory and methods.
problem Improving performance of primary tasks using auxiliary data.
method Adaptive Robust Transfer Learning (ART) pipeline with theoretical guarantees.
result ART provides a provable theoretical guarantee for adaptive transfer and robustness.
We extend graph neural networks to transfer performance across different input sizes.
problem Transferability of graph neural networks across varying input dimensions.
method Introduce a general framework for transferability across dimensions, showing it corresponds to continuity in a limit space.
result Transferability of graph neural networks is driven by data and learning task, and can be ensured with design principles.
Defines related tasks for transfer learning using foliations.
problem Lack of a foundational description of related tasks in transfer learning.
method Introduces foliations as a mathematical framework for relatedness between tasks.
result Identifies foliations as a way to represent relatedness in transfer learning.
Survey on negative transfer in machine learning.
problem Negative transfer in transfer learning reduces target domain performance.
method Systematic review of 50+ approaches to mitigate negative transfer.
result Lack of systematic survey on negative transfer.
A new boosting method reduces overfitting and negative transfer in transfer learning.
problem Overfitting and negative transfer in transfer learning.
method Importance sampling in boosting and random-forest based ensemble methodology.
result Performs better than competitive transfer learning methodologies 63% of the time.
The paper investigates what enables successful transfer learning and separates feature reuse from data statistics.
problem Understanding what enables successful transfer learning and identifying the responsible parts of the network.
method Analyzes transfer learning on block-shuffled images to distinguish feature reuse from data statistics.
result Some benefit of transfer learning comes from learning low-level statistics of data, not just feature reuse.
Double descent in transfer learning explained for linear regression problems.
problem Understanding generalization errors in transferring parameters between overparameterized linear regression tasks.
method Analytical characterization of generalization error in terms of transfer learning factors.
result Generalization error follows a two-dimensional double descent trend controlled by transfer learning factors.