Weight Squeezing transfers knowledge from large models to smaller ones, improving performance and speed.
problem Transfer learning and model compression for faster and more efficient training.
method Reparameterization of weights from a large model to a smaller one.
result Weight Squeezing outperforms other methods on GLUE benchmark with faster training.
Improves deep transfer learning by preventing performance degradation.
problem Deep transfer learning can degrade performance when using inappropriate pre-trained weights.
method Proposes a novel strategy to compute new descent directions that preserve regularization effects.
result DTNH strategy improves performance of deep transfer learning tasks by 0.1%--7%.
This paper tackles negative transfer in multi-task learning by introducing class-wise weights.
problem Negative transfer hampers function from achieving optimality in multi-task learning.
method Introduces class-wise weights to drive positive transfer and suppress negative transfer.
result Demonstrates improved performance in multi-task learning by reducing negative transfer.
AdaTrans adapts to feature and sample transfer in high-dimensional regression.
problem High-dimensional linear regression with more features than samples.
method F-AdaTrans and S-AdaTrans methods using fused-penalties and adaptive weights.
result AdaTrans achieves convergence rates close to oracle estimators and near-minimax optimal rates.
Kähler information manifolds for signal filters in weighted Hardy spaces are explored.
problem Developing a geometric framework for signal processing filters in weighted Hardy spaces.
method Introducing weighted Hardy spaces and smooth transformations of transfer functions, demonstrating the Kähler manifold structure.
result The Riemannian geometry of weighted Hardy norms for transfer functions forms a Kähler manifold.
LGV boosts adversarial attacks by improving surrogate models.
problem Improving the transferability of black-box adversarial attacks.
method LGV uses a pretrained surrogate model and multiple weight sets from additional training epochs to generate an effective surrogate ensemble.
result LGV outperforms other test-time transformations by significant margins.
Study improves Bayesian optimisation with ensemble transfer learning.
problem Improving sample efficiency in Bayesian optimisation of expensive functions.
method Empirical analysis of ensemble-based transfer learning methods and pipeline components.
result Two components (warm start initialisation and positive weight constraint) improve transfer learning Bayesian optimisation performance.
Improves transfer learning by weighting importance based on test-over-training density.
problem Distribution shift in training and test data.
method Joint and dynamic importance-predictor estimation, causal mechanism transfer.
result Enhanced transfer learning performance in complex, high-dimensional tasks.
Transfer learning offers little benefit in medical imaging tasks.
problem Understanding transfer learning's impact on medical imaging tasks.
method Evaluation of transfer learning on two large-scale medical imaging tasks.
result Simple, lightweight models perform comparably to ImageNet architectures.
Improved transfer learning method considers target and source data balance.
problem Improving transfer learning performance with varying target and source data.
method Weighted Multisource Tradaboost builds on Multisource Tradaboost, weighting datapoint importance based on data availability.
result The proposed method outperforms the base method as target sample count increases.
Connections between nodes of fully connected neural networks are usually represented by weight matrices. In this article, functional transfer matrices are introduced as alternatives to the weight matrices: Instead of using real weights, a functional transfer matrix uses real functions with trainable parameters to repre…
PAC-Net prunes deep models for better transfer learning.
problem Improving transfer learning with over-parameterized models.
method Prune, Allocate, Calibrate (PAC) approach.
result PAC-Net achieves state-of-the-art performance in inductive transfer learning.
In this paper, we propose a novel learning framework for the problem of domain transfer learning. We map the data of two domains to one single common space, and learn a classifier in this common space. Then we adapt the common classifier to the two domains by adding two adaptive functions to it respectively. In the com…
A new hybrid method improves transfer learning performance.
problem Data scarcity in healthcare applications.
method Probabilistic weighting strategy to fuse source and target domain information.
result Our method outperforms existing instance-based transfer learning approaches.
Proposes batch-weight method to correct mode imbalance in domain adaptation.
problem Mode imbalance between source and target distributions affects unsupervised domain transfer.
method Proposes batch-weight method to re-weight training samples.
result Effective in several image-to-image translation tasks.
We consider the transfer of experience samples (i.e., tuples < s, a, s', r >) in reinforcement learning (RL), collected from a set of source tasks to improve the learning process in a given target task. Most of the related approaches focus on selecting the most relevant source samples for solving the target task, but t…
Paper tackles hypothesis transfer learning for black-box models.
problem Difficult to build universal machine learning models across different institutions.
method Dynamic Knowledge Distillation (dkdHTL) with instance-wise weighting.
result Empirical results show the effectiveness of dkdHTL.
Proposes improved classification via transfer learning with regularized linear discriminant analysis.
problem High dimensionality and small sample sizes lead to poor classification performance.
method Regularized random-effects linear discriminant analysis, combining ridge estimates from target and source models.
result Explicit derivation of asymptotic weights and classification error rates in high-dimensional settings.
Sparse sub-networks win transfer learning tasks.
problem Transfer learning with deep networks.
method Unstructured magnitude pruning to find winning tickets.
result Sparse sub-networks achieve similar or better accuracy than original networks.
This work automates decision tree construction from neural networks.
problem Creating optimal decision tree architectures from neural networks.
method Automatic induction of non-greedy decision trees using weights transfer from neural networks.
result Improved model performance over fixed hyperparameters.
A new framework for federated continual learning reduces interference and improves performance.
problem Learning from a sequence of tasks with limited data from each client.
method Federated Weighted Inter-client Transfer (FedWeIT) framework.
result FedWeIT significantly outperforms existing methods with reduced communication cost.
Paper proposes a novel online transfer learning method to reduce domain discrepancy.
problem Online transfer learning with online distribution discrepancy minimization.
method The method seeks a new feature representation to simultaneously reduce marginal and conditional distribution discrepancies.
result The proposed method outperforms state-of-the-art methods in comprehensive experiments.
Transfer learning framework for fragility modeling under domain shift and class imbalance
problem Data gaps in structural fragility modeling
method Transfer learning
result Improves failure detection and predictive stability in low-data regimes
Gradient-based method prunes large models to create transferable architectures.
problem Creating transferable architectures from large models with limited fine-tuning data.
method Gradient-based algorithm for architecture pruning and subset selection.
result Successfully retrain architectures on new tasks with few fine-tuning data.
The paper improves classification accuracy by leveraging a shared signal across domains in high-dimensional classification.
problem Improving classification accuracy in high-dimensional data with shared signals across domains.
method Transfer learning for linear discriminant analysis, decomposing mean differences into common and domain-specific components.
result Deterministic limits for transfer performance, leading to optimal weights and corrections for bias.
We analyze Gibbs-based transfer learning algorithms using information theory.
problem Understanding the generalization error of transfer learning.
method Information-theoretic analysis focusing on α α α -weighted-ERM and two-stage-ERM. result Exact characterization of generalization behavior using conditional symmetrized KL information.
Paper proposes methods for transfer learning with random coefficient ridge regression.
problem Estimation and prediction in high-dimensional settings with related models.
method Two estimators using weighted sums of ridge estimates from target and source models.
result Explicit expression of estimation and prediction risks derived using random matrix theory.
Behavior Transfer improves reinforcement learning by leveraging pre-trained policies.
problem Efficient transfer of knowledge in reinforcement learning.
method Behavior Transfer (BT) that uses pre-trained policies for exploration.
result BT combined with pre-training leads to better solutions than without pre-training.
Paper classifies Parkinson's disease from speech in three languages using CNNs and transfer learning.
problem Classifying Parkinson's disease from speech in multiple languages.
method Convolutional Neural Networks (CNNs) with transfer learning among Spanish, German, and Czech.
result Transfer learning improves model accuracy by up to 8% and balances specificity-sensitivity.
A novel deep learning method for chemometric data improves performance over transfer learning.
problem Training deep neural networks from chemometric data with varying input sizes.
method Weight sharing in deep convolutional neural networks trained on multiple data sets of different sizes.
result Superior performance compared to transfer learning, especially when training on medium and small data sets.
Model projection transfers convolutional network properties to feedforward networks.
problem Transferring properties between feedforward and convolutional networks.
method Unified node-level framework with tensor-valued activations, model projection.
result Projected CNN nodes inherit GFFN-style trainable structure.
The focus in machine learning has branched beyond training classifiers on a single task to investigating how previously acquired knowledge in a source domain can be leveraged to facilitate learning in a related target domain, known as inductive transfer learning. Three active lines of research have independently explor…
DELTA improves transfer learning by aligning feature maps of target networks.
problem Limited accuracy in fine-tuning pre-trained networks for new tasks.
method DELTA preserves outer layer outputs of target networks through constrained feature maps learned by attention.
result DELTA outperforms state-of-the-art methods in accuracy for new tasks.
This work studies adversarial transferability and proposes ensemble methods to improve robustness.
problem Adversarial transferability in neural networks and its implications for robustness.
method Investigates the effect of various factors on adversarial transferability and proposes ensemble attack methods.
result Transferability is significantly hampered by input quantization and architectural mismatch, but not by initialization.
We introduce a new weight-decay scaling rule to maintain sublayer gains across different widths in modern scale-invariant architectures.
problem In modern scale-invariant architectures, training quickly enters a steady state where normalization layers create backward scale sensitivity, degrading learning-rate transfer.
method We introduce a weight-decay scaling rule for AdamW that preserves sublayer gain across widths by equalizing the effective learning rate.
result Our empirical weight-decay scaling rule λ 2 ∝ d λ_2\propto \sqrt{d} λ 2 ∝ d approximately keeps sublayer gains width invariant, enabling zero-shot transfer of learning rate and weight decay. Enhances trading signals using image analysis and weighted moving averages.
problem Improving price trend trading strategies in financial markets.
method Image-induced importance weights applied to weighted moving averages of trading signals.
result Significant enhancement of price trend trading signals with improved portfolio selection.
RIFLE improves deep transfer learning by reinitializing fully-connected layers.
problem Limited improvement in transfer learning accuracy with pre-trained models on small datasets.
method Re-Initializing fully-connected layers with random scratch during fine-tuning.
result Significant improvement in deep transfer learning accuracy across various datasets.
nGPT learns to transfer learning rates across model dimensions and token horizons.
problem nGPT does not transfer learning rates across model size and token horizon.
method Combining numerical experiments with alignment exponents, a novel nGPT parameterization νGPT is developed.
result νGPT exhibits learning rate transfer across width, depth, and token horizon.
The paper investigates what enables successful transfer learning and separates feature reuse from data statistics.
problem Understanding what enables successful transfer learning and identifying the responsible parts of the network.
method Analyzes transfer learning on block-shuffled images to distinguish feature reuse from data statistics.
result Some benefit of transfer learning comes from learning low-level statistics of data, not just feature reuse.
A novel transfer learning framework combines multiple data sources for PU learning.
problem Challenges in PU learning due to lack of negative labels and data scarcity.
method Model averaging of heterogeneous data sources, including binary labeled, semi-supervised, and PU data.
result Method outperforms other methods in predictive accuracy and robustness, especially under limited labeled data.
CLAW adapts weights to balance continual learning and catastrophic forgetting.
problem Balancing continual learning across multiple tasks without forgetting previous knowledge.
method Probabilistic modelling and variational inference to adaptively share network components.
result CLAW achieves state-of-the-art performance in continual learning benchmarks.
Paper studies transfer learning for nonparametric classification, establishing rates and proposing adaptive classifiers.
problem Transfer learning in nonparametric classification under different distributions.
method Established minimax rates and proposed adaptive classifiers based on weighted K-NN approach.
result Data-driven adaptive classifier achieves near-optimal rates over various parameter spaces.
Paper defines a new distance metric for comparing learning tasks.
problem Comparing difficulty of learning tasks between source and target.
method Information geometry, optimal transport, coupled transfer distance.
result Coupled transfer distance correlates with fine-tuning difficulty.
Transfer Learning (TL) in Deep Neural Networks is gaining importance because in most of the applications, the labeling of data is costly and time-consuming. Additionally, TL also provides an effective weight initialization strategy for Deep Neural Networks . This paper introduces the idea of Adaptive Transfer Learning …
Develops hyperparameter transfer methods for Dense Associative Memories.
problem Challenges in transferring hyperparameters for DenseAMs due to unique architecture and activation functions.
method Derives explicit prescriptions for hyperparameter transfer from small to large models.
result Excellent agreement between theoretical and empirical results.
Transfer learning has achieved promising results by leveraging knowledge from the source domain to annotate the target domain which has few or none labels. Existing methods often seek to minimize the distribution divergence between domains, such as the marginal distribution, the conditional distribution or both. Howeve…
New findings on representation changes in transfer learning.
problem Understanding representation changes in transfer learning.
method Analysis of pre-trained and fine-tuned representations in transfer learning.
result Representation changes mainly occur in the head, but early layers are also important for cross-domain adaptation.
Improves transfer learning performance with Lautum regularization.
problem Improving transfer learning performance with limited labeled data.
method Proposes Lautum information regularization for semi-supervised transfer learning.
result Demonstrates improved transferability of deep neural networks.