Study shows how pretraining robustness transfers to downstream tasks.
problem Understanding how robustness is transferred from pretraining to downstream tasks.
method Theoretical analysis and practical validation of robustness constraints.
result Robustness of a linear predictor on downstream tasks can be constrained by the robustness of its underlying representation.
Enhances performance on downstream tasks using multi-domain data.
problem Improving performance on tasks with limited downstream data.
method Deep transfer learning framework leveraging shared and domain-specific features.
result Significantly improves convergence rate for learning Lipschitz functions.
Optimal feature transfer identified through bias-variance analysis.
problem Optimizing feature transfer in transfer learning.
method Simple linear model with fine-grained bias-variance decomposition.
result Optimal pretrained feature transform is naturally sparse.
UBM transfers bias mitigation from upstream to downstream tasks efficiently.
problem Bias in fine-tuned language models across various tasks.
method Apply bias mitigation to an upstream model, then fine-tune a downstream model on this mitigated model.
result UBM effects transfer to new downstream tasks, creating less biased models.
This work investigates how neural collapse improves transfer learning for large-scale models.
problem Improving transfer learning for large-scale models with limited labeled data.
method Investigates neural collapse and develops a fine-tuning method using skip-connections.
result Feature collapse on downstream data correlates with higher transfer accuracy.
Adversarially robust models transfer better than standard models in image classification.
problem Improving transfer learning performance in image classification.
method Focused on adversarially robust ImageNet classifiers, compared to standard models.
result Adversarially robust models yield improved accuracy on downstream classification tasks.
This study examines how the size and alignment of pretraining data affect the performance of large language models on downstream tasks.
problem Understanding how the size and alignment of pretraining data impact the performance of large language models on downstream tasks.
method Investigated the scaling behavior of large language models in a transfer learning setting, focusing on machine translation tasks.
result The size of the finetuning dataset and the distribution alignment between pretraining and downstream data significantly influence the scaling behavior of downstream performance.
Training a source model optimally for its own task is suboptimal for downstream transfer.
problem The optimality of a source model for its own task hinders downstream transfer performance.
method Analyzes L2-SP ridge regression, characterizes transfer-optimal source penalty, and identifies alignment-dependent effects.
result Transfer benefits from stronger source regularization when aligned imperfectly, and from weaker regularization when aligned perfectly.
Fine-tunes GNNs by preserving generative patterns to improve transferability.
problem Vanilla fine-tuning fails due to structural divergence between pre-training and downstream graphs.
method G-Tuning, which reconstructs the generative patterns of the downstream graph using graphon bases.
result G-Tuning achieves an average improvement of 0.5% and 2.6% on in-domain and out-of-domain transfer learning experiments.
New theory explains when pre-trained models can improve downstream tasks.
problem Lack of theoretical understanding of task similarity for transfer learning.
method Feature-centric viewpoint, theoretical results on transferability phase diagram.
result Transfer learning outperforms training from scratch when target task is well represented in feature space.
Paper proposes a unified time series forecasting model with adaptive transfer.
problem General forecasting models for diverse time series data.
method Unified representations through Decomposed Frequency Learning and adaptive domain-specific features via Time Series Register.
result State-of-the-art forecasting performance on seven real-world benchmarks.
Ensemble pre-trained models for low data transfer learning.
problem Training good models from scratch in low data regime.
method Fine-tune nearest-neighbour ranked pre-trained models to create ensembles.
result Achieves state-of-the-art performance with lower inference budget.
Study shows cliff-learning in transfer learning from foundation models.
problem Data-scaling of transfer learning from foundation models in low data regimes.
method Investigation of cliff-learning phenomenon through foundation-model analysis and toy models.
result Cliff-learning reflects compatibility between priors and tasks.
URL benchmark evaluates uncertainty quantification in pretrained models.
problem Need for reliable uncertainty estimates in transferable pretrained models.
method Proposes URL benchmark to measure transferability of representations and uncertainty estimates.
result Transferable uncertainty quantification remains challenging but not contradictory to traditional goals.
New findings show BERT subnetworks can train independently and transfer to various tasks.
problem Finding smaller subnetworks that can train independently and transfer to other tasks.
method Examined pre-trained BERT models for subnetworks that can train independently and transfer to various downstream tasks.
result Found subnetworks at 40% to 90% sparsity that can train independently and transfer to various tasks.
The study analyzes transfer learning in infinite-width neural networks, improving generalization on target tasks.
problem Improving generalization in neural networks when using pretraining on a source task.
method Developed a theory under gradient flow for infinitely wide networks, analyzing fine-tuning and joint pretraining.
result Summary statistics of randomly initialized networks after pretraining are adaptive kernels that depend on both source and target data.
Recent works reveal that network embedding techniques enable many machine learning models to handle diverse downstream tasks on graph structured data. However, as previous methods usually focus on learning embeddings for a single network, they can not learn representations transferable on multiple networks. Hence, it i…
Develops a method to ensure accuracy of few-shot transfer learning models.
problem Lack of generalization guarantees for low-data transfer learning.
method Trains a distribution over PEFT parameters using upstream tasks and samples plausible PEFTs for downstream tasks.
result Demonstrates non-vacuous generalization guarantees compared to existing methods in the low-shot regime.
In recent years, deep learning models have shown great potential in source code modeling and analysis. Generally, deep learning-based approaches are problem-specific and data-hungry. A challenging issue of these approaches is that they require training from starch for a different related problem. In this work, we propo…
This paper explores the connection between adversarial and knowledge transferability.
problem Understanding the factors affecting knowledge transferability.
method Theoretical analysis and practical metrics for adversarial transferability.
result Adversarial transferability and knowledge transferability are closely related.
New method learns robust meta-representations for fast task adaptation.
problem Transferability of meta-representations in fast adaptation.
method Decoupled encoder-decoder approach with contrastive objective.
result Meta-representations improve downstream performance and noise robustness.
Study shows scaling up models doesn't always improve downstream tasks.
problem Understanding why scaling up models doesn't always improve downstream performance.
method Systematic study of 4800 experiments on various models, analyzing performance on 20 downstream tasks.
result Performance on downstream tasks saturates as model size increases, revealing a nonlinear relationship.
Proposes a new metric to quantify the difference between neural network representations based on downstream task performance.
problem The lack of a consistent metric to measure the difference between neural network representations.
method Introduced the Transferred Discrepancy (TD) metric, which evaluates the difference between representations based on their performance on downstream tasks.
result TD provides fine-grained information for various downstream tasks and can evaluate the effectiveness of different training strategies.
Study finds CLIP's caption-based learning outperforms image-only methods under certain conditions.
problem Comparing CLIP's performance with traditional image-only methods in learning transferable representations.
method Controlled comparison of CLIP and image-only methods using a dataset with descriptive captions and specific criteria.
result CLIP's caption-based learning outperforms image-only methods when certain conditions are met, but can be detrimental in others.
Modern deep transfer learning approaches have mainly focused on learning generic feature vectors from one task that are transferable to other tasks, such as word embeddings in language and pretrained convolutional features in vision. However, these approaches usually transfer unary features and largely ignore more stru…
MicAugment transfers audio style from few seconds of input to match target conditions.
problem Audio model robustness to diverse acquisition conditions.
method Identifies and applies transformations learned from target audio to input audio.
result MicAugment significantly improves model robustness in downstream tasks.
RID framework quantifies and regularizes task-relevant knowledge in distillation.
problem Distilling irrelevant information can hinder student model performance.
method Partial Information Decomposition to quantify and regularize task-relevant knowledge.
result RID framework leads to more resilient distillation under nuisance teachers.
Neural networks learn patterns in random data, improving downstream performance.
problem Understanding what deep networks learn with random labels.
method Analytical and empirical study of convolutional and fully connected networks pre-trained on random labels.
result Pre-trained networks on random labels transfer faster to real datasets, despite specialization effects.
MMD-B-Fair learns fair representations by minimizing MMD test power.
problem Learning fair representations of data while preserving target attributes.
method Kernel two-sample testing and block testing schemes.
result Minimizing MMD test power allows hiding sensitive attribute information.
Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, …
Improved transfer learning with expert models, reducing compute and speeding up performance.
problem Improving sample efficiency and reducing computational requirements for new tasks.
method Training a diverse set of experts using existing label structures and performance proxies to select the relevant expert for each target task.
result Significant speed-up of 2-3 orders of magnitude compared to competing approaches.
Proposes a transfer learning method for accurate latent variable estimation.
problem Accurate estimation of latent variables in networks with large parameter spaces.
method Leverages information from similar networks to improve estimation accuracy.
result The proposed methods improve estimation accuracy and are validated on real datasets.
Many applications of machine learning require a model to make accurate pre-dictions on test examples that are distributionally different from training ones, while task-specific labels are scarce during training. An effective approach to this challenge is to pre-train a model on related tasks where data is abundant, and…
SASSL improves self-supervised learning by preserving image structure.
problem Distorted augmented samples in self-supervised learning.
method Neural Style Transfer to decouple semantic and stylistic attributes.
result Boosts ImageNet top-1 accuracy by up to 2 percentage points.
The goal of network representation learning is to learn low-dimensional node embeddings that capture the graph structure and are useful for solving downstream tasks. However, despite the proliferation of such methods, there is currently no study of their robustness to adversarial attacks. We provide the first adversari…
Contrastive learning outperforms autoencoders and GANs in feature recovery and downstream tasks.
problem Theoretical understanding of contrastive learning's superiority in feature learning.
method Theoretical analysis of contrastive learning in linear representation settings.
result Contrastive learning outperforms autoencoders and GANs for feature recovery and in-domain downstream tasks.
Transfer learning benefits vary in DRL, affecting representation and policy learning.
problem Understanding how much of DRL's sample complexity is due to representation learning.
method Transfer learning experiments comparing performance on related games.
result Benefits of transfer learning are highly variable and non-symmetric across tasks.
Fine-tuning large pre-trained models is an effective transfer mechanism in NLP. However, in the presence of many downstream tasks, fine-tuning is parameter inefficient: an entire new model is required for every task. As an alternative, we propose transfer with adapter modules. Adapter modules yield a compact and extens…
Behavior Transfer improves reinforcement learning by leveraging pre-trained policies.
problem Efficient transfer of knowledge in reinforcement learning.
method Behavior Transfer (BT) that uses pre-trained policies for exploration.
result BT combined with pre-training leads to better solutions than without pre-training.
CLIP learns joint image-text representations for zero-shot learning.
problem Understanding and improving zero-shot transfer performance in CLIP.
method Formal study of transferrable representation learning and analysis of zero-shot transfer performance.
result Proposes a new CLIP-type approach that outperforms existing methods.
Model projection transfers convolutional network properties to feedforward networks.
problem Transferring properties between feedforward and convolutional networks.
method Unified node-level framework with tensor-valued activations, model projection.
result Projected CNN nodes inherit GFFN-style trainable structure.
Deep tabular models outperform GBDT in medical diagnosis tasks.
problem Transfer learning for tabular data in medical diagnosis.
method Proposes a pseudo-feature method for transfer learning between different feature sets.
result Tabular neural networks outperform GBDT in medical diagnosis tasks.
NCL improves interpretability of deep features by enforcing non-negativity.
problem Lack of interpretability in deep representations.
method Non-negative Contrastive Learning (NCL) using non-negativity constraints.
result NCL outperforms standard contrastive learning in feature disentanglement and selection.
Framework improves target domain prediction using quantile matching.
problem Improving prediction accuracy in data-scarce target domains.
method Conditional quantile matching for distributional alignment.
result Empirical risk minimizer achieves tighter excess risk bound.
The paper proposes a uniformity regularization scheme to improve deep neural network transferability.
problem Improving deep neural network transferability and adaptation to new tasks.
method Introduces a uniformity regularization scheme to encourage high uniformity in embedding space.
result Uniformity regularization consistently offers benefits over baseline methods and achieves state-of-the-art performance in Deep Metric Learning and Meta-Learning.
Paper proposes a new speech representation benchmark and model.
problem Lack of benchmarks for comparing speech representations.
method Unsupervised triplet-loss objective for training a universal non-semantic speech representation.
result Proposed representation outperforms other models on benchmark and transfer learning tasks.
New research shows deep models learn sparse features, limiting transfer learning; ensembling improves performance.
problem Sparse feature learning in deep models limits transfer learning performance.
method Developed a theoretical framework and proposed an ensembling strategy to aggregate multiple models.
result Ensembling yields a 9% improvement in transfer accuracy without extra pretraining cost.
GPT-GNN pre-trains GNNs on unlabeled graphs to improve downstream performance.
problem Training GNNs requires labeled data, which is expensive.
method Generative pre-training of GNNs on unlabeled data with self-supervision.
result GPT-GNN significantly outperforms state-of-the-art GNNs without pre-training.