Optimal transport metric transfers deep network representations efficiently.
problem Efficiently transfer deep network representations for new tasks.
method Use optimal transport to quantify representation similarity and regularize student network.
result Optimal transport distance promotes similarity between teacher and student representations.
Study shows how deep network representations can be transferred between datasets and tasks.
problem Transferability of deep network representations across datasets and tasks.
method Examined layer-wise transferability of representations in deep networks across multiple datasets and tasks.
result Interesting empirical observations on layer-wise transferability of representations.
Transfer knowledge from multiple sources to improve matrix completion.
problem Matrix completion with noisy data.
method Aggregating singular subspaces information from multiple sources to solve a two-way PCA problem and transform into a low-dimensional linear regression.
result Guaranteed statistical efficiency in transforming the high-dimensional target matrix completion problem.
URL benchmark evaluates uncertainty quantification in pretrained models.
problem Need for reliable uncertainty estimates in transferable pretrained models.
method Proposes URL benchmark to measure transferability of representations and uncertainty estimates.
result Transferable uncertainty quantification remains challenging but not contradictory to traditional goals.
LEEP measures transferability of learned representations efficiently.
problem Evaluating the transferability of learned representations in machine learning.
method LEEP: Log Expected Empirical Prediction, a simple measure requiring one pass through the target data set.
result LEEP predicts transfer and meta-transfer learning performance and convergence speed, outperforming existing measures.
New findings on representation changes in transfer learning.
problem Understanding representation changes in transfer learning.
method Analysis of pre-trained and fine-tuned representations in transfer learning.
result Representation changes mainly occur in the head, but early layers are also important for cross-domain adaptation.
Proposes a new KT technique for deep representation learning.
problem Efficient knowledge transfer for representation learning tasks.
method Trains a student model maintaining mutual information with teacher model.
result Outperforms existing KT techniques and allows new applications.
Transfer learning benefits vary in DRL, affecting representation and policy learning.
problem Understanding how much of DRL's sample complexity is due to representation learning.
method Transfer learning experiments comparing performance on related games.
result Benefits of transfer learning are highly variable and non-symmetric across tasks.
TimbreTron transfers musical timbre using CQT and WaveNet.
problem Transfer musical timbre while preserving pitch, rhythm, and loudness.
method Apply image domain style transfer to CQT representation, then generate high-quality waveform with WaveNet.
result TimbreTron recognizably transfers timbre while preserving musical content.
The paper tackles transfer learning for growing matrix representations, improving estimation accuracy.
problem Structured matrix estimation under growing ambient dimensions and latent representations.
method Proposes a general transfer framework decomposing target parameters into embedded source components, low-rank innovations, and sparse edits. Develops an anchored alternating projection estimator.
result Establishes deterministic error bounds that separate target noise, representation growth, and source estimation error, yielding improved rates.
Successor Features improve transfer in RL by decoupling feature and reward.
problem Improving feature representation for task transfer in reinforcement learning.
method Decouples feature representation from reward function, allowing domain transfer.
result Advantages and limitations of Successor Features for transfer identified.
Enhances source domain knowledge with target data for transfer learning.
problem Limited data in target domains and rigid model assumptions in transfer learning.
method Transfer learning through Enhanced Sufficient Representation (TESR).
result TESR enhances source domain knowledge with target data, improving transfer learning performance.
Optimal feature transfer identified through bias-variance analysis.
problem Optimizing feature transfer in transfer learning.
method Simple linear model with fine-grained bias-variance decomposition.
result Optimal pretrained feature transform is naturally sparse.
Paper improves sample efficiency of transfer learning in diffusion models.
problem Diffusion models need too much data to train from scratch.
method Assumes shared low-dimensional representation across tasks for improved sample efficiency.
result Sample complexity of target tasks can be reduced with a well-learned representation.
Learn generic latent relational graphs for better transfer learning.
problem Lack of transferable structured graphical representations in deep transfer learning.
method Unsupervisedly learned latent relational graphs from unlabeled data.
result Improves performance on various downstream tasks.
Paper proposes embedding framework to efficiently represent IDs in e-commerce.
problem Efficient representation of IDs with consideration of relationships among different types of IDs.
method Embedding based framework using item ID sequences and structural connections.
result Effective representation and transfer of IDs across different scenarios in e-commerce.
Method transfers feature representation from large to small models using perception coherence.
problem Transfer feature representation from large to small models.
method Defines perception coherence, proposes loss function to minimize.
result Method outperforms or achieves on-par performance compared to strong baseline methods.
A new method for faster learning in reinforcement learning.
problem Learning from multiple tasks with different goals.
method Universal Successor Representations (USR) and USR Approximator (USRA).
result Agents initialized with USRA trained on USR can achieve goals faster than random initialization.
New method improves knowledge transfer from large to small networks.
problem Improving knowledge transfer from large to small neural networks.
method Contrastive learning to capture more structural knowledge.
result Contrastive learning outperforms knowledge distillation on various transfer tasks.
Model Features improve transfer in reinforcement learning by clustering states.
problem Improving knowledge transfer between tasks with shared transition dynamics.
method Introduces Model Features, a feature representation that clusters behaviourally equivalent states.
result Learning Successor Features is equivalent to learning a Model-Reduction.
HUBERT combines BERT's structure with TPRs to improve NLP task transfer.
problem Improving transferability among NLP tasks.
method Combines BERT's bidirectional Transformer structure with TPRs to learn shared data structures.
result HUBERT outperforms BERT on GLUE and HANS datasets, showing better transferability.
Improved transfer learning for MOOCs using auto-encoders.
problem Poor prediction performance of models trained on historical MOOC data.
method Automated transductive transfer learning with auto-encoders.
result Improved transferability of dropout prediction across similar and dissimilar MOOCs.
Paper explores zero-shot cross-lingual reading comprehension using pre-trained multi-lingual model.
problem Lack of training data for every language in reading comprehension tasks.
method Systematic exploration of zero-shot cross-lingual transfer learning with a multi-lingual language representation model.
result Zero-shot cross-lingual transfer learning is feasible and translating source data into target language is not necessary.
Proposes KTAN for better training of student networks with both intermediate representations and probability distributions.
problem Reduces large computation and storage cost of deep networks by transferring generalization ability.
method Holistically considers intermediate representations and probability distributions; uses a Teacher-to-Student layer and adversarial learning.
result Significantly improves performance of student networks on image classification and object detection tasks.
Bayesian optimisation learns hyperparameters faster with task transfer.
problem Bayesian optimisation requires initial exploration even for similar tasks.
method Transfer information across tasks using learnt representations of training datasets.
result Joint Gaussian process model on hyperparameters and data representations results in faster convergence.
Anti-transfer learning prevents misleading representations for speech tasks.
problem Misleading representations learned from orthogonal tasks in speech processing.
method Penalizes similarity between activations of a network and another trained on an orthogonal task.
result Improves classification accuracy and invariance to the orthogonal task.
Improved transfer learning with expert models, reducing compute and speeding up performance.
problem Improving sample efficiency and reducing computational requirements for new tasks.
method Training a diverse set of experts using existing label structures and performance proxies to select the relevant expert for each target task.
result Significant speed-up of 2-3 orders of magnitude compared to competing approaches.
Deep features transfer well across datasets, improving generalization and training speed.
problem Understanding why deep features transfer well across different datasets.
method Analyzed transferability from improved generalization, optimization, and feasibility perspectives.
result Transferred models find flatter minima and have more favorable loss landscapes, leading to better training.
Advances fair representation learning for unknown third-party uses.
problem Mitigating unfair prediction outcomes when representations are used by third parties with unknown objectives.
method Adversarial representation learning to ensure fairness.
result Demonstrated fair transfer learning and maintained utility.
Adversarial transfer learning improves stress assessment across users.
problem Transfer learning challenges in physiological biosignals.
method Disentangled nuisance-robust representations using adversarial networks.
result Adversarial framework enhances cross-subjects stress assessment.
The paper develops methods to create fair and transferable representations without subgroup discrimination.
problem Creating fair and transferable representations without discriminating subgroups in the population.
method The approach involves modifying data representations to meet fairness constraints, leveraging task similarities via low rank matrix factorization.
result The learned fair representation transfers well to novel tasks, improving prediction performance and fairness metrics.
Proposes a novel framework for unsupervised domain adaptation using causal representations.
problem Transferability of deep model representations across domains is limited.
method Integrates causal inference into deep learning pipeline for domain-invariant feature learning.
result Demonstrates superior performance in unsupervised domain adaptation using causal representations.
Efficiently learns representations across domains and tasks with few labels.
problem Learning representations that generalize across different domains and tasks with limited labeled data.
method Combines domain adversarial loss and metric learning for representation transfer. Optimizes on both labeled and unlabeled data in the target domain.
result Significantly outperforms fine-tuning on novel classes in new domains with few labeled examples.
CLIP learns joint image-text representations for zero-shot learning.
problem Understanding and improving zero-shot transfer performance in CLIP.
method Formal study of transferrable representation learning and analysis of zero-shot transfer performance.
result Proposes a new CLIP-type approach that outperforms existing methods.
New statistical guarantees for transfer learning with diverse tasks.
problem Statistical guarantees for transfer learning across different tasks.
method Formal analysis of t+1 tasks with shared representation. result Sample complexity reduction for new tasks using shared representation.
Improved vision-language embeddings boost cross-task learning.
problem Creating general vision systems with better cross-task learning.
method Aligning image-word representations for better cross-task transfer.
result Improved inductive transfer from visual recognition to visual question answering.
New research shows deep models learn sparse features, limiting transfer learning; ensembling improves performance.
problem Sparse feature learning in deep models limits transfer learning performance.
method Developed a theoretical framework and proposed an ensembling strategy to aggregate multiple models.
result Ensembling yields a 9% improvement in transfer accuracy without extra pretraining cost.
STM learns robust features by transferring manifold structure from data to feature space.
problem Representation learning in unknown distribution data.
method Structure Transfer Machine (STM) method that learns features with manifold structure regularization.
result STM achieves better results on various benchmarks compared to state-of-the-art CNNs.
The paper uses thermodynamics to improve machine learning representation quality.
problem Improving the quality of learned representations for transfer learning.
method Formal connection with thermodynamics, iso-classification process, traversing the equilibrium surface.
result Demonstrates how to transfer representations while keeping classification loss constant.
Paper tackles label noise in transfer learning, proposing a new framework.
problem Label noise in source domain affects learning of invariant representations and correcting label shift.
method Proposes Denoising Conditional Invariant Component (DCIC) framework to handle noisy labels.
result Proposed framework ensures extraction of invariant representations and unbiased estimation of target domain labels.
Model learns multiple tasks using visual and textual representations.
problem Training visual navigation agents for multiple tasks.
method Dual-Attention unit for task-invariant alignment of visual and textual representations.
result Model outperforms baselines on semantic goal navigation and embodied question answering.
Proposes a method to learn fair and transferable representations.
problem Learning fair and transferable data representations.
method Information-theoretic objective with expressiveness-fairness trade-off control.
result Demonstrates higher expressiveness at lower computational cost compared to existing methods.
Paper proposes a new speech representation benchmark and model.
problem Lack of benchmarks for comparing speech representations.
method Unsupervised triplet-loss objective for training a universal non-semantic speech representation.
result Proposed representation outperforms other models on benchmark and transfer learning tasks.
Paper proposes a unified time series forecasting model with adaptive transfer.
problem General forecasting models for diverse time series data.
method Unified representations through Decomposed Frequency Learning and adaptive domain-specific features via Time Series Register.
result State-of-the-art forecasting performance on seven real-world benchmarks.
MetaVAE learns transferable latent representations across related distributions.
problem Generative models struggle to adapt to new distributions.
method Doubly-amortized variational inference sharing computation across related models.
result MetaVAE significantly outperforms baselines on image classification tasks.
Decouples RL components for better task transfer and robustness.
problem Poor generalization in reinforcement learning to task domain changes.
method Separates learning into task representation, dynamics, and reward.
result Decoupling improves performance and robustness to domain changes.
Enhances transfer learning with semantic reasoning for robust predictions.
problem Improving robustness of transfer learning models.
method Integrates semantic representations for better knowledge transfer.
result Demonstrated robustness in bus delay and air quality forecasting.
This study investigates how much knowledge from natural images can be transferred to pathology images.
problem Quantifying how much knowledge from natural images can be transferred to pathology images.
method Proposes a framework to quantify knowledge gain by a particular layer, conducts empirical investigation in pathology image centered transfer learning.
result Early layers of deep models can transfer knowledge to pathology image classification tasks.