Recent years, transfer learning has attracted much attention in the community of machine learning. In this paper, we mainly focus on the tasks of parameter transfer under the framework of extreme learning machine (ELM). Unlike the existing parameter transfer approaches, which incorporate the source model information in…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “parameter transfer”
We consider a transfer-learning problem by using the parameter transfer approach, where a suitable parameter of feature mapping is learned through one task and applied to another objective task. Then, we introduce the notion of the local stability and parameter transfer learnability of parametric feature mapping,and th…
Double Double Descent: On Generalization Errors in Transfer Learning between Linear Regression Taskscs.LG
Double descent in transfer learning explained for linear regression problems.
problem Understanding generalization errors in transferring parameters between overparameterized linear regression tasks.
method Analytical characterization of generalization error in terms of transfer learning factors.
result Generalization error follows a two-dimensional double descent trend controlled by transfer learning factors.
Parameters in deep neural networks which are trained on large-scale databases can generalize across multiple domains, which is referred as "transferability". Unfortunately, the transferability is usually defined as discrete states and it differs with domains and network architectures. Existing works usually heuristical…
We introduce two approaches for combining neural evolution strategy (NES) and proximal policy optimization (PPO): parameter transfer and parameter space noise. Parameter transfer is a PPO agent with parameters transferred from a NES agent. Parameter space noise is to directly add noise to the PPO agent`s parameters. We…
Paper proposes a statistical test for transfer learning in linear regression.
problem Theoretical framework for parameter transfer in linear regression.
method Developed a statistical test to predict transfer quality.
result The test can predict if a fine-tuned model has lower prediction risk.
Paper tackles high-dimensional quantile regression with distribution shift using transfer learning.
problem Efficiency of knowledge transfer is severely impacted by distribution shift in high-dimensional regression.
method Proposes a novel transferable set and framework for three types of distribution shift: parameter, covariate, and residual.
result Establishes estimation error bounds and source detection consistency for the proposed method.
Fine-tuning large pre-trained models is an effective transfer mechanism in NLP. However, in the presence of many downstream tasks, fine-tuning is parameter inefficient: an entire new model is required for every task. As an alternative, we propose transfer with adapter modules. Adapter modules yield a compact and extens…
Many tasks in natural language understanding require learning relationships between two sequences for various tasks such as natural language inference, paraphrasing and entailment. These aforementioned tasks are similar in nature, yet they are often modeled individually. Knowledge transfer can be effective for closely …
FiT combines transfer and meta-learning for efficient few-shot image classification.
problem Few-shot image classification in personalized and federated learning settings.
method Combines transfer learning and meta-learning with fixed pretrained backbones and fine-tuned FiLM adapter layers.
result Achieves state-of-the-art accuracy on VTAB-1k benchmark with fewer than 1% of updateable parameters.
Patients whose transfer to the Intensive Care Unit (ICU) is unplanned are prone to higher mortality rates than those who were admitted directly to the ICU. Recent advances in machine learning to predict patient deterioration have introduced the possibility of \emph{proactive transfer} from the ward to the ICU. In this …
We investigate unification of two systems of identical elements having different dimensions which may be of interest for both physics and economics. Characteristic parameters as well as explicit formulae for the temperature (in economics - capital turnover) and dimension of the united system are obtained as functions o…
Paper proposes transferring neural network learning trajectories.
problem Training deep neural networks is computationally expensive.
method Formulated and solved the learning transfer problem by matching gradients.
result Transferred parameters achieve non-trivial accuracy before training.
The paper investigates what enables successful transfer learning and separates feature reuse from data statistics.
problem Understanding what enables successful transfer learning and identifying the responsible parts of the network.
method Analyzes transfer learning on block-shuffled images to distinguish feature reuse from data statistics.
result Some benefit of transfer learning comes from learning low-level statistics of data, not just feature reuse.
New algorithms improve multi-task bandit performance by transferring reward samples.
problem Sequential multi-task bandit problems with adjacent similar tasks.
method Two UCB-based algorithms that transfer reward samples between tasks.
result Transfer of reward samples reduces overall regret compared to no transfer.
We introduce a novel method that enables parameter-efficient transfer and multi-task learning with deep neural networks. The basic approach is to learn a model patch - a small set of parameters - that will specialize to each task, instead of fine-tuning the last layer or the entire network. For instance, we show that l…
Paper proposes SOTL framework for improving transfer learning accuracy and efficiency.
problem Statistical bias and computational efficiency in multi-source domain adaptation.
method Sparse Optimization for Transfer Learning (SOTL) with L0-regularization.
result SOTL significantly improves estimation accuracy and computational speed, especially under adversarial conditions.
Estimating Posterior Ratio for Classification: Transfer Learning from Probabilistic Perspectivestat.ML
Transfer learning assumes classifiers of similar tasks share certain parameter structures. Unfortunately, modern classifiers uses sophisticated feature representations with huge parameter spaces which lead to costly transfer. Under the impression that changes from one classifier to another should be ``simple'', an effi…
In complex transfer learning scenarios new tasks might not be tightly linked to previous tasks. Approaches that transfer information contained only in the final parameters of a source model will therefore struggle. Instead, transfer learning at a higher level of abstraction is needed. We propose Leap, a framework that …
State-of-the-art named entity recognition (NER) systems have been improving continuously using neural architectures over the past several years. However, many tasks including NER require large sets of annotated data to achieve such performance. In particular, we focus on NER from clinical notes, which is one of the mos…
Enhances VAR model estimation using transfer learning.
problem Estimating high-dimensional VAR models with temporal dependencies.
method Transfer learning for VAR models with low-rank and sparse structures.
result Theoretical guarantees for model parameter consistency and informative set selection.
Background: Pharmacokinetic evaluation is one of the key processes in drug discovery and development. However, current absorption, distribution, metabolism, excretion prediction models still have limited accuracy. Aim: This study aims to construct an integrated transfer learning and multitask learning approach for deve…
Framework reuses pre-trained models for data-free transfer learning.
problem Challenges in retrieving source data for model training.
method Model Recycling Framework for parameter-efficient training.
result Makes multi-source data-free supervised transfer learning possible.
Transfer learning framework for fragility modeling under domain shift and class imbalance
problem Data gaps in structural fragility modeling
method Transfer learning
result Improves failure detection and predictive stability in low-data regimes
Deep neural networks trained on a wide range of datasets demonstrate impressive transferability. Deep features appear general in that they are applicable to many datasets and tasks. Such property is in prevalent use in real-world applications. A neural network pretrained on large datasets, such as ImageNet, can signifi…
Parameter-transfer is a well-known and versatile approach for meta-learning, with applications including few-shot learning, federated learning, and reinforcement learning. However, parameter-transfer algorithms often require sharing models that have been trained on the samples from specific tasks, thus leaving the task…
Extends hyperparameter transfer across model sizes and modules, improving training speed.
problem Training stability and performance of large-scale models with optimal hyperparameters.
method Complete Parameterisation, per-module hyperparameter optimisation and transfer.
result Hyperparameter transfer holds even in the per-module hyperparameter regime, improving training speed.
LOL-GP model improves surrogate modeling of expensive simulators.
problem Costly computer simulations for complex systems.
method Local transfer learning Gaussian process.
result Improved surrogate performance over existing methods.
We develop DTs for PDE models using KL-NN and TL, analyzing TL's moment equations and one-shot learning for exactness.
problem Creating accurate digital twins for systems governed by PDEs under changing conditions.
method We use KL-NN surrogate models and transfer learning to construct DTs, analyzing the moment equations and proposing one-shot and few-shot learning methods.
result For linear PDEs, one-shot TL is exact; for nonlinear PDEs, some parameters can be transferred with minimal error.
KM method reduces ConvNet parameters to 9% higher accuracy with minimal additional memory.
problem Expensive memory usage for training ConvNets on embedded devices.
method Kernel Modulation (KM) method that adapts all network parameters for each task.
result KM delivers up to 9% higher accuracy than other parameter-efficient methods.
Single neural network predicts ImageNet model parameters for faster training.
problem Training diverse ImageNet models requires significant resources and time.
method Trained a neural network to predict ImageNet model parameters and used them for initialization.
result Models initialized with predicted parameters converge faster and achieve competitive performance.
Bayesian optimization adapts domain parameters for more robust robot policies.
problem Learning policies for robot control from simulation data often fails in the real world due to the 'reality gap'.
method Bayesian Domain Randomization (BayRn) uses Bayesian optimization to adapt domain parameter distributions during training.
result BayRn achieves better sim-to-real transfer compared to fixed distribution methods.
Due to the widespread use of positron emission tomography (PET) in clinical practice, the potential risk of PET-associated radiation dose to patients needs to be minimized. However, with the reduction in the radiation dose, the resultant images may suffer from noise and artifacts that compromise diagnostic performance.…
Bayesian active learning tackles nuisance parameters, leading to bias and dilemmas.
problem Bayesian active learning with nuisance parameters leads to bias and dilemmas.
method Characterizes and mitigates negative interference by accurately estimating nuisance parameters.
result The extent of negative interference can be extremely large, and accurate estimation of nuisance parameters is critical.
Learning from small amounts of labeled data is a challenge in the area of deep learning. This is currently addressed by Transfer Learning where one learns the small data set as a transfer task from a larger source dataset. Transfer Learning can deliver higher accuracy if the hyperparameters and source dataset are chose…
A new framework for federated continual learning reduces interference and improves performance.
problem Learning from a sequence of tasks with limited data from each client.
method Federated Weighted Inter-client Transfer (FedWeIT) framework.
result FedWeIT significantly outperforms existing methods with reduced communication cost.
Model projection transfers convolutional network properties to feedforward networks.
problem Transferring properties between feedforward and convolutional networks.
method Unified node-level framework with tensor-valued activations, model projection.
result Projected CNN nodes inherit GFFN-style trainable structure.
Bayesian transfer learning improves predictive performance with limited source data.
problem Improving statistical procedures with limited target and source datasets.
method Total risk prior for joint parameter distribution, Bayesian Lasso, model averaging, Gibbs sampling.
result Superior predictive performance compared to frequentist baseline, especially with limited source data.
The paper studies practical estimation and interpretation of Rényi transfer entropy.
problem Challenges in accurately estimating and interpreting Rényi transfer entropy.
method Systematic study of k-nearest neighbor estimator for Rényi entropy and transfer entropy.
result Effective estimates of effective Rényi transfer entropy can accurately capture directional information flow.
Develops a method to ensure accuracy of few-shot transfer learning models.
problem Lack of generalization guarantees for low-data transfer learning.
method Trains a distribution over PEFT parameters using upstream tasks and samples plausible PEFTs for downstream tasks.
result Demonstrates non-vacuous generalization guarantees compared to existing methods in the low-shot regime.
Pre-training a deep neural network on the ImageNet dataset is a common practice for training deep learning models, and generally yields improved performance and faster training times. The technique of pre-training on one task and then retraining on a new one is called transfer learning. In this paper we analyse the eff…
Transfer learning for bandits with latent Lipschitz continuity.
problem Learning to transfer structural information from prior tasks to new tasks.
method Proposes a framework to estimate Lipschitz constant from prior tasks and apply it to new tasks.
result Regret bound close to oracle algorithm with full knowledge of Lipschitz constant under mild assumptions.
Paper tackles transfer learning for contextual multi-armed bandits under covariate shift.
problem Nonparametric contextual multi-armed bandits with covariate shift.
method Established minimax rate of convergence, proposed transfer learning algorithm.
result Achieved near-optimal statistical guarantees for learning in target domain.
Transfer learning has received a lot of attention in the machine learning community over the last years, and several effective algorithms have been developed. However, relatively little is known about their theoretical properties, especially in the setting of lifelong learning, where the goal is to transfer information…
A new boosting method reduces overfitting and negative transfer in transfer learning.
problem Overfitting and negative transfer in transfer learning.
method Importance sampling in boosting and random-forest based ensemble methodology.
result Performs better than competitive transfer learning methodologies 63% of the time.
New estimator achieves minimax optimal risk in transfer learning.
problem Nonparametric regression with transfer learning.
method Confidence thresholding estimator and data-driven adaptive algorithm.
result Adaptive algorithm achieves minimax risk up to a logarithmic factor.
Transfer learning is a recent field of machine learning research that aims to resolve the challenge of dealing with insufficient training data in the domain of interest. This is a particular issue with traditional deep neural networks where a large amount of training data is needed. Recently, StochasticNets was propose…
Study geometric structures in transfer learning to avoid negative transfer.
problem Understanding information-theoretic limits of transfer learning without exploiting domain geometry.
method Integrates geometric structure into linear regression models, using Gram matrices of source and target domains.
result Proposes an interpolation estimator that matches minimax lower bound and outperforms existing methods.