arXiv research

New framework explains fast transfer of hyperparameters across model scales.

Extends hyperparameter transfer across model sizes and modules, improving training speed.

New methods improve electricity load forecasting using hierarchical transfer learning.

Transfer learning improves chaotic dynamics predictions with less data.

This paper quantifies hyperparameter transfer and finds embedding layer learning rate is key.

Study shows cliff-learning in transfer learning from foundation models.

Deep learning models are vulnerable to adversarial examples crafted by applying human-imperceptible perturbations on benign inputs. However, under the black-box setting, most existing adversaries often have a poor transferability to attack other defense models. In this work, from the perspective of regarding the advers…

New method improves model robustness against adversarial attacks.

Transfer learning from natural image datasets, particularly ImageNet, using standard large models and corresponding pretrained weights has become a de-facto method for deep learning applications to medical imaging. However, there are fundamental differences in data sizes, features and task specifications between natura…

Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.

Paper tackles entity matching over multi-source data, optimizing alignment and mitigating negative transfer.

We introduce a new weight-decay scaling rule to maintain sublayer gains across different widths in modern scale-invariant architectures.

This work aims to create a large-scale model for critical care time series data.

Time-warping improves RNN transfer learning for diverse time scales.

This work investigates how neural collapse improves transfer learning for large-scale models.

This paper presents novel Gaussian process decentralized data fusion algorithms exploiting the notion of agent-centric support sets for distributed cooperative perception of large-scale environmental phenomena. To overcome the limitations of scale in existing works, our proposed algorithms allow every mobile sensing ag…

This work improves GNN training efficiency by maximizing ego-graph information.

New method upscales models and transfers hyperparameters efficiently.

Optimal scaling found to depend on operator norm across large models and datasets.

Simple methods improve regression transferability estimation.

New optimizers control network width scaling, improving stability and transfer across different model sizes.

VNNs transfer well across datasets for brain age prediction using cortical thickness features.

nGPT learns to transfer learning rates across model dimensions and token horizons.

Many problems in sequential decision making and stochastic control often have natural multiscale structure: sub-tasks are assembled together to accomplish complex goals. Systematically inferring and leveraging hierarchical structure, particularly beyond a single level of abstraction, has remained a longstanding challen…

DD-SP uses ML to improve SP for Lorenz 96 systems, outperforming LR and DD-P.

As a new classification platform, deep learning has recently received increasing attention from researchers and has been successfully applied to many domains. In some domains, like bioinformatics and robotics, it is very difficult to construct a large-scale well-annotated dataset due to the expense of data acquisition …

Researchers in functional neuroimaging mostly use activation coordinates to formulate their hypotheses. Instead, we propose to use the full statistical images to define regions of interest (ROIs). This paper presents two machine learning approaches, transfer learning and selection transfer, that are compared upon their…

Modern deep transfer learning approaches have mainly focused on learning generic feature vectors from one task that are transferable to other tasks, such as word embeddings in language and pretrained convolutional features in vision. However, these approaches usually transfer unary features and largely ignore more stru…

One question central to Reinforcement Learning is how to learn a feature representation that supports algorithm scaling and re-use of learned information from different tasks. Successor Features approach this problem by learning a feature representation that satisfies a temporal constraint. We present an implementation…

Parameters in deep neural networks which are trained on large-scale databases can generalize across multiple domains, which is referred as "transferability". Unfortunately, the transferability is usually defined as discrete states and it differs with domains and network architectures. Existing works usually heuristical…

Unified view on GP transfer learning for Bayesian optimization.

A new autoencoder architecture captures multiscale data.

Develops hyperparameter transfer methods for Dense Associative Memories.

This study examines how the size and alignment of pretraining data affect the performance of large language models on downstream tasks.

Neural planners for RDDL MDPs produce deep reactive policies in an offline fashion. These scale well with large domains, but are sample inefficient and time-consuming to train from scratch for each new problem. To mitigate this, recent work has studied neural transfer learning, so that a generic planner trained on othe…

We propose a novel approach for estimating the difficulty and transferability of supervised classification tasks. Unlike previous work, our approach is solution agnostic and does not require or assume trained models. Instead, we estimate these values using an information theoretic approach: treating training labels as …

Improved transfer learning with expert models, reducing compute and speeding up performance.

We develop new algorithms for estimating heterogeneous treatment effects, combining recent developments in transfer learning for neural networks with insights from the causal inference literature. By taking advantage of transfer learning, we are able to efficiently use different data sources that are related to the sam…

Transfer learning, where a model is first pre-trained on a data-rich task before being fine-tuned on a downstream task, has emerged as a powerful technique in natural language processing (NLP). The effectiveness of transfer learning has given rise to a diversity of approaches, methodology, and practice. In this paper, …

The study addresses negative transfer in multi-output Gaussian processes by proposing latent structures.

Convolutional Neural Networks (ConvNets) are commonly developed at a fixed resource budget, and then scaled up for better accuracy if more resources are available. In this paper, we systematically study model scaling and identify that carefully balancing network depth, width, and resolution can lead to better performan…

PAC-Net prunes deep models for better transfer learning.

Develops methods to measure and set function-space learning rates in neural networks.

Time series anomaly detection plays a critical role in automated monitoring systems. Most previous deep learning efforts related to time series anomaly detection were based on recurrent neural networks (RNN). In this paper, we propose a time series segmentation approach based on convolutional neural networks (CNN) for …

Novel framework for systemic risk analysis in financial markets.

New statistical guarantees for transfer learning with diverse tasks.

Transfer learning adapted for hybrid classical-quantum neural networks.

New method $μP^2$ improves neural network training by scaling perturbations layerwise.