Researchers develop method to protect against 'weight poisoning' attacks on pre-trained models.
problem The security threat of downloading untrusted pre-trained weights that can be manipulated after fine-tuning.
method RIPPLe regularization method and Embedding Surgery initialization procedure.
result Demonstrated that weight poisoning attacks are possible even with limited knowledge of the dataset and fine-tuning procedure.
Improved LLM pre-training performance through better weight and variance control.
problem Improper weight and variance control in LLM pre-training affects downstream task performance.
method Introduced Layer Index Rescaling (LIR) and Target Variance Rescaling (TVR) techniques.
result Substantial improvements in downstream task performance (up to 4.6%) and reduced extreme activation values.
Mimetic initialization improves Transformer training on small datasets.
problem Difficulty in training Transformers on small datasets.
method Initialize self-attention layers to look like pre-trained models.
result Vanilla Transformers trained with mimetic initialization achieve higher accuracy.
Behavior Transfer improves reinforcement learning by leveraging pre-trained policies.
problem Efficient transfer of knowledge in reinforcement learning.
method Behavior Transfer (BT) that uses pre-trained policies for exploration.
result BT combined with pre-training leads to better solutions than without pre-training.
Improves deep transfer learning by preventing performance degradation.
problem Deep transfer learning can degrade performance when using inappropriate pre-trained weights.
method Proposes a novel strategy to compute new descent directions that preserve regularization effects.
result DTNH strategy improves performance of deep transfer learning tasks by 0.1%--7%.
ImageNet pre-training has been regarded as essential for training accurate object detectors for a long time. Recently, it has been shown that object detectors trained from randomly initialized weights can be on par with those fine-tuned from ImageNet pre-trained models. However, the effects of pre-training and the diff…
Deep Reinforcement Learning (DRL) algorithms are known to be data inefficient. One reason is that a DRL agent learns both the feature and the policy tabula rasa. Integrating prior knowledge into DRL algorithms is one way to improve learning efficiency since it helps to build helpful representations. In this work, we co…
Our method upweights easy samples to mitigate forgetting in fine-tuning.
problem Catastrophic forgetting in fine-tuning pre-trained models.
method Sample weighting based on pre-trained model's losses.
result Our method reduces forgetting by up to 0.8% on MetaMathQA while preserving more accuracy on pre-training datasets.
Ensemble pre-trained models for low data transfer learning.
problem Training good models from scratch in low data regime.
method Fine-tune nearest-neighbour ranked pre-trained models to create ensembles.
result Achieves state-of-the-art performance with lower inference budget.
We investigate whether ResNet architectures can outperform more traditional Convolutional Neural Networks on the task of fine-grained vehicle classification. We train and test ResNet-18, ResNet-34 and ResNet-50 on the Comprehensive Cars dataset without pre-training on other datasets. We then modify the networks to use …
This work proves composite neural networks outperform their components under certain conditions.
problem Understanding the performance of composite neural networks.
method Theoretical investigation of a composite neural network combining pre-trained models.
result A composite neural network, with high probability, performs better than any of its pre-trained components.
E-Stitchup augments pre-trained embeddings for better model performance and confidence calibration.
problem Improving classification accuracy and confidence calibration of models using pre-trained embeddings.
method Data augmentation methods inspired by Mixup combined with label softening.
result Significantly increased classification accuracy and reduced training time.
Pre-trained LLM adapted with LoRA improves offline RL for quantitative trading.
problem Challenges in offline RL for quantitative trading due to complex temporal dependencies and overfitting.
method Integrates pre-trained GPT-2 weights and LoRA for efficient fine-tuning of a Decision Transformer.
result Outperforms existing offline RL methods in certain trading scenarios.
New findings on representation changes in transfer learning.
problem Understanding representation changes in transfer learning.
method Analysis of pre-trained and fine-tuned representations in transfer learning.
result Representation changes mainly occur in the head, but early layers are also important for cross-domain adaptation.
Deep Neural Networks (DNNs) have become increasingly popular in computer vision, natural language processing, and other areas. However, training and fine-tuning a deep learning model is computationally intensive and time-consuming. We propose a new method to improve the performance of nearly every model including pre-t…
AutoScale improves LLM pre-training by adjusting data mixtures at different scales.
problem Data mixtures that work well at small scales may not perform as well at larger scales.
method AutoScale uses a two-stage approach: fitting a model to predict loss under different compositions and extrapolating optimal compositions to larger scales.
result AutoScale accelerates convergence and improves downstream performance.
Transformer models improve query-document retrieval efficiency and accuracy.
problem Efficiently retrieve relevant documents from large corpora for query matching.
method Designed paragraph-level pre-training tasks to optimize embedding-based Transformer models.
result Transformer models significantly outperform BM-25 and non-Transformer embedding models.
Post-hoc uncertainty quantification improves on pre-trained neural networks without underfitting.
problem Uncertainty quantification in neural networks is underfitting or computationally demanding.
method Gaussian Process Activation function (GAPA) for neuron-level uncertainty, with two methods: GAPA-Free and GAPA-Variational.
result GAPA-Variational outperforms Laplace approximation on most datasets in uncertainty quantification metrics.
Cautious Weight Decay modifies weight decay for better optimization.
problem Improving optimization in deep learning models.
method Applies weight decay selectively based on parameter sign alignment.
result Consistently improves model performance across various tasks and scales.
Fine-tuning neural networks is widely used to transfer valuable knowledge from high-resource to low-resource domains. In a standard fine-tuning scheme, source and target problems are trained using the same architecture. Although capable of adapting to new domains, pre-trained units struggle with learning uncommon targe…
PAC-Net prunes deep models for better transfer learning.
problem Improving transfer learning with over-parameterized models.
method Prune, Allocate, Calibrate (PAC) approach.
result PAC-Net achieves state-of-the-art performance in inductive transfer learning.
In this paper, we present a novel approach for fine-tuning a decoder-side neural network in the context of image compression, such that the weight-updates are better compressible. At encoder side, we fine-tune a pre-trained artifact removal network on target data by using a compression objective applied on the weight-u…
We analyze the dynamics of training deep ReLU networks and their implications on generalization capability. Using a teacher-student setting, we discovered a novel relationship between the gradient received by hidden student nodes and the activations of teacher nodes for deep ReLU networks. With this relationship and th…
Weight Squeezing transfers knowledge from large models to smaller ones, improving performance and speed.
problem Transfer learning and model compression for faster and more efficient training.
method Reparameterization of weights from a large model to a smaller one.
result Weight Squeezing outperforms other methods on GLUE benchmark with faster training.
Revises Bayesian model averaging for foundation models.
problem Ensemble pre-trained and lightly-finetuned foundation models for improved classification performance.
method Introduces trainable linear classifiers and computationally cheaper model averaging scheme (OMA).
result Ensembled models can better predict on various datasets.
LEMON uses pre-trained models to scale neural networks efficiently.
problem Efficiency in scaling deep neural networks, especially Transformers, which are resource-intensive to train from scratch.
method LEMON initializes scaled models using pre-trained weights and optimizes learning rates.
result Significant reduction in training time and computational costs for Vision Transformers and BERT.
Regularization and data augmentation can be class-dependent, leading to poor performance on some classes.
problem Class-dependent effects of regularization and data augmentation.
method Evaluation of regularization and data augmentation techniques on Imagenet and INaturalist datasets.
result Regularization and data augmentation can lead to significant performance drops on some classes.
Proposes efficient deep causal generative models for high-dimensional causal inference.
problem Inefficient training of deep generative models for high-dimensional data.
method Modular training of deep causal generative models using adversarial training and pre-trained models.
result First algorithm that provably samples from any identifiable causal query in the presence of latent confounders.
Study shows pre-trained models can handle long-tailed relations well, improving classifier performance.
problem Challenges in long-tailed relation classification due to class imbalance.
method Used instance-balanced sampling to pre-train models and then improved classifier performance through attentive relation routing.
result Robust classifier with attentive relation routing achieves better long-tailed classification ability.
Deep convolutional neural networks (CNNs) can be applied to malware binary detection via image classification. The performance, however, is degraded due to the imbalance of malware families (classes). To mitigate this issue, we propose a simple yet effective weighted softmax loss which can be employed as the final laye…
Low bit-width integer weights and activations are very important for efficient inference, especially with respect to lower power consumption. We propose Monte Carlo methods to quantize the weights and activations of pre-trained neural networks without any re-training. By performing importance sampling we obtain quantiz…
Novel alignment framework for text-to-image generation using diffusion models and flow matching.
problem Improving text-to-image generation with minimal computational resources.
method Proposes a novel alignment framework that decomposes the score function into pre-trained score plus a conditional expectation of the reward.
result Achieves comparable performance to finetuning-based models with reduced computational cost.
Adapts example weights to optimize black-box metrics.
problem Optimizing metrics defined by black-box functions.
method Adaptive example weighting and iterative post-shifting.
result Improves classification performance compared to baselines.
Paper proposes SERT model for US stock pricing, outperforming standard models during market shocks.
problem Capturing patterns of temporal sparsity in asset pricing during market fluctuations.
method Introduces SERT model based on pre-trained Transformer, compares with standard models in three periods.
result SERT model achieves highest out-of-sample R2 (11.94\% and 11.47\%) during extreme market fluctuations. Deep neural network models have recently achieved state-of-the-art performance gains in a variety of natural language processing (NLP) tasks (Young, Hazarika, Poria, & Cambria, 2017). However, these gains rely on the availability of large amounts of annotated examples, without which state-of-the-art performance is rare…
In this study, a novel sparsity-driven weighted ensemble classifier (SDWEC) that improves classification accuracy and minimizes the number of classifiers is proposed. Using pre-trained classifiers, an ensemble in which base classifiers votes according to assigned weights is formed. These assigned weights directly affec…
The backpropagation algorithm for calculating gradients has been widely used in computation of weights for deep neural networks (DNNs). This method requires derivatives of objective functions and has some difficulties finding appropriate parameters such as learning rate. In this paper, we propose a novel approach for c…
GP-TS optimizes TLM pre-training hyperparameters efficiently.
problem Resource inefficiency in TLM pre-training.
method Bayesian optimization with Thompson sampling and Gaussian process.
result GP-TS achieves lower MLM loss in fewer epochs.
Self-training outperforms pre-training on COCO object detection and segmentation datasets.
problem The effectiveness of pre-training in improving object detection and segmentation models is limited.
method Investigated self-training as an alternative method to utilize additional data.
result Self-training consistently improves model performance across various dataset sizes and data augmentation levels.
He et al. (2018) have called into question the utility of pre-training by showing that training from scratch can often yield similar performance to pre-training. We show that although pre-training may not improve performance on traditional classification metrics, it improves model robustness and uncertainty estimates. …
RoBERTa outperforms other pre-trained models in NER tasks.
problem Improving Named Entity Recognition (NER) performance.
method Fine-tuning four pre-trained models (BERT, ERNIE, ERNIE2.0-tiny, RoBERTa) on NER task.
result RoBERTa achieved state-of-the-art results on MSRA-2006 dataset.
New framework explains how larger pre-trained models reduce downstream learning sample complexity.
problem Understanding why larger pre-trained models reduce sample complexity in downstream tasks.
method Introducing a novel framework called Caulking inspired by PEFT methods.
result Improved pre-trained models provably decrease downstream task sample complexity.
Generative AI decodes quantum codes without labeled data.
problem Efficient decoding of quantum error-correcting codes.
method Generative Transformers learn logical operators from unsupervised syndromes.
result Significantly better decoding accuracy than traditional methods.
Pre-training on different modalities improves Transformer performance in offline reinforcement learning.
problem Improving Transformer-based models for offline reinforcement learning tasks.
method Empirical investigation of pre-training on language and vision data, analysis of internal representations, and fine-tuning with no context.
result Pre-trained Transformers with language data can utilize context-like information to solve reinforcement learning tasks efficiently.
VQShape learns interpretable time-series representations and achieves comparable performance to specialist models.
problem Lack of interpretability in existing time-series models.
method Vector quantization of time-series data into abstracted shapes.
result VQShape achieves comparable performance to specialist models in classification tasks.
Deep neural networks have achieved impressive performance in many applications but their large number of parameters lead to significant computational and storage overheads. Several recent works attempt to mitigate these overheads by designing compact networks using pruning of connections. However, we observe that most …
The study finds a theoretical bound for pre-training iterations needed for pruning to yield good subnetwork performance.
problem Discovering efficient subnetworks within pre-trained dense networks.
method Mathematical analysis of a two-layer, fully-connected network, validating with a multi-layer perceptron trained on MNIST.
result A logarithmically dependent threshold on dataset size for successful pruning.
RIFLE improves deep transfer learning by reinitializing fully-connected layers.
problem Limited improvement in transfer learning accuracy with pre-trained models on small datasets.
method Re-Initializing fully-connected layers with random scratch during fine-tuning.
result Significant improvement in deep transfer learning accuracy across various datasets.