Method transfers knowledge without label overlap, source data, or target architecture consistency.
problem Difficulties in transfer learning due to label mismatch, restricted source data, and specialized target architectures.
method Uses deep generative models in two stages: pseudo pre-training and pseudo semi-supervised learning.
result Outperforms scratch training and knowledge distillation methods.
A single pre-trained agent guides feature selection using knockoffs.
problem Feature selection challenges in AI-readiness of data.
method Generates knockoff features and uses reinforcement learning.
result Optimal feature subset identified with reduced dependency on target variable.
Paper proposes LMM-PQS for cross-domain few-shot learning.
problem Cross-domain few-shot learning problem.
method Generates pseudo query images and fine-tunes feature extraction modules with a large margin mechanism.
result LMM-PQS outperforms baseline models in cross-domain few-shot learning.
PNDMs accelerate DDPMs by treating them as differential equations on manifolds.
problem Accelerate DDPMs while maintaining sample quality.
method Propose pseudo numerical methods (PNDMs) to solve differential equations on manifolds.
result PNDMs generate higher quality images with only 50 steps compared to 1000-step DDIMs (20x speedup).
PC-GAIN improves GAIN's imputation by incorporating category information.
problem Missing data in incomplete datasets.
method Pre-training with pseudo-labels and incorporating an auxiliary classifier into GAIN.
result Significantly improved imputation quality compared to GAIN.
EnKG solves inverse problems without derivatives, using diffusion models.
problem Solving inverse problems with derivative-free methods.
method Ensemble Kalman Diffusion Guidance (EnKG) using diffusion models.
result EnKG can solve inverse problems with only forward model evaluations.
Deep neural networks are gaining increasing popularity for the classic text classification task, due to their strong expressive power and less requirement for feature engineering. Despite such attractiveness, neural text classification models suffer from the lack of training data in many real-world applications. Althou…
S2OSC improves OSC by filtering and re-training models with out-of-class instances.
problem Difficulty in classifying out-of-class instances during inference.
method S2OSC combines semi-supervised learning, out-of-class instance filtering, and model re-training.
result S2OSC achieves state-of-the-art performance in various OSC tasks.
GP-TS optimizes TLM pre-training hyperparameters efficiently.
problem Resource inefficiency in TLM pre-training.
method Bayesian optimization with Thompson sampling and Gaussian process.
result GP-TS achieves lower MLM loss in fewer epochs.
Self-training outperforms pre-training on COCO object detection and segmentation datasets.
problem The effectiveness of pre-training in improving object detection and segmentation models is limited.
method Investigated self-training as an alternative method to utilize additional data.
result Self-training consistently improves model performance across various dataset sizes and data augmentation levels.
He et al. (2018) have called into question the utility of pre-training by showing that training from scratch can often yield similar performance to pre-training. We show that although pre-training may not improve performance on traditional classification metrics, it improves model robustness and uncertainty estimates. …
New framework explains how larger pre-trained models reduce downstream learning sample complexity.
problem Understanding why larger pre-trained models reduce sample complexity in downstream tasks.
method Introducing a novel framework called Caulking inspired by PEFT methods.
result Improved pre-trained models provably decrease downstream task sample complexity.
Pre-training on different modalities improves Transformer performance in offline reinforcement learning.
problem Improving Transformer-based models for offline reinforcement learning tasks.
method Empirical investigation of pre-training on language and vision data, analysis of internal representations, and fine-tuning with no context.
result Pre-trained Transformers with language data can utilize context-like information to solve reinforcement learning tasks efficiently.
iTimER learns from reconstruction errors to represent irregularly sampled time series.
problem Learning from irregularly sampled time series with missing data.
method iTimER models reconstruction errors as a proxy for unobserved values, using a mixup strategy and a Wasserstein metric.
result iTimER outperforms state-of-the-art methods in classification, interpolation, and forecasting tasks.
The study finds a theoretical bound for pre-training iterations needed for pruning to yield good subnetwork performance.
problem Discovering efficient subnetworks within pre-trained dense networks.
method Mathematical analysis of a two-layer, fully-connected network, validating with a multi-layer perceptron trained on MNIST.
result A logarithmically dependent threshold on dataset size for successful pruning.
Unsupervised pre-training improves model generalization, but lacks theoretical understanding.
problem Lack of theoretical understanding of unsupervised pre-training's impact on model generalization.
method Introduces a novel theoretical framework to analyze and enhance generalization.
result Enhances understanding of unsupervised pre-training and fine-tuning, proposing a new regularization method.
Paper shows pre-training and transfer learning reduce sample complexity for neural networks.
problem Training high-dimensional supervised learning with limited labeled data.
method Study of single-layer neural networks via online stochastic gradient descent, considering concept shift.
result Pre-training and transfer learning reduce sample complexity by polynomial factors under general assumptions.
Fine-tuning with pre-training data improves performance.
problem Limited training data for tasks.
method Theoretical analysis of excess risk bound and selection of pre-training data subset.
result Improvement in generalization performance with pre-training data.
This paper connects masked pre-training to Bayesian model selection.
problem Understanding the success of masked pre-training and its generalization.
method The paper shows masked pre-training corresponds to maximizing the marginal likelihood.
result Masked pre-training with a suitable scoring function maximizes the marginal likelihood.
New method improves ensemble quality by exploring the pre-train basin more effectively.
problem Limited diversity in ensembles trained from a single pre-trained checkpoint.
method Proposed StarSSE modification of Snapshot Ensembles for transfer learning.
result Stronger ensembles and uniform model soups achieved.
New attack reveals memorization patterns in pre-trained LLMs.
problem Determining if a data point was part of a pre-trained LLM's training set.
method Adapts MIA statistical tests to LLM's perplexity dynamics of subsequences.
result Significantly outperforms prior approaches in membership inference attacks.
Named Entity Recognition (NER) is a fundamental Natural Language Processing (NLP) task to extract entities from unstructured data. The previous methods for NER were based on machine learning or deep learning. Recently, pre-training models have significantly improved performance on multiple NLP tasks. In this paper, fir…
This work studies the impact of intra-/inter-class diversity on pre-training datasets and finds a balance for optimal performance.
problem The impact of intra-/inter-class diversity on supervised pre-training datasets and their effect on downstream tasks.
method Empirical study and theoretical analysis of the relationship between diversity types and downstream performance.
result The optimal class-to-sample ratio is invariant to the size of the pre-training dataset and can be predicted.
Paper develops a theory explaining contrastive pre-training for multimodal AI.
problem Limited theoretical understanding of contrastive pre-training for multi-modal AI.
method Introduces approximate sufficient statistics and Joint Generative Hierarchical Model.
result Near-minimizers of contrastive loss are approximately sufficient, enabling diverse downstream tasks.
Ensemble pre-trained models for low data transfer learning.
problem Training good models from scratch in low data regime.
method Fine-tune nearest-neighbour ranked pre-trained models to create ensembles.
result Achieves state-of-the-art performance with lower inference budget.
The paper develops an asymptotic theory of self-supervised pre-training.
problem Sharpness of current rates in self-supervised pre-training and their accuracy.
method Two-stage M-estimation and tools from Riemannian geometry.
result Characterization of the limiting distribution of the downstream test risk.
In this paper, we present our method of using fixed-size ordinally forgetting encoding (FOFE) to solve the word sense disambiguation (WSD) problem. FOFE enables us to encode variable-length sequence of words into a theoretically unique fixed-size representation that can be fed into a feed forward neural network (FFNN),…
New analysis shows diverse classes in pre-training boost NLP performance.
problem Improving sample efficiency in downstream NLP tasks.
method Proved that diverse classes in pre-training lead to better performance, using a large last linear layer singular value.
result Transfer learning excess risk improves with large ildeν and $O\left(\frac{1}{ ildeν \sqrt{n}}
ight)$ rate. Pre-training is crucial for learning deep neural networks. Most of existing pre-training methods train simple models (e.g., restricted Boltzmann machines) and then stack them layer by layer to form the deep structure. This layer-wise pre-training has found strong theoretical foundation and broad empirical support. Howe…
Many applications of machine learning require a model to make accurate pre-dictions on test examples that are distributionally different from training ones, while task-specific labels are scarce during training. An effective approach to this challenge is to pre-train a model on related tasks where data is abundant, and…
Personalizes pre-trained models for nonparametric regression with limited data.
problem Improving data efficiency in nonparametric regression with few samples.
method Develops a theoretical framework and algorithms for few-shot personalization of black-box models.
result Achieves minimax optimal rate for personalization in nonparametric regression.
Behavior Transfer improves reinforcement learning by leveraging pre-trained policies.
problem Efficient transfer of knowledge in reinforcement learning.
method Behavior Transfer (BT) that uses pre-trained policies for exploration.
result BT combined with pre-training leads to better solutions than without pre-training.
Bridging the exponentially growing gap between the numbers of unlabeled and labeled protein sequences, several studies adopted semi-supervised learning for protein sequence modeling. In these studies, models were pre-trained with a substantial amount of unlabeled data, and the representations were transferred to variou…
Adaptive kernel density estimation improves accuracy in high dimensions.
problem Challenges in high-dimensional density estimation with traditional methods.
method Pre-training a neural network to recommend location-adaptive kernels.
result Effective density estimation in high dimensions with improved accuracy.
This paper ranks pre-trained DNNs using a novel SI measure.
problem Optimizing pre-trained DNN selection for transfer learning.
method Automated ranking via Separation Index (SI) on target datasets.
result Ranked pre-trained DNNs improve classification performance.
Study builds a Japanese financial-specific LLM through continual pre-training.
problem Lack of domain-specific Japanese financial LLMs.
method Continual pre-training on Japanese financial-focused datasets using a base Japanese LLM.
result Tuned model outperforms original model on Japanese financial benchmarks.
Improved tabular models learn better from real-world data.
problem Tabular models perform poorly on real-world datasets when trained only on synthetic data.
method Continued pre-training on a curated set of real-world datasets.
result Real-TabPFN achieves superior predictive accuracy on 29 datasets.
GCC pre-trains graph neural networks to transfer across diverse datasets.
problem Non-transferable graph models trained for specific datasets.
method Graph Contrastive Coding (GCC) framework using contrastive learning.
result GCC pre-trained models achieve competitive performance across multiple datasets.
Downscaled models outperform larger ones on GLUE tasks.
problem Difficulty in attributing performance changes to specific factors in large language models.
method Pre-trained down-scaled versions of Transformer-based architectures on a common corpus, benchmarked on GLUE tasks.
result MLM + NSP (BERT-style) consistently outperforms other objectives.
Improved LLM pre-training performance through better weight and variance control.
problem Improper weight and variance control in LLM pre-training affects downstream task performance.
method Introduced Layer Index Rescaling (LIR) and Target Variance Rescaling (TVR) techniques.
result Substantial improvements in downstream task performance (up to 4.6%) and reduced extreme activation values.
Researchers develop method to protect against 'weight poisoning' attacks on pre-trained models.
problem The security threat of downloading untrusted pre-trained weights that can be manipulated after fine-tuning.
method RIPPLe regularization method and Embedding Surgery initialization procedure.
result Demonstrated that weight poisoning attacks are possible even with limited knowledge of the dataset and fine-tuning procedure.
New measure assesses time series pre-training data quality without labels.
problem Challenges in collecting diverse pre-training datasets for time series classification.
method Contrastive-learning-based foundation model and contrastive accuracy measure.
result Contrastive accuracy correlates with model performance on downstream tasks.
Graph neural networks (GNNs) are shown to be successful in modeling applications with graph structures. However, training an accurate GNN model requires a large collection of labeled data and expressive features, which might be inaccessible for some applications. To tackle this problem, we propose a pre-training framew…
MeLa learns task relations by inferring global labels for robust FSL.
problem Few-shot learning with limited global labels.
method Meta Label Learning (MeLa) and augmented pre-training.
result MeLa outperforms existing methods across diverse benchmarks.
GPT-GNN pre-trains GNNs on unlabeled graphs to improve downstream performance.
problem Training GNNs requires labeled data, which is expensive.
method Generative pre-training of GNNs on unlabeled data with self-supervision.
result GPT-GNN significantly outperforms state-of-the-art GNNs without pre-training.
Survey on self-supervised pre-training for neural networks using unlabeled data.
problem Improving model performance using unlabeled data.
method Pre-training on unlabeled data followed by task-specific adaptation.
result Enhanced model performance through self-supervised pre-training.
Pre-training improves model coverage, crucial for downstream performance.
problem Understanding why pre-training enhances model performance.
method Coverage principle, focusing on next-token prediction and model quality.
result Coverage generalizes faster than cross-entropy, improving downstream performance.
New findings show BERT subnetworks can train independently and transfer to various tasks.
problem Finding smaller subnetworks that can train independently and transfer to other tasks.
method Examined pre-trained BERT models for subnetworks that can train independently and transfer to various downstream tasks.
result Found subnetworks at 40% to 90% sparsity that can train independently and transfer to various tasks.