RoBERTa outperforms other pre-trained models in NER tasks.
problem Improving Named Entity Recognition (NER) performance.
method Fine-tuning four pre-trained models (BERT, ERNIE, ERNIE2.0-tiny, RoBERTa) on NER task.
result RoBERTa achieved state-of-the-art results on MSRA-2006 dataset.
He et al. (2018) have called into question the utility of pre-training by showing that training from scratch can often yield similar performance to pre-training. We show that although pre-training may not improve performance on traditional classification metrics, it improves model robustness and uncertainty estimates. …
New analysis shows diverse classes in pre-training boost NLP performance.
problem Improving sample efficiency in downstream NLP tasks.
method Proved that diverse classes in pre-training lead to better performance, using a large last linear layer singular value.
result Transfer learning excess risk improves with large ildeν and $O\left(\frac{1}{ ildeν \sqrt{n}}
ight)$ rate. New framework explains how larger pre-trained models reduce downstream learning sample complexity.
problem Understanding why larger pre-trained models reduce sample complexity in downstream tasks.
method Introducing a novel framework called Caulking inspired by PEFT methods.
result Improved pre-trained models provably decrease downstream task sample complexity.
New findings show BERT subnetworks can train independently and transfer to various tasks.
problem Finding smaller subnetworks that can train independently and transfer to other tasks.
method Examined pre-trained BERT models for subnetworks that can train independently and transfer to various downstream tasks.
result Found subnetworks at 40% to 90% sparsity that can train independently and transfer to various tasks.
Many applications of machine learning require a model to make accurate pre-dictions on test examples that are distributionally different from training ones, while task-specific labels are scarce during training. An effective approach to this challenge is to pre-train a model on related tasks where data is abundant, and…
Paper develops a theory explaining contrastive pre-training for multimodal AI.
problem Limited theoretical understanding of contrastive pre-training for multi-modal AI.
method Introduces approximate sufficient statistics and Joint Generative Hierarchical Model.
result Near-minimizers of contrastive loss are approximately sufficient, enabling diverse downstream tasks.
Fine-tuning with pre-training data improves performance.
problem Limited training data for tasks.
method Theoretical analysis of excess risk bound and selection of pre-training data subset.
result Improvement in generalization performance with pre-training data.
GCC pre-trains graph neural networks to transfer across diverse datasets.
problem Non-transferable graph models trained for specific datasets.
method Graph Contrastive Coding (GCC) framework using contrastive learning.
result GCC pre-trained models achieve competitive performance across multiple datasets.
MeLa learns task relations by inferring global labels for robust FSL.
problem Few-shot learning with limited global labels.
method Meta Label Learning (MeLa) and augmented pre-training.
result MeLa outperforms existing methods across diverse benchmarks.
Graph neural networks (GNNs) are shown to be successful in modeling applications with graph structures. However, training an accurate GNN model requires a large collection of labeled data and expressive features, which might be inaccessible for some applications. To tackle this problem, we propose a pre-training framew…
Auxiliary Tuning adapts pre-trained models for novel tasks efficiently.
problem Adapting pre-trained models for new tasks efficiently.
method Supplementing pre-trained model with an auxiliary model that shifts output distribution.
result Achieved similar results to training from scratch with fewer resources.
GPT-GNN pre-trains GNNs on unlabeled graphs to improve downstream performance.
problem Training GNNs requires labeled data, which is expensive.
method Generative pre-training of GNNs on unlabeled data with self-supervision.
result GPT-GNN significantly outperforms state-of-the-art GNNs without pre-training.
Bridging the exponentially growing gap between the numbers of unlabeled and labeled protein sequences, several studies adopted semi-supervised learning for protein sequence modeling. In these studies, models were pre-trained with a substantial amount of unlabeled data, and the representations were transferred to variou…
Unsupervised pre-training improves model generalization, but lacks theoretical understanding.
problem Lack of theoretical understanding of unsupervised pre-training's impact on model generalization.
method Introduces a novel theoretical framework to analyze and enhance generalization.
result Enhances understanding of unsupervised pre-training and fine-tuning, proposing a new regularization method.
Survey on self-supervised pre-training for neural networks using unlabeled data.
problem Improving model performance using unlabeled data.
method Pre-training on unlabeled data followed by task-specific adaptation.
result Enhanced model performance through self-supervised pre-training.
Ensemble pre-trained models for low data transfer learning.
problem Training good models from scratch in low data regime.
method Fine-tune nearest-neighbour ranked pre-trained models to create ensembles.
result Achieves state-of-the-art performance with lower inference budget.
New measure assesses time series pre-training data quality without labels.
problem Challenges in collecting diverse pre-training datasets for time series classification.
method Contrastive-learning-based foundation model and contrastive accuracy measure.
result Contrastive accuracy correlates with model performance on downstream tasks.
Optimus pre-trains sentences in a latent space for various NLP tasks.
problem Training large-scale language models for diverse NLP tasks.
method Pre-trained Variational Autoencoder (VAE) on large text corpus, fine-tuned for various tasks.
result Optimus achieves state-of-the-art on VAE language modeling benchmarks.
Theoretical study shows adversarial training improves robustness in deep learning models.
problem Ensuring robustness in pre-trained deep learning models.
method Theoretical analysis of adversarial training and feature purification in two-layer neural networks.
result Adversarial training leads to feature purification, making models more robust to attacks.
Pre-training on different modalities improves Transformer performance in offline reinforcement learning.
problem Improving Transformer-based models for offline reinforcement learning tasks.
method Empirical investigation of pre-training on language and vision data, analysis of internal representations, and fine-tuning with no context.
result Pre-trained Transformers with language data can utilize context-like information to solve reinforcement learning tasks efficiently.
Transformer models improve query-document retrieval efficiency and accuracy.
problem Efficiently retrieve relevant documents from large corpora for query matching.
method Designed paragraph-level pre-training tasks to optimize embedding-based Transformer models.
result Transformer models significantly outperform BM-25 and non-Transformer embedding models.
Downscaled models outperform larger ones on GLUE tasks.
problem Difficulty in attributing performance changes to specific factors in large language models.
method Pre-trained down-scaled versions of Transformer-based architectures on a common corpus, benchmarked on GLUE tasks.
result MLM + NSP (BERT-style) consistently outperforms other objectives.
Study finds multi-task learning and pre-training can improve healthcare models on EHR data.
problem Improving model performance on diverse EHR tasks using multi-task learning.
method Examined multi-task learning across various EHR tasks and training schemes, using pre-training and fine-tuning.
result Significant gains in model performance achieved via multi-task pre-training and single-task fine-tuning.
Foundation models trained on collider data improve jet generation tasks.
problem Improving foundation models for jet generation tasks.
method Pre-training OmniJet-α model on AspenOpenJets dataset. result Pre-trained model improves performance on jet generation tasks with domain shift.
This work studies the impact of intra-/inter-class diversity on pre-training datasets and finds a balance for optimal performance.
problem The impact of intra-/inter-class diversity on supervised pre-training datasets and their effect on downstream tasks.
method Empirical study and theoretical analysis of the relationship between diversity types and downstream performance.
result The optimal class-to-sample ratio is invariant to the size of the pre-training dataset and can be predicted.
Improved LLM pre-training performance through better weight and variance control.
problem Improper weight and variance control in LLM pre-training affects downstream task performance.
method Introduced Layer Index Rescaling (LIR) and Target Variance Rescaling (TVR) techniques.
result Substantial improvements in downstream task performance (up to 4.6%) and reduced extreme activation values.
Our method upweights easy samples to mitigate forgetting in fine-tuning.
problem Catastrophic forgetting in fine-tuning pre-trained models.
method Sample weighting based on pre-trained model's losses.
result Our method reduces forgetting by up to 0.8% on MetaMathQA while preserving more accuracy on pre-training datasets.
This paper presents our contribution to PolEval 2019 Task 6: Hate speech and bullying detection. We describe three parallel approaches that we followed: fine-tuning a pre-trained ULMFiT model to our classification task, fine-tuning a pre-trained BERT model to our classification task, and using the TPOT library to find …
Personalizes pre-trained models for nonparametric regression with limited data.
problem Improving data efficiency in nonparametric regression with few samples.
method Develops a theoretical framework and algorithms for few-shot personalization of black-box models.
result Achieves minimax optimal rate for personalization in nonparametric regression.
Behavior Transfer improves reinforcement learning by leveraging pre-trained policies.
problem Efficient transfer of knowledge in reinforcement learning.
method Behavior Transfer (BT) that uses pre-trained policies for exploration.
result BT combined with pre-training leads to better solutions than without pre-training.
Deep neural networks have shown promising results for various clinical prediction tasks. However, training deep networks such as those based on Recurrent Neural Networks (RNNs) requires large labeled data, significant hyper-parameter tuning effort and expertise, and high computational resources. In this work, we invest…
MARGE learns to reconstruct text by paraphrasing, achieving strong performance across multiple tasks.
problem Training sequence-to-sequence models with limited supervision.
method Unsupervised multi-lingual multi-document paraphrasing objective.
result Strong zero-shot performance on document translation and various tasks in multiple languages.
SuNCEt accelerates contrastive learning with minimal labeled data.
problem Efficiently learning visual representations with limited labeled data.
method Noise-contrastive estimation and neighbourhood component analysis-based semi-supervised loss.
result SuNCEt achieves semi-supervised learning accuracy with less than half the labeled data.
Pre-trained language model boosts RL efficiency.
problem Low sample efficiency in RL, especially in lifelong learning.
method Use a pre-trained task-independent language model for transfer learning.
result Goal-conditional RL agents become more sample efficient.
Paper provides conditions for reliable use of pre-trained embeddings in econometrics.
problem Uncertainty in using pre-trained embeddings for econometric tasks.
method Derives sufficient conditions and convergence rates for machine learning models with pre-trained embeddings.
result Establishes theoretical foundations for reliable use of pre-trained embeddings in econometrics.
This study explores how examples influence ICL in LLMs.
problem Understanding how examples impact in-context learning in large language models.
method Theoretical study with a probabilistic model extending from Gaussian mixture model.
result The impact of pre-training knowledge and examples on ICL prediction accuracy.
Paper develops large time series models using pre-trained transformers.
problem Performance bottlenecks in small models on data-scarce scenarios.
method Large-scale pre-training, unified time series format, GPT-style architecture.
result Generative pre-trained Time Series Transformer (Timer) for diverse tasks.
Recently, pre-trained language representation flourishes as the mainstay of the natural language understanding community, e.g., BERT. These pre-trained language representations can create state-of-the-art results on a wide range of downstream tasks. Along with continuous significant performance improvement, the size an…
Deep neural networks have shown promising results for various clinical prediction tasks such as diagnosis, mortality prediction, predicting duration of stay in hospital, etc. However, training deep networks -- such as those based on Recurrent Neural Networks (RNNs) -- requires large labeled data, high computational res…
Deep convolutional neural networks have achieved great success in various applications. However, training an effective DNN model for a specific task is rather challenging because it requires a prior knowledge or experience to design the network architecture, repeated trial-and-error process to tune the parameters, and …
Fine-tuning large language models requires minimal data, making them efficient.
problem Achieving state-of-the-art performance with large language models.
method Using BERT as an example, fine-tuning only the most critical layers of the pre-trained model.
result Fine-tuned models are close in parameter space to the pre-trained model, with many good solutions found in sparsified versions.
Delphyne improves financial time series models with pre-trained language models.
problem Lack of financial data and negative transfer effect in existing time-series pre-trained models.
method Delphyne is a pre-trained model for financial time series that addresses the lack of financial data and negative transfer effect.
result Delphyne achieves competitive performance and superior performances on various financial tasks.
Paper proposes Meta Label Learning to infer global labels for robust few-shot models.
problem Few-shot learning with limited training data.
method Meta Label Learning (MeLa) framework that infers global labels.
result MeLa framework is competitive with existing methods and robust for few-shot learning.
TCGPN improves stock forecasting by capturing temporal correlation patterns.
problem Stock forecasting with minimal periodicity and large node numbers.
method TCGPN uses Temporal-Correlation fusion encoder and pre-training methods to handle large datasets.
result TCGPN achieves state-of-the-art results on real stock market data.
HyperBO+ pre-trains a universal prior for Bayesian optimization across different domains.
problem Bayesian optimization requires domain-specific priors, limiting its applicability.
method Two-step pre-training method for hierarchical Gaussian processes.
result HyperBO+ achieves lower regrets on unseen search spaces.
Kronos improves financial time series analysis with a pre-trained model.
problem Limited application of large-scale models to financial candlestick data.
method Unified, scalable pre-training framework for financial K-line modeling.
result Kronos excels in financial tasks like price forecasting and volatility prediction.
New theory explains when pre-trained models can improve downstream tasks.
problem Lack of theoretical understanding of task similarity for transfer learning.
method Feature-centric viewpoint, theoretical results on transferability phase diagram.
result Transfer learning outperforms training from scratch when target task is well represented in feature space.