On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,695 papers · 148 categories

Trend · papers per month

count % of corpus

48 results for “pretraining data”

How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers?cs.LG

Task-agnostic data augmentation shows little benefit for pretrained transformers.

problem Evaluating the effectiveness of task-agnostic data augmentation on pretrained transformers.

method Conducted a systematic examination of two data augmentation techniques (Easy Data Augmentation and Back-Translation) across 5 tasks, 6 datasets, and 3 pretrained transformer models.

result Data augmentation techniques previously effective for non-pretrained models fail to consistently improve performance for pretrained transformers, even with limited training data.

data augmentation pretrained transformers task agnostic NLP computer vision

2020-10-05abs ↗pdf ↗

Improving Pretraining Data Using Perplexity Correlationscs.CL

New framework selects high-quality pretraining data without training LLMs.

problem Slow progress in understanding pretraining data due to costly experiments.

method Statistical framework based on perplexity-benchmark correlations.

result Approach outperforms existing methods on multiple benchmarks.

pretraining data language models correlation downstream performance statistical selection

2024-09-09abs ↗pdf ↗

Why Fine-grained Labels in Pretraining Benefit Generalization?cs.LG

Fine-grained pretraining improves neural network's ability to learn rare features.

problem Improving generalization in deep learning models.

method Introducing a hierarchical multi-view structure to confine input data distribution.

result Fine-grained pretraining leads to better accuracy on hard downstream test samples.

deep learning pretraining generalization neural networks feature learning

2024-10-30abs ↗pdf ↗

The Construction of Instruction-tuned LLMs for Finance without Instruction Data Using Continual Pretraining and Model Mergingcs.CL

Method constructs finance LLMs without instruction data using pretraining and model merging.

problem Developing domain-specific LLMs for finance is resource-intensive.

method Continual pretraining on financial data + model merging of instruction-tuned and domain-specific pretrained vectors.

result Successfully constructs instruction-tuned LLMs for finance without additional instruction data.

LLM finance instruction-tuning continual pretraining model merging

2024-09-30abs ↗pdf ↗

Theoretical Perspectives on Data Quality and Synergistic Effects in Pre- and Post-Training Reasoning Modelscs.LG

Theoretical analysis of data quality and synergies in LLMs.

problem Understanding why different training methods require different amounts of data.

method Theoretical analysis of transformers trained on a weight prediction task for linear regression.

result SFT excels on smaller datasets challenging for the pretrained model, while RL benefits from large, not overly difficult data.

Large Language Models Pretraining Fine-tuning Reinforcement Learning Data Quality

2026-03-01abs ↗pdf ↗

Synthetic continued pretrainingcs.LG

Synthetic continued pretraining enhances model performance with synthetic data.

problem Data inefficiency in pretrained models when adapting to domain-specific documents.

method Synthetic data augmentation using EntiGraph to create a large synthetic corpus.

result Language models can answer questions and follow instructions without access to domain-specific documents.

pretraining synthetic data continued pretraining language models domain adaptation

2024-09-11abs ↗pdf ↗

To Pretrain or Not to Pretrain: Examining the Benefits of Pretraining on Resource Rich Taskscs.CL

Pretraining models improves text classification accuracy, but diminishing returns are observed with large datasets.

problem Improving text classification accuracy with pretrained models.

method Examined the benefits of pretrained models on text classification tasks with varying amounts of training data.

result As the number of training examples grows into the millions, the accuracy gap between pretrained BERT-based models and vanilla LSTM narrows to within 1%.

pretraining text classification MLM BERT LSTM

2020-06-15abs ↗pdf ↗

Scaling Laws for Downstream Task Performance of Large Language Modelscs.CL

This study examines how the size and alignment of pretraining data affect the performance of large language models on downstream tasks.

problem Understanding how the size and alignment of pretraining data impact the performance of large language models on downstream tasks.

method Investigated the scaling behavior of large language models in a transfer learning setting, focusing on machine translation tasks.

result The size of the finetuning dataset and the distribution alignment between pretraining and downstream data significantly influence the scaling behavior of downstream performance.

large language models transfer learning machine translation scaling laws downstream performance

2024-02-06abs ↗pdf ↗

Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcs.LG

Publicly pretraining models on Web data may undermine differential privacy.

problem The use of large Web-scraped datasets in differential privacy models.

method Critical review of leveraging pretrained models on public datasets for differential privacy.

result Publicizing pretrained models as 'private' could harm trust and generalize poorly.

differential privacy public pretraining machine learning privacy web data

2022-12-13abs ↗pdf ↗

Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Modelscs.LG

Transformers can learn new tasks from diverse pretraining data but struggle with out-of-domain tasks.

problem Transformer models' ability to learn new tasks in-context is limited by their pretraining data coverage.

method Investigation of transformer models trained on

(x, f(x))

pairs, comparing in-context learning capabilities across different task families.

result Transformers can identify and learn within task families in their pretraining data but fail with out-of-domain tasks.

Transformer models In-context learning Pretraining data coverage Task families Out-of-domain tasks

2023-11-01abs ↗pdf ↗

The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shiftcs.LG

Study shows pretraining and finetuning can effectively tackle covariate shift in linear regression.

problem Linear regression under covariate shift where source and target distributions differ but conditional distribution remains similar.

method Pretraining on source data and finetuning on target data using online SGD.

result Transfer learning with

O(N^2)

source data is as effective as supervised learning with

N

target data.

covariate shift linear regression pretraining finetuning online SGD

2022-08-03abs ↗pdf ↗

Transfer Learning or Self-supervised Learning? A Tale of Two Pretraining Paradigmscs.CV

This study compares transfer learning and self-supervised learning for better model performance.

problem Choosing between transfer learning and self-supervised learning for optimal model performance.

method Comprehensive comparative study of transfer learning and self-supervised learning under various data and task properties.

result Self-supervised learning outperforms transfer learning in certain applications, and vice versa.

pretraining transfer learning self-supervised learning model performance data properties

2020-06-19abs ↗pdf ↗

Learning with Incomplete Context: Linear Contextual Bandits with Pretrained Imputationstat.ML

Algorithm improves decision-making with partially observed contexts using pretrained models.

problem Improving decision-making with partially observed contexts in online linear contextual bandits.

method PULSE-UCB algorithm that uses pretrained models trained on auxiliary data to impute missing features.

result Achieves near-optimal performance in i.i.d. context case with Hölder-smooth missing features.

contextual bandits pretrained models imputation synthetic features online decision-making

2025-10-10abs ↗pdf ↗

How Does the Pretraining Distribution Shape In-Context Learning? A Fundamental Trade-Offcs.LG

New research reveals how the pretraining distribution affects in-context learning in large language models.

problem Understanding how the pretraining distribution influences in-context learning in large language models.

method Developed a theoretical framework to characterize the relationship between pretraining distribution properties and in-context learning performance.

result Characterized a fundamental trade-off between robust task selection and generalization in ICL due to the pretraining distribution's statistical properties.

in-context learning large language models pretraining distribution statistical properties task selection

2025-10-01abs ↗pdf ↗

Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcs.LG

Transformers learn to make decisions in new contexts from offline data.

problem Understanding when and how transformers can perform in-context reinforcement learning.

method Theoretical framework analyzing supervised pretraining for ICRL, including algorithm distillation and decision-pretrained transformers.

result Transformers can efficiently approximate optimal reinforcement learning algorithms for various environments.

transformers in-context reinforcement learning supervised pretraining algorithm distillation decision-pretrained transformers

2023-10-12abs ↗pdf ↗

Fine-tuned In-Context Learning Transformers are Excellent Tabular Data Classifierscs.LG

Transformers fine-tuned on synthetic data boost tabular data classification performance.

problem Improving tabular data classification accuracy.

method Fine-tuning ICL-transformers on synthetic datasets with complex decision boundaries.

result Fine-tuned ICL-transformers outperform regular neural networks on real-world datasets.

tabular data classification in-context learning transformers synthetic data complex decision boundaries

2024-05-22abs ↗pdf ↗

Multi-Stage Influence Functioncs.LG

Paper introduces a multi-stage influence function to track model predictions.

problem Improving natural language processing and computer vision performance.

method Develops a multi-stage influence function to track predictions from finetuned models back to pretraining data.

result Identifies pretraining examples contributing most to finetuning task predictions.

influence function pretraining finetuning natural language processing computer vision

2020-07-17abs ↗pdf ↗

Masking Orchestration: Multi-task Pretraining for Multi-role Dialogue Representation Learningcs.CL

Pretraining method enhances dialogue representation learning across various tasks.

problem Scarce labeled data for specific dialogue tasks.

method Multi-task unsupervised pretraining with natural training objectives.

result Significant improvement in downstream tasks without encoder discrimination.

dialogue representation unsupervised pretraining multi-task learning downstream tasks dialogue mining

2020-02-27abs ↗pdf ↗

On the Provable Advantage of Unsupervised Pretrainingstat.ML

This paper proves the theoretical advantage of unsupervised pretraining for machine learning tasks.

problem Understanding why unsupervised pretraining helps in machine learning tasks.

method A generic framework using Maximum Likelihood Estimation (MLE) for unsupervised pretraining and Empirical Risk Minimization (ERM) for downstream tasks.

result Proves an excess risk of

ilde{\mathcal{O}}(\sqrt{\mathcal{C}_Φ/m} + \sqrt{\mathcal{C}_Ψ/n})

for downstream tasks under mild conditions.

unsupervised pretraining maximum likelihood estimation empirical risk minimization latent variable models downstream tasks

2023-03-02abs ↗pdf ↗

Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learningcs.LG

Data-efficient PDE operator learning without expensive simulations.

problem Expensive numerical PDE solutions limit data efficiency in machine learning.

method Unsupervised pretraining and in-context learning.

result Highly data-efficient and more generalizable than conventional models.

PDE operator learning unsupervised pretraining in-context learning data efficiency generalization

2024-02-24abs ↗pdf ↗

Towards Understanding Why Mask-Reconstruction Pretraining Helps in Downstream Taskscs.LG

Mask-reconstruction pretraining helps in downstream tasks by capturing more semantic features.

problem How mask-reconstruction pretraining helps in downstream tasks and why it surpasses supervised learning.

method Theoretical analysis and experimental validation of mask-reconstruction pretraining (MRP) on auto-encoders.

result MRP provably captures more semantic features than supervised learning, leading to better performance in downstream tasks.

mask-reconstruction pretraining unsupervised learning auto-encoders semantic feature learning downstream tasks

2022-06-08abs ↗pdf ↗

WeatherFormer: A Pretrained Encoder Model for Learning Robust Weather Representations from Small Datasetscs.CV

WeatherFormer learns robust weather features from small datasets.

problem Modeling complex weather dynamics from limited data.

method Pretrained transformer encoder on large satellite dataset, with spatiotemporal encoding.

result State-of-the-art performance in county-level soybean yield prediction and influenza forecasting.

weather modeling transformer model pretraining small datasets agriculture

2024-05-22abs ↗pdf ↗

Self-Supervised Pretraining for Heterogeneous Hypergraph Neural Networkscs.LG

Self-supervised pretraining for heterogeneous hypergraphs improves graph-based tasks.

problem Capturing higher-order relations in heterogeneous hypergraphs.

method SPHH framework for self-supervised pretraining of heterogeneous HyperGNNs.

result SPHH consistently outperforms state-of-the-art baselines in various downstream tasks.

Hypergraphs Self-supervised Pretraining Heterogeneous Graph Neural Networks

2023-11-19abs ↗pdf ↗

Pretraining Federated Text Models for Next Word Predictioncs.LG

Research improves federated text models for next word prediction.

problem Training models on distributed devices efficiently and effectively.

method Employ transfer learning in federated learning for next word prediction.

result Enhancements to current baselines with pretrained embeddings and whole model pretraining.

federated learning next word prediction transfer learning word embeddings model pretraining

2020-05-11abs ↗pdf ↗

Data Shapley in One Training Runcs.LG

In-Run Data Shapley offers efficient data attribution for large-scale models.

problem Existing data attribution methods are computationally intensive and cannot target specific models.

method In-Run Data Shapley, which efficiently attributes data contributions to a specific model without re-training.

result In-Run Data Shapley achieves significant efficiency, enabling data attribution for pretraining models.

Data Shapley Machine Learning Data Attribution Efficiency Generative AI

2024-06-16abs ↗pdf ↗

A Comparative Study of Pretrained Language Models on Thai Social Text Categorizationcs.LG

The ever-growing volume of data of user-generated content on social media provides a nearly unlimited corpus of unlabeled data even in languages where resources are scarce. In this paper, we demonstrate that state-of-the-art results on two Thai social text categorization tasks can be realized by pretraining a language …

2019-12-03abs ↗pdf ↗

Understanding Optimal Feature Transfer via a Fine-Grained Bias-Variance Analysisstat.ML

Optimal feature transfer identified through bias-variance analysis.

problem Optimizing feature transfer in transfer learning.

method Simple linear model with fine-grained bias-variance decomposition.

result Optimal pretrained feature transform is naturally sparse.

transfer learning feature transfer bias-variance sparse representation phase transition

2024-04-18abs ↗pdf ↗

Universal priors: solving empirical Bayes via Bayesian inference and pretrainingstat.ML

Transformer pretraining yields strong EB performance without explicit adaptation.

problem Empirical Bayes problems with unknown test distributions.

method Indirect analysis of pretrained transformer's performance under universal priors.

result Near-optimal regret bound of

\widetilde{O}(\frac{1}{n})

for arbitrary test distributions.

Transformer pretraining Empirical Bayes Universal priors Posterior contraction Bayesian inference

2026-02-16abs ↗pdf ↗

Selfie: Self-supervised Pretraining for Image Embeddingcs.LG

We introduce a pretraining technique called Selfie, which stands for SELFie supervised Image Embedding. Selfie generalizes the concept of masked language modeling of BERT (Devlin et al., 2019) to continuous data, such as images, by making use of the Contrastive Predictive Coding loss (Oord et al., 2018). Given masked-o…

2019-06-07abs ↗pdf ↗

Time Series Foundation Models for Multivariate Financial Time Series Forecastingq-fin.GN

TSFMs improve financial forecasting across diverse tasks with strong transferability.

problem Complex nonlinear relationships, temporal dependencies, and limited data in financial time series forecasting.

method Pretraining on diverse time series corpora followed by task-specific adaptation.

result Tiny Time Mixers (TTM) achieved 25-50% better performance on limited data and 15-30% improvements on longer datasets.

financial forecasting time series models transfer learning pretraining financial time series

2025-07-09abs ↗pdf ↗

Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcs.LG

Analysis of pretrained models' effectiveness in downstream tasks.

problem Understanding why pretrained models perform well in NLP tasks.

method Analyzed head and prompt tuning approaches using latent variable models.

result Prompt tuning provides stronger guarantees than head tuning.

pretrained models downstream tasks latent variable models HMM long-term memory

2021-06-17abs ↗pdf ↗

Fake News Detection with Different Modelscs.CL

Paper explores different models for fake news detection.

problem Detecting fake news with various models.

method Used machine learning algorithms and pretrained features like TFIDF, CV, and W2V.

result Developed models for better fake news detection.

fake news model machine learning pretrained textual data

2020-02-15abs ↗pdf ↗

Boosting Statistic Learning with Synthetic Data from Pretrained Large Modelsstat.ML

Framework uses synthetic data from pretrained models to improve predictive modeling.

problem Limited effectiveness of synthetic data from generative models for improving predictive performance.

method Proposes an end-to-end framework that generates and filters synthetic data through domain-specific statistical methods.

result Consistent improvements in predictive performance across various settings.

synthetic data statistic learning pretrained models predictive modeling generative models

2025-05-08abs ↗pdf ↗

Pretrained Language Models are Symbolic Mathematics Solvers too!stat.ML

Transformer models can solve complex math problems with less data.

problem Solving complex symbolic mathematics problems with limited data.

method Pretrain transformer models on language translation tasks and fine-tune for symbolic math.

result Pretrained transformer models achieve comparable accuracy to state-of-the-art models with less data.

Transformer models Symbolic mathematics Language translation Fine-tuning Data efficiency

2021-10-07abs ↗pdf ↗

CERT: Contrastive Self-supervised Learning for Language Understandingcs.CL

CERT improves language understanding by contrastively learning sentence-level semantics.

problem Lack of sentence-level semantics in existing pretraining tasks.

method Contrastive self-supervised learning at the sentence level using back-translation augmentations.

result CERT outperforms BERT on 7 out of 11 GLUE benchmark tasks, achieving the same performance as BERT on 2 tasks.

BERT GPT contrastive learning language understanding pretraining

2020-05-16abs ↗pdf ↗

Predicting trends in the quality of state-of-the-art neural networks without access to training or testing datacs.LG

New methods predict neural network quality without access to training data.

problem Predicting neural network quality without access to training or testing data.

method Meta-analysis of pretrained models using norm and power law based metrics.

result Power law based metrics can better distinguish well-trained from poorly-trained models.

neural networks pretrained models meta-analysis quality prediction power law

2020-02-17abs ↗pdf ↗

Enhancing Compositional Generalization via Compositional Feature Alignmentcs.CV

CFA improves model's ability to generalize across unseen domain-class combinations.

problem Challenges in real-world machine learning applications due to data distribution shifts and limited training data.

method Developed Compositional Feature Alignment (CFA) technique to improve CG ability of pretrained models.

result CFA outperforms common finetuning techniques in compositional generalization.

Compositional Generalization Feature Alignment Pretrained Models Data Distribution Shifts Multi-Class Multi-Domain

2024-02-05abs ↗pdf ↗

Mining Data Impressions from Deep Models as Substitute for the Unavailable Training Datacs.CV

Extract synthetic data from pretrained models for tasks without training data.

problem Lack of training data for tasks requiring model initialization.

method Extract 'Data Impressions' from pretrained deep models' parameters.

result Data Impressions enable various tasks like unsupervised domain adaptation and continual learning.

Data Impressions Pretrained Models Unsupervised Learning Continual Learning Knowledge Distillation

2021-01-15abs ↗pdf ↗

How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?stat.ML

Paper studies how few pretraining tasks are needed for a linear model to solve new tasks.

problem How many pretraining tasks are needed for a linear model to solve new tasks?

method Pretrained a linear attention model for linear regression with a Gaussian prior.

result Effective pretraining requires a small number of independent tasks, and the model closely matches Bayes optimal.

in-context learning linear regression attention model pretraining tasks Bayes optimal

2023-10-12abs ↗pdf ↗

These Are Not All the Features You Are Looking For: A Fundamental Bottleneck in Supervised Pretrainingcs.LG

New research shows deep models learn sparse features, limiting transfer learning; ensembling improves performance.

problem Sparse feature learning in deep models limits transfer learning performance.

method Developed a theoretical framework and proposed an ensembling strategy to aggregate multiple models.

result Ensembling yields a 9% improvement in transfer accuracy without extra pretraining cost.

transfer learning sparse representations model ensembles deep learning feature aggregation

2025-06-23abs ↗pdf ↗

On the Benefits of Public Representations for Private Transfer Learning under Distribution Shiftcs.LG

Public pretraining improves private model training even in extreme distribution shift scenarios.

problem Improving private model training accuracy in settings with large distribution shift.

method Empirical evaluation and theoretical explanation of public representations improving private training accuracy.

result Public representations can improve private training accuracy by up to 67% over private training from scratch in settings with large distribution shift.

private transfer learning distribution shift public pretraining differential privacy sample complexity

2023-12-24abs ↗pdf ↗

Your Finetuned Large Language Model is Already a Powerful Out-of-distribution Detectorcs.CL

A pretrained LLM and its finetuned version can detect OOD data effectively.

problem Detecting out-of-distribution data in language models.

method Using the likelihood ratio between a pretrained and finetuned LLM.

result The likelihood ratio is an effective criterion for OOD detection.

out-of-distribution detection large language models pretrained models finetuned models likelihood ratio

2024-04-07abs ↗pdf ↗

A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuningcs.LG

The paper develops a theory linking pretraining and fine-tuning in neural networks.

problem Understanding how initialization choices impact feature learning and generalization in neural networks.

method Analytical theory of diagonal linear networks, deriving generalization error as a function of initialization parameters and task statistics.

result Different initialization choices place networks into four fine-tuning regimes with varying abilities to support feature learning and generalization.

neural networks pretraining fine-tuning feature learning generalization

2026-02-23abs ↗pdf ↗

Optimal In-context Adaptivity and Distributional Robustness of Transformersstat.ML

Transformers learn to adapt to different task difficulties and resist distribution shifts.

problem Understanding and optimizing a Transformer's performance across various task difficulties and distribution shifts.

method Analyzing a pretrained Transformer on a mixture distribution of tasks, proving optimal convergence rates.

result Transformers achieve optimal convergence rates on tasks of specific difficulty levels, robust to distribution shifts.

Transformers in-context learning distributional robustness pretraining convergence rates

2025-10-27abs ↗pdf ↗

Pretrain-Test Task Alignment Governs Generalization in In-Context Learningstat.ML

New study reveals task alignment is key to ICL performance.

problem Understanding how task alignment affects generalization in in-context learning.

method Derived an exact expression for ICL generalization error in high dimensions under task covariance mismatch.

result Identified train-test task alignment as a key determinant of generalization in ICL.

in-context learning Transformer models task alignment generalization linear attention

2025-09-30abs ↗pdf ↗

Pretraining Deep Actor-Critic Reinforcement Learning Algorithms With Expert Demonstrationscs.AI

Pretraining with expert demonstrations have been found useful in speeding up the training process of deep reinforcement learning algorithms since less online simulation data is required. Some people use supervised learning to speed up the process of feature learning, others pretrain the policies by imitating expert dem…

2018-01-31abs ↗pdf ↗

Data Weighted Training Strategies for Grammatical Error Correctioncs.CL

Improved GEC models use scored data from large pretraining to outperform.

problem Addressing data sparsity in Grammatical Error Correction.

method Derive example-level scores from a smaller, higher-quality dataset and incorporate delta-log-perplexity into training schedules.

result Models trained on scored data achieve state-of-the-art results.

Grammatical Error Correction Data Sparsity Neural Machine Translation Delta-Log-Perplexity Training Schedules

2020-08-07abs ↗pdf ↗

Can Graph Neural Networks Go "Online"? An Analysis of Pretraining and Inferencecs.LG

Large-scale graph data in real-world applications is often not static but dynamic, i. e., new nodes and edges appear over time. Current graph convolution approaches are promising, especially, when all the graph's nodes and edges are available during training. When unseen nodes and edges are inserted after training, it …

2019-05-15abs ↗pdf ↗