On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,657 papers · 148 categories

Trend · papers per month

count % of corpus

48 results for “pretraining”

How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers?cs.LG

Task-agnostic data augmentation shows little benefit for pretrained transformers.

problem Evaluating the effectiveness of task-agnostic data augmentation on pretrained transformers.

method Conducted a systematic examination of two data augmentation techniques (Easy Data Augmentation and Back-Translation) across 5 tasks, 6 datasets, and 3 pretrained transformer models.

result Data augmentation techniques previously effective for non-pretrained models fail to consistently improve performance for pretrained transformers, even with limited training data.

data augmentation pretrained transformers task agnostic NLP computer vision

2020-10-05abs ↗pdf ↗

To Pretrain or Not to Pretrain: Examining the Benefits of Pretraining on Resource Rich Taskscs.CL

Pretraining models improves text classification accuracy, but diminishing returns are observed with large datasets.

problem Improving text classification accuracy with pretrained models.

method Examined the benefits of pretrained models on text classification tasks with varying amounts of training data.

result As the number of training examples grows into the millions, the accuracy gap between pretrained BERT-based models and vanilla LSTM narrows to within 1%.

pretraining text classification MLM BERT LSTM

2020-06-15abs ↗pdf ↗

Improving Pretraining Data Using Perplexity Correlationscs.CL

New framework selects high-quality pretraining data without training LLMs.

problem Slow progress in understanding pretraining data due to costly experiments.

method Statistical framework based on perplexity-benchmark correlations.

result Approach outperforms existing methods on multiple benchmarks.

pretraining data language models correlation downstream performance statistical selection

2024-09-09abs ↗pdf ↗

How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?stat.ML

Paper studies how few pretraining tasks are needed for a linear model to solve new tasks.

problem How many pretraining tasks are needed for a linear model to solve new tasks?

method Pretrained a linear attention model for linear regression with a Gaussian prior.

result Effective pretraining requires a small number of independent tasks, and the model closely matches Bayes optimal.

in-context learning linear regression attention model pretraining tasks Bayes optimal

2023-10-12abs ↗pdf ↗

Why Fine-grained Labels in Pretraining Benefit Generalization?cs.LG

Fine-grained pretraining improves neural network's ability to learn rare features.

problem Improving generalization in deep learning models.

method Introducing a hierarchical multi-view structure to confine input data distribution.

result Fine-grained pretraining leads to better accuracy on hard downstream test samples.

deep learning pretraining generalization neural networks feature learning

2024-10-30abs ↗pdf ↗

The Construction of Instruction-tuned LLMs for Finance without Instruction Data Using Continual Pretraining and Model Mergingcs.CL

Method constructs finance LLMs without instruction data using pretraining and model merging.

problem Developing domain-specific LLMs for finance is resource-intensive.

method Continual pretraining on financial data + model merging of instruction-tuned and domain-specific pretrained vectors.

result Successfully constructs instruction-tuned LLMs for finance without additional instruction data.

LLM finance instruction-tuning continual pretraining model merging

2024-09-30abs ↗pdf ↗

Towards Understanding Why Mask-Reconstruction Pretraining Helps in Downstream Taskscs.LG

Mask-reconstruction pretraining helps in downstream tasks by capturing more semantic features.

problem How mask-reconstruction pretraining helps in downstream tasks and why it surpasses supervised learning.

method Theoretical analysis and experimental validation of mask-reconstruction pretraining (MRP) on auto-encoders.

result MRP provably captures more semantic features than supervised learning, leading to better performance in downstream tasks.

mask-reconstruction pretraining unsupervised learning auto-encoders semantic feature learning downstream tasks

2022-06-08abs ↗pdf ↗

Multi-Stage Influence Functioncs.LG

Paper introduces a multi-stage influence function to track model predictions.

problem Improving natural language processing and computer vision performance.

method Develops a multi-stage influence function to track predictions from finetuned models back to pretraining data.

result Identifies pretraining examples contributing most to finetuning task predictions.

influence function pretraining finetuning natural language processing computer vision

2020-07-17abs ↗pdf ↗

Synthetic continued pretrainingcs.LG

Synthetic continued pretraining enhances model performance with synthetic data.

problem Data inefficiency in pretrained models when adapting to domain-specific documents.

method Synthetic data augmentation using EntiGraph to create a large synthetic corpus.

result Language models can answer questions and follow instructions without access to domain-specific documents.

pretraining synthetic data continued pretraining language models domain adaptation

2024-09-11abs ↗pdf ↗

WeatherFormer: A Pretrained Encoder Model for Learning Robust Weather Representations from Small Datasetscs.CV

WeatherFormer learns robust weather features from small datasets.

problem Modeling complex weather dynamics from limited data.

method Pretrained transformer encoder on large satellite dataset, with spatiotemporal encoding.

result State-of-the-art performance in county-level soybean yield prediction and influenza forecasting.

weather modeling transformer model pretraining small datasets agriculture

2024-05-22abs ↗pdf ↗

Theoretical Perspectives on Data Quality and Synergistic Effects in Pre- and Post-Training Reasoning Modelscs.LG

Theoretical analysis of data quality and synergies in LLMs.

problem Understanding why different training methods require different amounts of data.

method Theoretical analysis of transformers trained on a weight prediction task for linear regression.

result SFT excels on smaller datasets challenging for the pretrained model, while RL benefits from large, not overly difficult data.

Large Language Models Pretraining Fine-tuning Reinforcement Learning Data Quality

2026-03-01abs ↗pdf ↗

On Transfer of Adversarial Robustness from Pretraining to Downstream Taskscs.LG

Study shows how pretraining robustness transfers to downstream tasks.

problem Understanding how robustness is transferred from pretraining to downstream tasks.

method Theoretical analysis and practical validation of robustness constraints.

result Robustness of a linear predictor on downstream tasks can be constrained by the robustness of its underlying representation.

Adversarial robustness Pretraining Downstream tasks Representation function Robust classification

2022-08-07abs ↗pdf ↗

Scaling Laws for Downstream Task Performance of Large Language Modelscs.CL

This study examines how the size and alignment of pretraining data affect the performance of large language models on downstream tasks.

problem Understanding how the size and alignment of pretraining data impact the performance of large language models on downstream tasks.

method Investigated the scaling behavior of large language models in a transfer learning setting, focusing on machine translation tasks.

result The size of the finetuning dataset and the distribution alignment between pretraining and downstream data significantly influence the scaling behavior of downstream performance.

large language models transfer learning machine translation scaling laws downstream performance

2024-02-06abs ↗pdf ↗

How Does the Pretraining Distribution Shape In-Context Learning? A Fundamental Trade-Offcs.LG

New research reveals how the pretraining distribution affects in-context learning in large language models.

problem Understanding how the pretraining distribution influences in-context learning in large language models.

method Developed a theoretical framework to characterize the relationship between pretraining distribution properties and in-context learning performance.

result Characterized a fundamental trade-off between robust task selection and generalization in ICL due to the pretraining distribution's statistical properties.

in-context learning large language models pretraining distribution statistical properties task selection

2025-10-01abs ↗pdf ↗

Pretraining Federated Text Models for Next Word Predictioncs.LG

Research improves federated text models for next word prediction.

problem Training models on distributed devices efficiently and effectively.

method Employ transfer learning in federated learning for next word prediction.

result Enhancements to current baselines with pretrained embeddings and whole model pretraining.

federated learning next word prediction transfer learning word embeddings model pretraining

2020-05-11abs ↗pdf ↗

Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcs.LG

Transformers learn to make decisions in new contexts from offline data.

problem Understanding when and how transformers can perform in-context reinforcement learning.

method Theoretical framework analyzing supervised pretraining for ICRL, including algorithm distillation and decision-pretrained transformers.

result Transformers can efficiently approximate optimal reinforcement learning algorithms for various environments.

transformers in-context reinforcement learning supervised pretraining algorithm distillation decision-pretrained transformers

2023-10-12abs ↗pdf ↗

Knowledge-Aware Language Model Pretrainingcs.CL

This paper enhances language models with knowledge awareness.

problem Understanding how much knowledge pretrained language models grasp.

method Inserting explicit knowledge layers into pretraining without changing transformer architecture.

result Significantly more knowledge packed into transformer parameters.

language models knowledge awareness pretraining transformers entity signals

2020-06-29abs ↗pdf ↗

Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcs.LG

Publicly pretraining models on Web data may undermine differential privacy.

problem The use of large Web-scraped datasets in differential privacy models.

method Critical review of leveraging pretrained models on public datasets for differential privacy.

result Publicizing pretrained models as 'private' could harm trust and generalize poorly.

differential privacy public pretraining machine learning privacy web data

2022-12-13abs ↗pdf ↗

What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cs.CL

New study finds best language model architecture and pretraining objective for zero-shot tasks.

problem Evaluating which language model architectures and pretraining objectives best enable zero-shot generalization.

method Compared three model architectures and two pretraining objectives across 170 billion tokens, with and without finetuning.

result Causal decoder-only models trained on autoregressive language modeling exhibit strongest zero-shot generalization.

language models zero-shot generalization Transformer model architectures pretraining objectives

2022-04-12abs ↗pdf ↗

Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Modelscs.LG

Transformers can learn new tasks from diverse pretraining data but struggle with out-of-domain tasks.

problem Transformer models' ability to learn new tasks in-context is limited by their pretraining data coverage.

method Investigation of transformer models trained on

(x, f(x))

pairs, comparing in-context learning capabilities across different task families.

result Transformers can identify and learn within task families in their pretraining data but fail with out-of-domain tasks.

Transformer models In-context learning Pretraining data coverage Task families Out-of-domain tasks

2023-11-01abs ↗pdf ↗

Deep Anomaly Detection by Residual Adaptationcs.LG

Enhances anomaly detection in high dimensions with pretrained networks.

problem Difficult to characterize anomaly in high-dimensional data.

method Residual adaptation to adjust pretrained networks for anomaly detection.

result Significantly outperforms existing methods on anomaly detection benchmarks.

anomaly detection pretrained networks residual adaptation high dimensions

2020-10-05abs ↗pdf ↗

On the Provable Advantage of Unsupervised Pretrainingstat.ML

This paper proves the theoretical advantage of unsupervised pretraining for machine learning tasks.

problem Understanding why unsupervised pretraining helps in machine learning tasks.

method A generic framework using Maximum Likelihood Estimation (MLE) for unsupervised pretraining and Empirical Risk Minimization (ERM) for downstream tasks.

result Proves an excess risk of

ilde{\mathcal{O}}(\sqrt{\mathcal{C}_Φ/m} + \sqrt{\mathcal{C}_Ψ/n})

for downstream tasks under mild conditions.

unsupervised pretraining maximum likelihood estimation empirical risk minimization latent variable models downstream tasks

2023-03-02abs ↗pdf ↗

When can in-context learning generalize out of task distribution?cs.LG

Transformers learn to generalize out-of-distribution with diverse pretraining tasks.

problem Conditions for pretrained transformers to generalize out-of-distribution.

method Empirical study of task diversity and pretraining distribution.

result As task diversity increases, transformers transition from specialized to generalized solutions.

in-context learning pretrained transformers task diversity generalization phase diagram

2025-06-05abs ↗pdf ↗

Multiple Physics Pretraining for Physical Surrogate Modelscs.LG

MPP trains a transformer to predict multiple physical systems, improving accuracy across various tasks.

problem Training models for specific physical systems is inefficient and requires fine-tuning.

method MPP trains a shared transformer on multiple heterogeneous physical systems, projecting fields into a shared embedding space.

result A single MPP-pretrained transformer outperforms task-specific models on all pretraining sub-tasks and downstream tasks.

Transformer Physical Systems Surrogate Modeling Shared Embedding Autoregressive

2023-10-04abs ↗pdf ↗

Universal priors: solving empirical Bayes via Bayesian inference and pretrainingstat.ML

Transformer pretraining yields strong EB performance without explicit adaptation.

problem Empirical Bayes problems with unknown test distributions.

method Indirect analysis of pretrained transformer's performance under universal priors.

result Near-optimal regret bound of

\widetilde{O}(\frac{1}{n})

for arbitrary test distributions.

Transformer pretraining Empirical Bayes Universal priors Posterior contraction Bayesian inference

2026-02-16abs ↗pdf ↗

Test-Time Scaling Makes Overtraining Compute-Optimalcs.LG

New scaling laws optimize model size, training, and inference for better performance.

problem Trade-off between model size and inference cost in modern LLMs.

method Train-to-Test (

T^2

) scaling laws that jointly optimize model size, training tokens, and inference samples.

result Optimal pretraining decisions shift into overtraining regime, leading to stronger performance.

LLM scaling test-time scaling overtraining joint optimization downstream tasks

2026-04-01abs ↗pdf ↗

Masking Orchestration: Multi-task Pretraining for Multi-role Dialogue Representation Learningcs.CL

Multi-role dialogue understanding comprises a wide range of diverse tasks such as question answering, act classification, dialogue summarization etc. While dialogue corpora are abundantly available, labeled data, for specific learning tasks, can be highly scarce and expensive. In this work, we investigate dialogue cont…

2020-02-27abs ↗pdf ↗

Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcs.LG

Analysis of pretrained models' effectiveness in downstream tasks.

problem Understanding why pretrained models perform well in NLP tasks.

method Analyzed head and prompt tuning approaches using latent variable models.

result Prompt tuning provides stronger guarantees than head tuning.

pretrained models downstream tasks latent variable models HMM long-term memory

2021-06-17abs ↗pdf ↗

The Role of Pretrained Representations for the OOD Generalization of Reinforcement Learning Agentscs.LG

Robust reinforcement learning agents generalize well to out-of-distribution settings using pretrained representations.

problem Achieving sample-efficient reinforcement learning agents that generalize to real-world settings.

method Trained 240 representations and 10,000 RL policies on a simulated robotic setup, evaluating different pretrained VAE-based representations' effects on OOD generalization.

result Many reinforcement learning agents are surprisingly robust to realistic distribution shifts, including sim-to-real cases.

Reinforcement Learning Pretrained Representations OOD Generalization Robotic Setup Robustness

2021-07-12abs ↗pdf ↗

Experience-Efficient Model-Free Deep Reinforcement Learning Using Pre-Trainingcs.LG

PPOPT uses pretraining to speed up reinforcement learning in physics simulations.

problem High computational costs and inefficiency in reinforcement learning with small training samples.

method A novel policy neural network architecture that combines pretraining and fully-connected networks.

result PPOPT outperforms classic PPO on small training samples in terms of rewards and stability.

reinforcement learning deep learning pretraining model-free physics simulation

2025-10-11abs ↗pdf ↗

Pretrain Soft Q-Learning with Imperfect Demonstrationscs.LG

Pretraining reinforcement learning methods with demonstrations has been an important concept in the study of reinforcement learning since a large amount of computing power is spent on online simulations with existing reinforcement learning algorithms. Pretraining reinforcement learning remains a significant challenge i…

2019-05-09abs ↗pdf ↗

CERT: Contrastive Self-supervised Learning for Language Understandingcs.CL

CERT improves language understanding by contrastively learning sentence-level semantics.

problem Lack of sentence-level semantics in existing pretraining tasks.

method Contrastive self-supervised learning at the sentence level using back-translation augmentations.

result CERT outperforms BERT on 7 out of 11 GLUE benchmark tasks, achieving the same performance as BERT on 2 tasks.

BERT GPT contrastive learning language understanding pretraining

2020-05-16abs ↗pdf ↗

The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shiftcs.LG

Study shows pretraining and finetuning can effectively tackle covariate shift in linear regression.

problem Linear regression under covariate shift where source and target distributions differ but conditional distribution remains similar.

method Pretraining on source data and finetuning on target data using online SGD.

result Transfer learning with

O(N^2)

source data is as effective as supervised learning with

N

target data.

covariate shift linear regression pretraining finetuning online SGD

2022-08-03abs ↗pdf ↗

Self-Supervised Pretraining for Heterogeneous Hypergraph Neural Networkscs.LG

Self-supervised pretraining for heterogeneous hypergraphs improves graph-based tasks.

problem Capturing higher-order relations in heterogeneous hypergraphs.

method SPHH framework for self-supervised pretraining of heterogeneous HyperGNNs.

result SPHH consistently outperforms state-of-the-art baselines in various downstream tasks.

Hypergraphs Self-supervised Pretraining Heterogeneous Graph Neural Networks

2023-11-19abs ↗pdf ↗

Denoised Smoothing: A Provable Defense for Pretrained Classifierscs.LG

We present a method for provably defending any pretrained image classifier against $\ell_p$ adversarial attacks. This method, for instance, allows public vision API providers and users to seamlessly convert pretrained non-robust classification services into provably robust ones. By prepending a custom-trained denoiser …

2020-03-04abs ↗pdf ↗

Learning with Incomplete Context: Linear Contextual Bandits with Pretrained Imputationstat.ML

Algorithm improves decision-making with partially observed contexts using pretrained models.

problem Improving decision-making with partially observed contexts in online linear contextual bandits.

method PULSE-UCB algorithm that uses pretrained models trained on auxiliary data to impute missing features.

result Achieves near-optimal performance in i.i.d. context case with Hölder-smooth missing features.

contextual bandits pretrained models imputation synthetic features online decision-making

2025-10-10abs ↗pdf ↗

A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuningcs.LG

The paper develops a theory linking pretraining and fine-tuning in neural networks.

problem Understanding how initialization choices impact feature learning and generalization in neural networks.

method Analytical theory of diagonal linear networks, deriving generalization error as a function of initialization parameters and task statistics.

result Different initialization choices place networks into four fine-tuning regimes with varying abilities to support feature learning and generalization.

neural networks pretraining fine-tuning feature learning generalization

2026-02-23abs ↗pdf ↗

Optimal In-context Adaptivity and Distributional Robustness of Transformersstat.ML

Transformers learn to adapt to different task difficulties and resist distribution shifts.

problem Understanding and optimizing a Transformer's performance across various task difficulties and distribution shifts.

method Analyzing a pretrained Transformer on a mixture distribution of tasks, proving optimal convergence rates.

result Transformers achieve optimal convergence rates on tasks of specific difficulty levels, robust to distribution shifts.

Transformers in-context learning distributional robustness pretraining convergence rates

2025-10-27abs ↗pdf ↗

Understanding Optimal Feature Transfer via a Fine-Grained Bias-Variance Analysisstat.ML

Optimal feature transfer identified through bias-variance analysis.

problem Optimizing feature transfer in transfer learning.

method Simple linear model with fine-grained bias-variance decomposition.

result Optimal pretrained feature transform is naturally sparse.

transfer learning feature transfer bias-variance sparse representation phase transition

2024-04-18abs ↗pdf ↗

Transfer Learning or Self-supervised Learning? A Tale of Two Pretraining Paradigmscs.CV

This study compares transfer learning and self-supervised learning for better model performance.

problem Choosing between transfer learning and self-supervised learning for optimal model performance.

method Comprehensive comparative study of transfer learning and self-supervised learning under various data and task properties.

result Self-supervised learning outperforms transfer learning in certain applications, and vice versa.

pretraining transfer learning self-supervised learning model performance data properties

2020-06-19abs ↗pdf ↗

Explore and Exploit the Diverse Knowledge in Model Zoo for Domain Generalizationcs.LG

This paper improves model generalization by integrating diverse pretrained models.

problem Leveraging diverse pretrained models for robust out-of-distribution generalization.

method Characterize and integrate diverse pretrained models based on diversity and correlation shifts.

result Demonstrates state-of-the-art out-of-distribution generalization performance.

pretrained models domain generalization diversity correlation out-of-distribution

2023-06-05abs ↗pdf ↗

Fine-tuned In-Context Learning Transformers are Excellent Tabular Data Classifierscs.LG

Transformers fine-tuned on synthetic data boost tabular data classification performance.

problem Improving tabular data classification accuracy.

method Fine-tuning ICL-transformers on synthetic datasets with complex decision boundaries.

result Fine-tuned ICL-transformers outperform regular neural networks on real-world datasets.

tabular data classification in-context learning transformers synthetic data complex decision boundaries

2024-05-22abs ↗pdf ↗

Pretrain-Test Task Alignment Governs Generalization in In-Context Learningstat.ML

New study reveals task alignment is key to ICL performance.

problem Understanding how task alignment affects generalization in in-context learning.

method Derived an exact expression for ICL generalization error in high dimensions under task covariance mismatch.

result Identified train-test task alignment as a key determinant of generalization in ICL.

in-context learning Transformer models task alignment generalization linear attention

2025-09-30abs ↗pdf ↗

Pretraining Deep Actor-Critic Reinforcement Learning Algorithms With Expert Demonstrationscs.AI

Pretraining with expert demonstrations have been found useful in speeding up the training process of deep reinforcement learning algorithms since less online simulation data is required. Some people use supervised learning to speed up the process of feature learning, others pretrain the policies by imitating expert dem…

2018-01-31abs ↗pdf ↗

URL: A Representation Learning Benchmark for Transferable Uncertainty Estimatescs.LG

URL benchmark evaluates uncertainty quantification in pretrained models.

problem Need for reliable uncertainty estimates in transferable pretrained models.

method Proposes URL benchmark to measure transferability of representations and uncertainty estimates.

result Transferable uncertainty quantification remains challenging but not contradictory to traditional goals.

representation learning uncertainty quantification pretrained models transferability downstream datasets

2023-07-07abs ↗pdf ↗

Pretrained Time-Series Foundation Models for Financial Return Forecastingq-fin.MF

Pretrained time-series models outperform train-from-scratch baselines in financial return forecasting.

problem Financial return forecasting

method Pretrained time-series foundation models

result Pretrained TSFMs dominate the ranking distribution, accounting for 8 of 10 task-level wins.

financial return forecasting pretrained models time-series foundation models TSFMs financial markets

2026-06-25abs ↗pdf ↗

Data-Efficient Operator Learning via Unsupervised Pretraining and In-Context Learningcs.LG

Data-efficient PDE operator learning without expensive simulations.

problem Expensive numerical PDE solutions limit data efficiency in machine learning.

method Unsupervised pretraining and in-context learning.

result Highly data-efficient and more generalizable than conventional models.

PDE operator learning unsupervised pretraining in-context learning data efficiency generalization

2024-02-24abs ↗pdf ↗

Selfie: Self-supervised Pretraining for Image Embeddingcs.LG

We introduce a pretraining technique called Selfie, which stands for SELFie supervised Image Embedding. Selfie generalizes the concept of masked language modeling of BERT (Devlin et al., 2019) to continuous data, such as images, by making use of the Contrastive Predictive Coding loss (Oord et al., 2018). Given masked-o…

2019-06-07abs ↗pdf ↗