On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,695 papers · 148 categories

Trend · papers per month

count % of corpus

48 results for “subset pretraining”

Efficient Neural Network Training via Subset Pretrainingcs.LG

Subset pretraining speeds up neural network training.

problem Efficiently training neural networks with partial gradients.

method Use small subsets to approximate the loss minimum of the full training set.

result Subset minima can approximate the full training set's loss minimum efficiently.

neural networks subset pretraining stochastic approximation MNIST CIFAR-10

2024-10-21abs ↗pdf ↗

A Theory of How Pretraining Shapes Inductive Bias in Fine-Tuningcs.LG

The paper develops a theory linking pretraining and fine-tuning in neural networks.

problem Understanding how initialization choices impact feature learning and generalization in neural networks.

method Analytical theory of diagonal linear networks, deriving generalization error as a function of initialization parameters and task statistics.

result Different initialization choices place networks into four fine-tuning regimes with varying abilities to support feature learning and generalization.

neural networks pretraining fine-tuning feature learning generalization

2026-02-23abs ↗pdf ↗

One Model for One Graph: A New Perspective for Pretraining with Cross-domain Graphscs.LG

A new framework pretrains a single GNN model for diverse graphs, overcoming domain-specific challenges.

problem Difficulty in generalizing across graphs from different domains using existing GNNs.

method Cross-domain pretraining framework with gating functions to choose experts for new graphs.

result Superior performance on link prediction and node classification tasks across various domains.

Graph Neural Networks Cross-domain Pretraining Gating Functions Node Classification Link Prediction

2024-11-30abs ↗pdf ↗

Boosting Statistic Learning with Synthetic Data from Pretrained Large Modelsstat.ML

Framework uses synthetic data from pretrained models to improve predictive modeling.

problem Limited effectiveness of synthetic data from generative models for improving predictive performance.

method Proposes an end-to-end framework that generates and filters synthetic data through domain-specific statistical methods.

result Consistent improvements in predictive performance across various settings.

synthetic data statistic learning pretrained models predictive modeling generative models

2025-05-08abs ↗pdf ↗

Activation-Space Uncertainty Quantification for Pretrained Networksstat.ML

GAPA method provides efficient uncertainty quantification for pretrained networks.

problem Reliable uncertainty estimates for pretrained models are challenging.

method Post-hoc Gaussian Process Activations (GAPA) method that shifts Bayesian modeling from weights to activations.

result GAPA method provides efficient uncertainty quantification without altering the backbone's predictions.

uncertainty quantification pretrained networks Gaussian Processes activations Bayesian modeling

2026-02-16abs ↗pdf ↗

Data Shapley in One Training Runcs.LG

In-Run Data Shapley offers efficient data attribution for large-scale models.

problem Existing data attribution methods are computationally intensive and cannot target specific models.

method In-Run Data Shapley, which efficiently attributes data contributions to a specific model without re-training.

result In-Run Data Shapley achieves significant efficiency, enabling data attribution for pretraining models.

Data Shapley Machine Learning Data Attribution Efficiency Generative AI

2024-06-16abs ↗pdf ↗

Training Subset Selection for Weak Supervisionstat.ML

Subset selection improves weak supervision performance.

problem Optimizing the use of weakly-labeled data.

method Combining pretrained data representations with the cut statistic for subset selection.

result Subset selection improves weak supervision performance by up to 19%.

subset selection weak supervision weak labels data representation cut statistic

2022-06-06abs ↗pdf ↗

How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers?cs.LG

Task-agnostic data augmentation shows little benefit for pretrained transformers.

problem Evaluating the effectiveness of task-agnostic data augmentation on pretrained transformers.

method Conducted a systematic examination of two data augmentation techniques (Easy Data Augmentation and Back-Translation) across 5 tasks, 6 datasets, and 3 pretrained transformer models.

result Data augmentation techniques previously effective for non-pretrained models fail to consistently improve performance for pretrained transformers, even with limited training data.

data augmentation pretrained transformers task agnostic NLP computer vision

2020-10-05abs ↗pdf ↗

To Pretrain or Not to Pretrain: Examining the Benefits of Pretraining on Resource Rich Taskscs.CL

Pretraining models improves text classification accuracy, but diminishing returns are observed with large datasets.

problem Improving text classification accuracy with pretrained models.

method Examined the benefits of pretrained models on text classification tasks with varying amounts of training data.

result As the number of training examples grows into the millions, the accuracy gap between pretrained BERT-based models and vanilla LSTM narrows to within 1%.

pretraining text classification MLM BERT LSTM

2020-06-15abs ↗pdf ↗

Improving Pretraining Data Using Perplexity Correlationscs.CL

New framework selects high-quality pretraining data without training LLMs.

problem Slow progress in understanding pretraining data due to costly experiments.

method Statistical framework based on perplexity-benchmark correlations.

result Approach outperforms existing methods on multiple benchmarks.

pretraining data language models correlation downstream performance statistical selection

2024-09-09abs ↗pdf ↗

How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?stat.ML

Paper studies how few pretraining tasks are needed for a linear model to solve new tasks.

problem How many pretraining tasks are needed for a linear model to solve new tasks?

method Pretrained a linear attention model for linear regression with a Gaussian prior.

result Effective pretraining requires a small number of independent tasks, and the model closely matches Bayes optimal.

in-context learning linear regression attention model pretraining tasks Bayes optimal

2023-10-12abs ↗pdf ↗

Why Fine-grained Labels in Pretraining Benefit Generalization?cs.LG

Fine-grained pretraining improves neural network's ability to learn rare features.

problem Improving generalization in deep learning models.

method Introducing a hierarchical multi-view structure to confine input data distribution.

result Fine-grained pretraining leads to better accuracy on hard downstream test samples.

deep learning pretraining generalization neural networks feature learning

2024-10-30abs ↗pdf ↗

The Construction of Instruction-tuned LLMs for Finance without Instruction Data Using Continual Pretraining and Model Mergingcs.CL

Method constructs finance LLMs without instruction data using pretraining and model merging.

problem Developing domain-specific LLMs for finance is resource-intensive.

method Continual pretraining on financial data + model merging of instruction-tuned and domain-specific pretrained vectors.

result Successfully constructs instruction-tuned LLMs for finance without additional instruction data.

LLM finance instruction-tuning continual pretraining model merging

2024-09-30abs ↗pdf ↗

Towards Understanding Why Mask-Reconstruction Pretraining Helps in Downstream Taskscs.LG

Mask-reconstruction pretraining helps in downstream tasks by capturing more semantic features.

problem How mask-reconstruction pretraining helps in downstream tasks and why it surpasses supervised learning.

method Theoretical analysis and experimental validation of mask-reconstruction pretraining (MRP) on auto-encoders.

result MRP provably captures more semantic features than supervised learning, leading to better performance in downstream tasks.

mask-reconstruction pretraining unsupervised learning auto-encoders semantic feature learning downstream tasks

2022-06-08abs ↗pdf ↗

Multi-Stage Influence Functioncs.LG

Paper introduces a multi-stage influence function to track model predictions.

problem Improving natural language processing and computer vision performance.

method Develops a multi-stage influence function to track predictions from finetuned models back to pretraining data.

result Identifies pretraining examples contributing most to finetuning task predictions.

influence function pretraining finetuning natural language processing computer vision

2020-07-17abs ↗pdf ↗

Masking Orchestration: Multi-task Pretraining for Multi-role Dialogue Representation Learningcs.CL

Pretraining method enhances dialogue representation learning across various tasks.

problem Scarce labeled data for specific dialogue tasks.

method Multi-task unsupervised pretraining with natural training objectives.

result Significant improvement in downstream tasks without encoder discrimination.

dialogue representation unsupervised pretraining multi-task learning downstream tasks dialogue mining

2020-02-27abs ↗pdf ↗

Synthetic continued pretrainingcs.LG

Synthetic continued pretraining enhances model performance with synthetic data.

problem Data inefficiency in pretrained models when adapting to domain-specific documents.

method Synthetic data augmentation using EntiGraph to create a large synthetic corpus.

result Language models can answer questions and follow instructions without access to domain-specific documents.

pretraining synthetic data continued pretraining language models domain adaptation

2024-09-11abs ↗pdf ↗

WeatherFormer: A Pretrained Encoder Model for Learning Robust Weather Representations from Small Datasetscs.CV

WeatherFormer learns robust weather features from small datasets.

problem Modeling complex weather dynamics from limited data.

method Pretrained transformer encoder on large satellite dataset, with spatiotemporal encoding.

result State-of-the-art performance in county-level soybean yield prediction and influenza forecasting.

weather modeling transformer model pretraining small datasets agriculture

2024-05-22abs ↗pdf ↗

Theoretical Perspectives on Data Quality and Synergistic Effects in Pre- and Post-Training Reasoning Modelscs.LG

Theoretical analysis of data quality and synergies in LLMs.

problem Understanding why different training methods require different amounts of data.

method Theoretical analysis of transformers trained on a weight prediction task for linear regression.

result SFT excels on smaller datasets challenging for the pretrained model, while RL benefits from large, not overly difficult data.

Large Language Models Pretraining Fine-tuning Reinforcement Learning Data Quality

2026-03-01abs ↗pdf ↗

Denoised Smoothing: A Provable Defense for Pretrained Classifierscs.LG

Denoised smoothing defends pretrained classifiers against adversarial attacks.

problem Adversarial attacks on pretrained classifiers.

method Prepending a denoiser to any off-the-shelf classifier using randomized smoothing.

result Guaranteed

\ell_p

-robustness to adversarial examples without modifying the pretrained classifier.

adversarial robustness pretrained models image classification randomized smoothing denoising

2020-03-04abs ↗pdf ↗

On Transfer of Adversarial Robustness from Pretraining to Downstream Taskscs.LG

Study shows how pretraining robustness transfers to downstream tasks.

problem Understanding how robustness is transferred from pretraining to downstream tasks.

method Theoretical analysis and practical validation of robustness constraints.

result Robustness of a linear predictor on downstream tasks can be constrained by the robustness of its underlying representation.

Adversarial robustness Pretraining Downstream tasks Representation function Robust classification

2022-08-07abs ↗pdf ↗

Scaling Laws for Downstream Task Performance of Large Language Modelscs.CL

This study examines how the size and alignment of pretraining data affect the performance of large language models on downstream tasks.

problem Understanding how the size and alignment of pretraining data impact the performance of large language models on downstream tasks.

method Investigated the scaling behavior of large language models in a transfer learning setting, focusing on machine translation tasks.

result The size of the finetuning dataset and the distribution alignment between pretraining and downstream data significantly influence the scaling behavior of downstream performance.

large language models transfer learning machine translation scaling laws downstream performance

2024-02-06abs ↗pdf ↗

How Does the Pretraining Distribution Shape In-Context Learning? A Fundamental Trade-Offcs.LG

New research reveals how the pretraining distribution affects in-context learning in large language models.

problem Understanding how the pretraining distribution influences in-context learning in large language models.

method Developed a theoretical framework to characterize the relationship between pretraining distribution properties and in-context learning performance.

result Characterized a fundamental trade-off between robust task selection and generalization in ICL due to the pretraining distribution's statistical properties.

in-context learning large language models pretraining distribution statistical properties task selection

2025-10-01abs ↗pdf ↗

Pretraining Federated Text Models for Next Word Predictioncs.LG

Research improves federated text models for next word prediction.

problem Training models on distributed devices efficiently and effectively.

method Employ transfer learning in federated learning for next word prediction.

result Enhancements to current baselines with pretrained embeddings and whole model pretraining.

federated learning next word prediction transfer learning word embeddings model pretraining

2020-05-11abs ↗pdf ↗

Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcs.LG

Transformers learn to make decisions in new contexts from offline data.

problem Understanding when and how transformers can perform in-context reinforcement learning.

method Theoretical framework analyzing supervised pretraining for ICRL, including algorithm distillation and decision-pretrained transformers.

result Transformers can efficiently approximate optimal reinforcement learning algorithms for various environments.

transformers in-context reinforcement learning supervised pretraining algorithm distillation decision-pretrained transformers

2023-10-12abs ↗pdf ↗

Knowledge-Aware Language Model Pretrainingcs.CL

This paper enhances language models with knowledge awareness.

problem Understanding how much knowledge pretrained language models grasp.

method Inserting explicit knowledge layers into pretraining without changing transformer architecture.

result Significantly more knowledge packed into transformer parameters.

language models knowledge awareness pretraining transformers entity signals

2020-06-29abs ↗pdf ↗

Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcs.LG

Publicly pretraining models on Web data may undermine differential privacy.

problem The use of large Web-scraped datasets in differential privacy models.

method Critical review of leveraging pretrained models on public datasets for differential privacy.

result Publicizing pretrained models as 'private' could harm trust and generalize poorly.

differential privacy public pretraining machine learning privacy web data

2022-12-13abs ↗pdf ↗

What Language Model Architecture and Pretraining Objective Work Best for Zero-Shot Generalization?cs.CL

New study finds best language model architecture and pretraining objective for zero-shot tasks.

problem Evaluating which language model architectures and pretraining objectives best enable zero-shot generalization.

method Compared three model architectures and two pretraining objectives across 170 billion tokens, with and without finetuning.

result Causal decoder-only models trained on autoregressive language modeling exhibit strongest zero-shot generalization.

language models zero-shot generalization Transformer model architectures pretraining objectives

2022-04-12abs ↗pdf ↗

Inheritance Between Feedforward and Convolutional Networks via Model Projectionstat.ML

Model projection transfers convolutional network properties to feedforward networks.

problem Transferring properties between feedforward and convolutional networks.

method Unified node-level framework with tensor-valued activations, model projection.

result Projected CNN nodes inherit GFFN-style trainable structure.

model inheritance feedforward networks convolutional networks model projection parameter-efficient transfer learning

2026-02-05abs ↗pdf ↗

Pretraining Data Mixtures Enable Narrow Model Selection Capabilities in Transformer Modelscs.LG

Transformers can learn new tasks from diverse pretraining data but struggle with out-of-domain tasks.

problem Transformer models' ability to learn new tasks in-context is limited by their pretraining data coverage.

method Investigation of transformer models trained on

(x, f(x))

pairs, comparing in-context learning capabilities across different task families.

result Transformers can identify and learn within task families in their pretraining data but fail with out-of-domain tasks.

Transformer models In-context learning Pretraining data coverage Task families Out-of-domain tasks

2023-11-01abs ↗pdf ↗

Deep Anomaly Detection by Residual Adaptationcs.LG

Enhances anomaly detection in high dimensions with pretrained networks.

problem Difficult to characterize anomaly in high-dimensional data.

method Residual adaptation to adjust pretrained networks for anomaly detection.

result Significantly outperforms existing methods on anomaly detection benchmarks.

anomaly detection pretrained networks residual adaptation high dimensions

2020-10-05abs ↗pdf ↗

On the Provable Advantage of Unsupervised Pretrainingstat.ML

This paper proves the theoretical advantage of unsupervised pretraining for machine learning tasks.

problem Understanding why unsupervised pretraining helps in machine learning tasks.

method A generic framework using Maximum Likelihood Estimation (MLE) for unsupervised pretraining and Empirical Risk Minimization (ERM) for downstream tasks.

result Proves an excess risk of

ilde{\mathcal{O}}(\sqrt{\mathcal{C}_Φ/m} + \sqrt{\mathcal{C}_Ψ/n})

for downstream tasks under mild conditions.

unsupervised pretraining maximum likelihood estimation empirical risk minimization latent variable models downstream tasks

2023-03-02abs ↗pdf ↗

Towards More Accurate Automatic Sleep Staging via Deep Transfer Learningcs.LG

Background: Despite recent significant progress in the development of automatic sleep staging methods, building a good model still remains a big challenge for sleep studies with a small cohort due to the data-variability and data-inefficiency issues. This work presents a deep transfer learning approach to overcome thes…

2019-07-30abs ↗pdf ↗

When can in-context learning generalize out of task distribution?cs.LG

Transformers learn to generalize out-of-distribution with diverse pretraining tasks.

problem Conditions for pretrained transformers to generalize out-of-distribution.

method Empirical study of task diversity and pretraining distribution.

result As task diversity increases, transformers transition from specialized to generalized solutions.

in-context learning pretrained transformers task diversity generalization phase diagram

2025-06-05abs ↗pdf ↗

Multiple Physics Pretraining for Physical Surrogate Modelscs.LG

MPP trains a transformer to predict multiple physical systems, improving accuracy across various tasks.

problem Training models for specific physical systems is inefficient and requires fine-tuning.

method MPP trains a shared transformer on multiple heterogeneous physical systems, projecting fields into a shared embedding space.

result A single MPP-pretrained transformer outperforms task-specific models on all pretraining sub-tasks and downstream tasks.

Transformer Physical Systems Surrogate Modeling Shared Embedding Autoregressive

2023-10-04abs ↗pdf ↗

Universal priors: solving empirical Bayes via Bayesian inference and pretrainingstat.ML

Transformer pretraining yields strong EB performance without explicit adaptation.

problem Empirical Bayes problems with unknown test distributions.

method Indirect analysis of pretrained transformer's performance under universal priors.

result Near-optimal regret bound of

\widetilde{O}(\frac{1}{n})

for arbitrary test distributions.

Transformer pretraining Empirical Bayes Universal priors Posterior contraction Bayesian inference

2026-02-16abs ↗pdf ↗

Test-Time Scaling Makes Overtraining Compute-Optimalcs.LG

New scaling laws optimize model size, training, and inference for better performance.

problem Trade-off between model size and inference cost in modern LLMs.

method Train-to-Test (

T^2

) scaling laws that jointly optimize model size, training tokens, and inference samples.

result Optimal pretraining decisions shift into overtraining regime, leading to stronger performance.

LLM scaling test-time scaling overtraining joint optimization downstream tasks

2026-04-01abs ↗pdf ↗

Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcs.LG

Analysis of pretrained models' effectiveness in downstream tasks.

problem Understanding why pretrained models perform well in NLP tasks.

method Analyzed head and prompt tuning approaches using latent variable models.

result Prompt tuning provides stronger guarantees than head tuning.

pretrained models downstream tasks latent variable models HMM long-term memory

2021-06-17abs ↗pdf ↗

The Role of Pretrained Representations for the OOD Generalization of Reinforcement Learning Agentscs.LG

Robust reinforcement learning agents generalize well to out-of-distribution settings using pretrained representations.

problem Achieving sample-efficient reinforcement learning agents that generalize to real-world settings.

method Trained 240 representations and 10,000 RL policies on a simulated robotic setup, evaluating different pretrained VAE-based representations' effects on OOD generalization.

result Many reinforcement learning agents are surprisingly robust to realistic distribution shifts, including sim-to-real cases.

Reinforcement Learning Pretrained Representations OOD Generalization Robotic Setup Robustness

2021-07-12abs ↗pdf ↗

Rich Insights from Cheap Signals: Efficient Evaluations via Tensor Factorizationcs.AI

Efficiently evaluate generative models at the prompt level using tensor factorization.

problem Fine-grained evaluations of generative models are costly and often misaligned with human judgment.

method Tensor factorization model that merges cheap autorater data with a small set of human gold-standard labels.

result The method provides accurate and tight confidence intervals for model performance.

tensor factorization generative models prompt evaluation autorater human gold-standard

2026-03-02abs ↗pdf ↗

Experience-Efficient Model-Free Deep Reinforcement Learning Using Pre-Trainingcs.LG

PPOPT uses pretraining to speed up reinforcement learning in physics simulations.

problem High computational costs and inefficiency in reinforcement learning with small training samples.

method A novel policy neural network architecture that combines pretraining and fully-connected networks.

result PPOPT outperforms classic PPO on small training samples in terms of rewards and stability.

reinforcement learning deep learning pretraining model-free physics simulation

2025-10-11abs ↗pdf ↗

Pretrain Soft Q-Learning with Imperfect Demonstrationscs.LG

Pretraining reinforcement learning methods with demonstrations has been an important concept in the study of reinforcement learning since a large amount of computing power is spent on online simulations with existing reinforcement learning algorithms. Pretraining reinforcement learning remains a significant challenge i…

2019-05-09abs ↗pdf ↗

CERT: Contrastive Self-supervised Learning for Language Understandingcs.CL

CERT improves language understanding by contrastively learning sentence-level semantics.

problem Lack of sentence-level semantics in existing pretraining tasks.

method Contrastive self-supervised learning at the sentence level using back-translation augmentations.

result CERT outperforms BERT on 7 out of 11 GLUE benchmark tasks, achieving the same performance as BERT on 2 tasks.

BERT GPT contrastive learning language understanding pretraining

2020-05-16abs ↗pdf ↗

The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shiftcs.LG

Study shows pretraining and finetuning can effectively tackle covariate shift in linear regression.

problem Linear regression under covariate shift where source and target distributions differ but conditional distribution remains similar.

method Pretraining on source data and finetuning on target data using online SGD.

result Transfer learning with

O(N^2)

source data is as effective as supervised learning with

N

target data.

covariate shift linear regression pretraining finetuning online SGD

2022-08-03abs ↗pdf ↗

Self-Supervised Pretraining for Heterogeneous Hypergraph Neural Networkscs.LG

Self-supervised pretraining for heterogeneous hypergraphs improves graph-based tasks.

problem Capturing higher-order relations in heterogeneous hypergraphs.

method SPHH framework for self-supervised pretraining of heterogeneous HyperGNNs.

result SPHH consistently outperforms state-of-the-art baselines in various downstream tasks.

Hypergraphs Self-supervised Pretraining Heterogeneous Graph Neural Networks

2023-11-19abs ↗pdf ↗

Learning with Incomplete Context: Linear Contextual Bandits with Pretrained Imputationstat.ML

Algorithm improves decision-making with partially observed contexts using pretrained models.

problem Improving decision-making with partially observed contexts in online linear contextual bandits.

method PULSE-UCB algorithm that uses pretrained models trained on auxiliary data to impute missing features.

result Achieves near-optimal performance in i.i.d. context case with Hölder-smooth missing features.

contextual bandits pretrained models imputation synthetic features online decision-making

2025-10-10abs ↗pdf ↗

One-Class Feature Learning Using Intra-Class Splittingcs.LG

This paper proposes a novel generic one-class feature learning method based on intra-class splitting. In one-class classification, feature learning is challenging, because only samples of one class are available during training. Hence, state-of-the-art methods require reference multi-class datasets to pretrain feature …

2018-12-20abs ↗pdf ↗

Optimal In-context Adaptivity and Distributional Robustness of Transformersstat.ML

Transformers learn to adapt to different task difficulties and resist distribution shifts.

problem Understanding and optimizing a Transformer's performance across various task difficulties and distribution shifts.

method Analyzing a pretrained Transformer on a mixture distribution of tasks, proving optimal convergence rates.

result Transformers achieve optimal convergence rates on tasks of specific difficulty levels, robust to distribution shifts.

Transformers in-context learning distributional robustness pretraining convergence rates

2025-10-27abs ↗pdf ↗