On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,694 papers · 148 categories

Trend · papers per month

count % of corpus

48 results for “Pretrained Representations”

Masking Orchestration: Multi-task Pretraining for Multi-role Dialogue Representation Learningcs.CL

Pretraining method enhances dialogue representation learning across various tasks.

problem Scarce labeled data for specific dialogue tasks.

method Multi-task unsupervised pretraining with natural training objectives.

result Significant improvement in downstream tasks without encoder discrimination.

dialogue representation unsupervised pretraining multi-task learning downstream tasks dialogue mining

2020-02-27abs ↗pdf ↗

The Role of Pretrained Representations for the OOD Generalization of Reinforcement Learning Agentscs.LG

Robust reinforcement learning agents generalize well to out-of-distribution settings using pretrained representations.

problem Achieving sample-efficient reinforcement learning agents that generalize to real-world settings.

method Trained 240 representations and 10,000 RL policies on a simulated robotic setup, evaluating different pretrained VAE-based representations' effects on OOD generalization.

result Many reinforcement learning agents are surprisingly robust to realistic distribution shifts, including sim-to-real cases.

Reinforcement Learning Pretrained Representations OOD Generalization Robotic Setup Robustness

2021-07-12abs ↗pdf ↗

Understanding Optimal Feature Transfer via a Fine-Grained Bias-Variance Analysisstat.ML

Optimal feature transfer identified through bias-variance analysis.

problem Optimizing feature transfer in transfer learning.

method Simple linear model with fine-grained bias-variance decomposition.

result Optimal pretrained feature transform is naturally sparse.

transfer learning feature transfer bias-variance sparse representation phase transition

2024-04-18abs ↗pdf ↗

URL: A Representation Learning Benchmark for Transferable Uncertainty Estimatescs.LG

URL benchmark evaluates uncertainty quantification in pretrained models.

problem Need for reliable uncertainty estimates in transferable pretrained models.

method Proposes URL benchmark to measure transferability of representations and uncertainty estimates.

result Transferable uncertainty quantification remains challenging but not contradictory to traditional goals.

representation learning uncertainty quantification pretrained models transferability downstream datasets

2023-07-07abs ↗pdf ↗

On Transfer of Adversarial Robustness from Pretraining to Downstream Taskscs.LG

Study shows how pretraining robustness transfers to downstream tasks.

problem Understanding how robustness is transferred from pretraining to downstream tasks.

method Theoretical analysis and practical validation of robustness constraints.

result Robustness of a linear predictor on downstream tasks can be constrained by the robustness of its underlying representation.

Adversarial robustness Pretraining Downstream tasks Representation function Robust classification

2022-08-07abs ↗pdf ↗

WeatherFormer: A Pretrained Encoder Model for Learning Robust Weather Representations from Small Datasetscs.CV

WeatherFormer learns robust weather features from small datasets.

problem Modeling complex weather dynamics from limited data.

method Pretrained transformer encoder on large satellite dataset, with spatiotemporal encoding.

result State-of-the-art performance in county-level soybean yield prediction and influenza forecasting.

weather modeling transformer model pretraining small datasets agriculture

2024-05-22abs ↗pdf ↗

An empirical study of pretrained representations for few-shot classificationcs.LG

Recent algorithms with state-of-the-art few-shot classification results start their procedure by computing data features output by a large pretrained model. In this paper we systematically investigate which models provide the best representations for a few-shot image classification task when pretrained on the Imagenet …

2019-10-03abs ↗pdf ↗

Self-Supervised Pretraining for Heterogeneous Hypergraph Neural Networkscs.LG

Self-supervised pretraining for heterogeneous hypergraphs improves graph-based tasks.

problem Capturing higher-order relations in heterogeneous hypergraphs.

method SPHH framework for self-supervised pretraining of heterogeneous HyperGNNs.

result SPHH consistently outperforms state-of-the-art baselines in various downstream tasks.

Hypergraphs Self-supervised Pretraining Heterogeneous Graph Neural Networks

2023-11-19abs ↗pdf ↗

Contextures: The Mechanism of Representation Learningcs.LG

The dissertation establishes a contexture theory to mathematically characterize representation learning.

problem The lack of a scientific understanding of representation learning in foundation models.

method Introduces the contexture theory as a unified framework for analyzing representation learning methods.

result Representation learning is optimal when the association between input and context is neither too strong nor too weak.

representation learning contexture theory downstream tasks pretraining methods statistical learning bounds

2025-04-28abs ↗pdf ↗

CERT: Contrastive Self-supervised Learning for Language Understandingcs.CL

CERT improves language understanding by contrastively learning sentence-level semantics.

problem Lack of sentence-level semantics in existing pretraining tasks.

method Contrastive self-supervised learning at the sentence level using back-translation augmentations.

result CERT outperforms BERT on 7 out of 11 GLUE benchmark tasks, achieving the same performance as BERT on 2 tasks.

BERT GPT contrastive learning language understanding pretraining

2020-05-16abs ↗pdf ↗

These Are Not All the Features You Are Looking For: A Fundamental Bottleneck in Supervised Pretrainingcs.LG

New research shows deep models learn sparse features, limiting transfer learning; ensembling improves performance.

problem Sparse feature learning in deep models limits transfer learning performance.

method Developed a theoretical framework and proposed an ensembling strategy to aggregate multiple models.

result Ensembling yields a 9% improvement in transfer accuracy without extra pretraining cost.

transfer learning sparse representations model ensembles deep learning feature aggregation

2025-06-23abs ↗pdf ↗

Efficient Adaptive Data Acquisition via Pretrained Belief Representationscs.LG

POLAR learns efficient data acquisition policies using pretrained belief representations.

problem Challenges in learning effective policies for adaptive data acquisition.

method POLAR decouples representation learning from policy learning by leveraging pretrained predictive foundation models as belief-state encoders.

result POLAR outperforms state-of-the-art methods across diverse tasks while requiring fewer training samples.

data acquisition policy learning belief representations amortized methods Bayesian experimental design

2026-06-23abs ↗pdf ↗

On the Provable Advantage of Unsupervised Pretrainingstat.ML

This paper proves the theoretical advantage of unsupervised pretraining for machine learning tasks.

problem Understanding why unsupervised pretraining helps in machine learning tasks.

method A generic framework using Maximum Likelihood Estimation (MLE) for unsupervised pretraining and Empirical Risk Minimization (ERM) for downstream tasks.

result Proves an excess risk of

ilde{\mathcal{O}}(\sqrt{\mathcal{C}_Φ/m} + \sqrt{\mathcal{C}_Ψ/n})

for downstream tasks under mild conditions.

unsupervised pretraining maximum likelihood estimation empirical risk minimization latent variable models downstream tasks

2023-03-02abs ↗pdf ↗

Self-Supervised Dynamical System Representations for Physiological Time-Seriescs.LG

PULSE learns representations from physiological time series.

problem Lack of effective pretraining objectives for physiological time series.

method Proposes a pretraining framework exploiting dynamical systems generative model.

result PULSE learns representations that distinguish semantic classes and improve transfer learning.

self-supervised learning physiological time series dynamical systems pretraining representation learning

2025-11-28abs ↗pdf ↗

Deep Anomaly Detection by Residual Adaptationcs.LG

Enhances anomaly detection in high dimensions with pretrained networks.

problem Difficult to characterize anomaly in high-dimensional data.

method Residual adaptation to adjust pretrained networks for anomaly detection.

result Significantly outperforms existing methods on anomaly detection benchmarks.

anomaly detection pretrained networks residual adaptation high dimensions

2020-10-05abs ↗pdf ↗

SEMASIA: A Large-Scale Dataset of Semantically Structured Latent Representationscs.LG

SEMASIA provides a large dataset of latent representations for model comparison.

problem Difficulty in comparing semantic structures across different neural network models.

method Collection of latent representations from 1700 pretrained models across various benchmarks.

result Consistent semantic organization across models and datasets.

latent representations neural networks semantic structure benchmark model comparison

2026-05-10abs ↗pdf ↗

History Compression via Language Models in Reinforcement Learningcs.LG

A new method uses a frozen language model to improve sample efficiency in reinforcement learning.

problem Improving sample efficiency in reinforcement learning with partially observable environments.

method FROZEN Hopfield network and HELM (History Embedding Language Model) method.

result HELM achieves new state-of-the-art results on Minigrid and Procgen environments.

Reinforcement Learning Language Models Hopfield Networks Sample Efficiency History Compression

2022-05-24abs ↗pdf ↗

Revisiting Theory of Contrastive Learning for Domain Generalizationstat.ML

New bounds for contrastive learning handle domain shifts and generalization.

problem Domain shifts and generalization challenges in downstream tasks.

method Novel generalization bounds accounting for both domain shift and generalization.

result Performance of contrastively learned representations depends on statistical discrepancy between pretraining and downstream distributions.

contrastive learning domain shifts domain generalization latent class space statistical discrepancy

2025-12-02abs ↗pdf ↗

Knowledge-Aware Language Model Pretrainingcs.CL

This paper enhances language models with knowledge awareness.

problem Understanding how much knowledge pretrained language models grasp.

method Inserting explicit knowledge layers into pretraining without changing transformer architecture.

result Significantly more knowledge packed into transformer parameters.

language models knowledge awareness pretraining transformers entity signals

2020-06-29abs ↗pdf ↗

Synthetic continued pretrainingcs.LG

Synthetic continued pretraining enhances model performance with synthetic data.

problem Data inefficiency in pretrained models when adapting to domain-specific documents.

method Synthetic data augmentation using EntiGraph to create a large synthetic corpus.

result Language models can answer questions and follow instructions without access to domain-specific documents.

pretraining synthetic data continued pretraining language models domain adaptation

2024-09-11abs ↗pdf ↗

On the Benefits of Public Representations for Private Transfer Learning under Distribution Shiftcs.LG

Public pretraining improves private model training even in extreme distribution shift scenarios.

problem Improving private model training accuracy in settings with large distribution shift.

method Empirical evaluation and theoretical explanation of public representations improving private training accuracy.

result Public representations can improve private training accuracy by up to 67% over private training from scratch in settings with large distribution shift.

private transfer learning distribution shift public pretraining differential privacy sample complexity

2023-12-24abs ↗pdf ↗

Extracting Sentence Embeddings from Pretrained Transformer Modelscs.CL

This study improves sentence embeddings from BERT models.

problem Capturing the underlying meaning of sentences using BERT models.

method Comprehensive review and testing of various sentence embedding extraction and refinement methods.

result Representation-shaping techniques significantly improve sentence embeddings from BERT-based and simple baseline models.

BERT sentence embeddings representation-shaping natural language processing pretrained models

2024-08-15abs ↗pdf ↗

Understanding Transferable Representation Learning and Zero-shot Transfer in CLIPcs.LG

CLIP learns joint image-text representations for zero-shot learning.

problem Understanding and improving zero-shot transfer performance in CLIP.

method Formal study of transferrable representation learning and analysis of zero-shot transfer performance.

result Proposes a new CLIP-type approach that outperforms existing methods.

CLIP representation learning zero-shot transfer contrastive pretraining natural image generation

2023-10-02abs ↗pdf ↗

Explore and Exploit the Diverse Knowledge in Model Zoo for Domain Generalizationcs.LG

This paper improves model generalization by integrating diverse pretrained models.

problem Leveraging diverse pretrained models for robust out-of-distribution generalization.

method Characterize and integrate diverse pretrained models based on diversity and correlation shifts.

result Demonstrates state-of-the-art out-of-distribution generalization performance.

pretrained models domain generalization diversity correlation out-of-distribution

2023-06-05abs ↗pdf ↗

Transfer Learning or Self-supervised Learning? A Tale of Two Pretraining Paradigmscs.CV

This study compares transfer learning and self-supervised learning for better model performance.

problem Choosing between transfer learning and self-supervised learning for optimal model performance.

method Comprehensive comparative study of transfer learning and self-supervised learning under various data and task properties.

result Self-supervised learning outperforms transfer learning in certain applications, and vice versa.

pretraining transfer learning self-supervised learning model performance data properties

2020-06-19abs ↗pdf ↗

Domain Adversarial Fine-Tuning as an Effective Regularizercs.LG

AFTER technique improves NLP models by preventing overfitting to task-specific domains.

problem Standard fine-tuning degrades pretraining domain representations.

method Complements task-specific loss with adversarial objective.

result AFTER leads to improved performance on various NLP tasks.

NLP pretrained models fine-tuning domain adaptation adversarial training

2020-09-28abs ↗pdf ↗

Unsupervised Learning using Pretrained CNN and Associative Memory Bankcs.CV

Deep Convolutional features extracted from a comprehensive labeled dataset, contain substantial representations which could be effectively used in a new domain. Despite the fact that generic features achieved good results in many visual tasks, fine-tuning is required for pretrained deep CNN models to be more effective …

2018-05-02abs ↗pdf ↗

Improved Baselines with Representation Autoencoderscs.CV

RAE improves image representation learning with simplified design choices.

problem Improving image representation learning using pretrained vision encoders.

method Generalized RAE formulation, complementary working mechanisms of RAE and REPA, and free CFG guidance.

result RAEv2 achieves state-of-the-art results with 10x faster convergence and less training time.

Representation Learning Autoencoders Pretrained Models Image Classification AutoGuidance

2026-05-18abs ↗pdf ↗

Transformers are Minimax Optimal Nonparametric In-Context Learnersstat.ML

Transformers learn new tasks from few examples via optimal approximation.

problem Learning new tasks from limited examples using large language models.

method Developed approximation and generalization error bounds for transformers trained on nonparametric regression tasks.

result Transformers achieve minimax optimal estimation risk in context.

Transformers In-context learning Statistical learning Nonparametric regression Minimax optimality

2024-08-22abs ↗pdf ↗

Flaky Performances when Pretraining on Relational Databasescs.LG

Flaky performance found in GNN SSL on RDBs, leading to worse linear evaluation.

problem Downstream task performances of GNN SSL on RDBs are poor.

method Proposed InfoNode to maximize mutual information between initial and final node representations.

result InfoNode improves GNN SSL performance on RDBs, supporting conjecture of conflict between SSL and GNN message passing.

Graph Neural Networks Relational Databases Self-Supervised Learning Contrastive Loss Mutual Information

2022-11-09abs ↗pdf ↗

Towards Understanding the Transferability of Deep Representationscs.LG

Deep neural networks trained on a wide range of datasets demonstrate impressive transferability. Deep features appear general in that they are applicable to many datasets and tasks. Such property is in prevalent use in real-world applications. A neural network pretrained on large datasets, such as ImageNet, can signifi…

2019-09-26abs ↗pdf ↗

Pretraining Transformer-Based Models on Diffusion-Generated Synthetic Graphs for Alzheimer's Disease Predictioncs.LG

Transformer model pretrains on synthetic graphs for AD detection.

problem Limited labeled data and class imbalance in AD diagnosis.

method Diffusion-generated synthetic graphs, Graph Transformers, transfer learning.

result Framework outperforms baselines in AD diagnosis metrics.

Alzheimer's disease Transformer models Graph Transformers Synthetic data Diffusion models

2025-11-24abs ↗pdf ↗

Representation Learning via Invariant Causal Mechanismscs.LG

Paper analyzes self-supervised learning using causal methods and proposes a new objective.

problem Lack of theoretical understanding of self-supervised learning success.

method Uses a causal framework to enforce invariance constraints on proxy classifiers.

result ReLIC objective improves generalization guarantees and outperforms existing methods.

self-supervised learning representation learning causal inference invariance contrastive learning

2020-10-15abs ↗pdf ↗

Can Tabular In-Context Learners Generalize to Biomolecular Property Prediction?cs.LG

Tabular in-context learners perform well on biomolecular tasks, but performance depends on the representation used.

problem Predicting biomolecular properties from limited labeled data.

method Evaluating tabular in-context learners on protein fitness regression and small-molecule classification tasks.

result Tabular in-context learners are competitive for protein fitness regression but not for small-molecule classification.

tabular in-context learning biomolecular property prediction protein engineering small-molecule design representation learning

2026-06-30abs ↗pdf ↗

How Effective is Task-Agnostic Data Augmentation for Pretrained Transformers?cs.LG

Task-agnostic data augmentation shows little benefit for pretrained transformers.

problem Evaluating the effectiveness of task-agnostic data augmentation on pretrained transformers.

method Conducted a systematic examination of two data augmentation techniques (Easy Data Augmentation and Back-Translation) across 5 tasks, 6 datasets, and 3 pretrained transformer models.

result Data augmentation techniques previously effective for non-pretrained models fail to consistently improve performance for pretrained transformers, even with limited training data.

data augmentation pretrained transformers task agnostic NLP computer vision

2020-10-05abs ↗pdf ↗

Evaluating Self-Supervised Learning via Risk Decompositioncs.LG

Researchers propose a new SSL risk decomposition method to evaluate and improve self-supervised learning models.

problem Self-supervised learning evaluation is limited to a single metric, providing little insight into model performance and improvement.

method Proposes an SSL risk decomposition that considers four error components: approximation, representation usability, probe generalization, and encoder generalization.

result Analysis of 169 SSL vision models reveals the main sources of error and provides insights for improving SSL models in specific settings.

self-supervised learning risk decomposition image classification representation learning probe generalization

2023-02-06abs ↗pdf ↗

Layer by layer, module by module: Choose both for optimal OOD probing of ViTcs.CV

Paper analyzes why deeper layers of ViTs perform worse on out-of-distribution tasks.

problem Performance degradation of intermediate layers in ViTs under distribution shift.

method Extensive linear probing experiments across various benchmarks and fine-grained analysis of transformer modules.

result Probing feedforward network activations yields best performance under significant distribution shift.

ViT Vision Transformers Intermediate Layers Distribution Shift OOD Probing

2026-03-05abs ↗pdf ↗

To Pretrain or Not to Pretrain: Examining the Benefits of Pretraining on Resource Rich Taskscs.CL

Pretraining models improves text classification accuracy, but diminishing returns are observed with large datasets.

problem Improving text classification accuracy with pretrained models.

method Examined the benefits of pretrained models on text classification tasks with varying amounts of training data.

result As the number of training examples grows into the millions, the accuracy gap between pretrained BERT-based models and vanilla LSTM narrows to within 1%.

pretraining text classification MLM BERT LSTM

2020-06-15abs ↗pdf ↗

Improving Pretraining Data Using Perplexity Correlationscs.CL

New framework selects high-quality pretraining data without training LLMs.

problem Slow progress in understanding pretraining data due to costly experiments.

method Statistical framework based on perplexity-benchmark correlations.

result Approach outperforms existing methods on multiple benchmarks.

pretraining data language models correlation downstream performance statistical selection

2024-09-09abs ↗pdf ↗

How Many Pretraining Tasks Are Needed for In-Context Learning of Linear Regression?stat.ML

Paper studies how few pretraining tasks are needed for a linear model to solve new tasks.

problem How many pretraining tasks are needed for a linear model to solve new tasks?

method Pretrained a linear attention model for linear regression with a Gaussian prior.

result Effective pretraining requires a small number of independent tasks, and the model closely matches Bayes optimal.

in-context learning linear regression attention model pretraining tasks Bayes optimal

2023-10-12abs ↗pdf ↗

Why Fine-grained Labels in Pretraining Benefit Generalization?cs.LG

Fine-grained pretraining improves neural network's ability to learn rare features.

problem Improving generalization in deep learning models.

method Introducing a hierarchical multi-view structure to confine input data distribution.

result Fine-grained pretraining leads to better accuracy on hard downstream test samples.

deep learning pretraining generalization neural networks feature learning

2024-10-30abs ↗pdf ↗

The Construction of Instruction-tuned LLMs for Finance without Instruction Data Using Continual Pretraining and Model Mergingcs.CL

Method constructs finance LLMs without instruction data using pretraining and model merging.

problem Developing domain-specific LLMs for finance is resource-intensive.

method Continual pretraining on financial data + model merging of instruction-tuned and domain-specific pretrained vectors.

result Successfully constructs instruction-tuned LLMs for finance without additional instruction data.

LLM finance instruction-tuning continual pretraining model merging

2024-09-30abs ↗pdf ↗

Towards Understanding Why Mask-Reconstruction Pretraining Helps in Downstream Taskscs.LG

Mask-reconstruction pretraining helps in downstream tasks by capturing more semantic features.

problem How mask-reconstruction pretraining helps in downstream tasks and why it surpasses supervised learning.

method Theoretical analysis and experimental validation of mask-reconstruction pretraining (MRP) on auto-encoders.

result MRP provably captures more semantic features than supervised learning, leading to better performance in downstream tasks.

mask-reconstruction pretraining unsupervised learning auto-encoders semantic feature learning downstream tasks

2022-06-08abs ↗pdf ↗

Accelerating Posterior Inference from Pulsar Light Curves via Learned Latent Representations and Local Simulator-Guided Optimizationstat.ML

Accelerates pulsar light curve inference with learned representations and optimization.

problem Computational expense of Markov chain Monte Carlo methods for posterior inference.

method Combining U-Net latent representations with local simulator-guided optimization.

result 120x reduction in inference time (24 hours to 12 minutes) with accuracy preserved.

pulsar light curves posterior inference Markov chain Monte Carlo U-Net simulator-guided optimization

2026-02-16abs ↗pdf ↗

Hierarchical Retrieval: The Geometry and a Pretrain-Finetune Recipecs.IR

Improves retrieval accuracy for hierarchical documents, especially for distant matches.

problem Limited expressive power of dual encoder models in hierarchical retrieval.

method Proves feasibility of DEs for HR, introduces pretrain-finetune recipe to improve long-distance retrieval.

result Pretrain-finetune boosts recall on long-distance pairs from 19% to 76%.

hierarchical retrieval pretrain-finetune long-distance retrieval Euclidean geometry wordnet hierarchy

2025-09-19abs ↗pdf ↗

Multi-Stage Influence Functioncs.LG

Paper introduces a multi-stage influence function to track model predictions.

problem Improving natural language processing and computer vision performance.

method Develops a multi-stage influence function to track predictions from finetuned models back to pretraining data.

result Identifies pretraining examples contributing most to finetuning task predictions.

influence function pretraining finetuning natural language processing computer vision

2020-07-17abs ↗pdf ↗

The SSL Interplay: Augmentations, Inductive Bias, and Generalizationstat.ML

SSL theory improves representation learning from raw data.

problem Challenges in SSL, including instability and collapse.

method Precise analysis of generalization performance with a theory-friendly setup.

result Insights for SSL practitioners on data augmentation, network architecture, and training algorithm.

self-supervised learning representation learning raw data generalization data augmentation

2023-02-06abs ↗pdf ↗

Large Scale Adversarial Representation Learningcs.CV

Adversarially trained generative models (GANs) have recently achieved compelling image synthesis results. But despite early successes in using GANs for unsupervised representation learning, they have since been superseded by approaches based on self-supervision. In this work we show that progress in image generation qu…

2019-07-04abs ↗pdf ↗