On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,742 papers · 148 categories

Trend · papers per month

count % of corpus

48 results for “supervised pretraining”

Self-Supervised Pretraining for Heterogeneous Hypergraph Neural Networkscs.LG

Self-supervised pretraining for heterogeneous hypergraphs improves graph-based tasks.

problem Capturing higher-order relations in heterogeneous hypergraphs.

method SPHH framework for self-supervised pretraining of heterogeneous HyperGNNs.

result SPHH consistently outperforms state-of-the-art baselines in various downstream tasks.

Hypergraphs Self-supervised Pretraining Heterogeneous Graph Neural Networks

2023-11-19abs ↗pdf ↗

Transformers as Decision Makers: Provable In-Context Reinforcement Learning via Supervised Pretrainingcs.LG

Transformers learn to make decisions in new contexts from offline data.

problem Understanding when and how transformers can perform in-context reinforcement learning.

method Theoretical framework analyzing supervised pretraining for ICRL, including algorithm distillation and decision-pretrained transformers.

result Transformers can efficiently approximate optimal reinforcement learning algorithms for various environments.

transformers in-context reinforcement learning supervised pretraining algorithm distillation decision-pretrained transformers

2023-10-12abs ↗pdf ↗

To Pretrain or Not to Pretrain: Examining the Benefits of Pretraining on Resource Rich Taskscs.CL

Pretraining models improves text classification accuracy, but diminishing returns are observed with large datasets.

problem Improving text classification accuracy with pretrained models.

method Examined the benefits of pretrained models on text classification tasks with varying amounts of training data.

result As the number of training examples grows into the millions, the accuracy gap between pretrained BERT-based models and vanilla LSTM narrows to within 1%.

pretraining text classification MLM BERT LSTM

2020-06-15abs ↗pdf ↗

Towards Understanding Why Mask-Reconstruction Pretraining Helps in Downstream Taskscs.LG

Mask-reconstruction pretraining helps in downstream tasks by capturing more semantic features.

problem How mask-reconstruction pretraining helps in downstream tasks and why it surpasses supervised learning.

method Theoretical analysis and experimental validation of mask-reconstruction pretraining (MRP) on auto-encoders.

result MRP provably captures more semantic features than supervised learning, leading to better performance in downstream tasks.

mask-reconstruction pretraining unsupervised learning auto-encoders semantic feature learning downstream tasks

2022-06-08abs ↗pdf ↗

Big Self-Supervised Models are Strong Semi-Supervised Learnerscs.LG

Big models pretrain and fine-tune for semi-supervised learning on ImageNet.

problem Learning from few labeled examples with a large amount of unlabeled data.

method Unsupervised pretraining of a big ResNet model followed by supervised fine-tuning and distillation.

result 73.9% ImageNet top-1 accuracy with just 1% of labels (

\le

13 labeled images per class).

semi-supervised learning ImageNet big models ResNet pretraining

2020-06-17abs ↗pdf ↗

CERT: Contrastive Self-supervised Learning for Language Understandingcs.CL

CERT improves language understanding by contrastively learning sentence-level semantics.

problem Lack of sentence-level semantics in existing pretraining tasks.

method Contrastive self-supervised learning at the sentence level using back-translation augmentations.

result CERT outperforms BERT on 7 out of 11 GLUE benchmark tasks, achieving the same performance as BERT on 2 tasks.

BERT GPT contrastive learning language understanding pretraining

2020-05-16abs ↗pdf ↗

Transfer Learning or Self-supervised Learning? A Tale of Two Pretraining Paradigmscs.CV

This study compares transfer learning and self-supervised learning for better model performance.

problem Choosing between transfer learning and self-supervised learning for optimal model performance.

method Comprehensive comparative study of transfer learning and self-supervised learning under various data and task properties.

result Self-supervised learning outperforms transfer learning in certain applications, and vice versa.

pretraining transfer learning self-supervised learning model performance data properties

2020-06-19abs ↗pdf ↗

Deep Nearest Neighbor Anomaly Detectioncs.LG

Deep nearest neighbors outperform self-supervised methods in anomaly detection.

problem Anomaly detection using self-supervised deep methods.

method Simple nearest-neighbor approach on Imagenet pretrained features.

result Nearest-neighbor method outperforms self-supervised methods in accuracy, few shot generalization, training time, and noise robustness.

anomaly detection nearest neighbors deep learning self-supervised few shot

2020-02-24abs ↗pdf ↗

An empirical study of pretrained representations for few-shot classificationcs.LG

Recent algorithms with state-of-the-art few-shot classification results start their procedure by computing data features output by a large pretrained model. In this paper we systematically investigate which models provide the best representations for a few-shot image classification task when pretrained on the Imagenet …

2019-10-03abs ↗pdf ↗

Selfie: Self-supervised Pretraining for Image Embeddingcs.LG

We introduce a pretraining technique called Selfie, which stands for SELFie supervised Image Embedding. Selfie generalizes the concept of masked language modeling of BERT (Devlin et al., 2019) to continuous data, such as images, by making use of the Contrastive Predictive Coding loss (Oord et al., 2018). Given masked-o…

2019-06-07abs ↗pdf ↗

Unsupervised Program Synthesis for Images By Sampling Without Replacementcs.LG

Unsupervised algorithm parses CSG images into CFG without pretraining.

problem Sparse reward problem in unsupervised program synthesis for images.

method Grammar-encoded tree LSTM, entropy regularization, sampling without replacement.

result Recover meaningful programs in large search spaces (up to

3.8 imes 10^{28}

unsupervised learning program synthesis image parsing grammar encoding entropy regularization

2020-01-27abs ↗pdf ↗

Probing Few-Shot Generalization with Attributescs.LG

Study reveals attributes help in few-shot classification generalization.

problem Understanding what makes some novel classes easier to learn.

method Defined attributes to quantify concept relatedness, used supervised and self-supervised learning.

result Combining self-supervised pretraining with supervised finetuning improves generalization.

attributes few-shot classification generalization self-supervised supervised finetuning

2020-12-10abs ↗pdf ↗

Unsupervised Learning using Pretrained CNN and Associative Memory Bankcs.CV

Deep Convolutional features extracted from a comprehensive labeled dataset, contain substantial representations which could be effectively used in a new domain. Despite the fact that generic features achieved good results in many visual tasks, fine-tuning is required for pretrained deep CNN models to be more effective …

2018-05-02abs ↗pdf ↗

Self-Supervised Dynamical System Representations for Physiological Time-Seriescs.LG

PULSE learns representations from physiological time series.

problem Lack of effective pretraining objectives for physiological time series.

method Proposes a pretraining framework exploiting dynamical systems generative model.

result PULSE learns representations that distinguish semantic classes and improve transfer learning.

self-supervised learning physiological time series dynamical systems pretraining representation learning

2025-11-28abs ↗pdf ↗

Training few-shot classification via the perspective of minibatch and pretrainingcs.LG

Advances few-shot classification by treating it as supervised learning and proposing new training techniques.

problem Formulating the ability of humans to learn from limited data in machine learning.

method Formulated few-shot classification as a supervised learning problem and introduced multi-episode and cross-way training techniques.

result Proposed training strategies accelerate the training process without accuracy loss.

few-shot classification meta-learning supervised learning minibatch pretraining

2020-04-10abs ↗pdf ↗

Incorporating Unlabelled Data into Bayesian Neural Networkscs.LG

New method uses unlabelled data to improve Bayesian Neural Networks.

problem Lack of ability to use unlabelled data in conventional Bayesian Neural Networks.

method Self-supervised Bayesian Neural Networks using contrastive pretraining and variational lower bound optimization.

result Prior predictive distributions capture problem semantics better and improve predictive performance.

Bayesian Neural Networks Unlabelled Data Contrastive Pretraining Variational Lower Bound Predictive Performance

2023-04-04abs ↗pdf ↗

Theoretical Perspectives on Data Quality and Synergistic Effects in Pre- and Post-Training Reasoning Modelscs.LG

Theoretical analysis of data quality and synergies in LLMs.

problem Understanding why different training methods require different amounts of data.

method Theoretical analysis of transformers trained on a weight prediction task for linear regression.

result SFT excels on smaller datasets challenging for the pretrained model, while RL benefits from large, not overly difficult data.

Large Language Models Pretraining Fine-tuning Reinforcement Learning Data Quality

2026-03-01abs ↗pdf ↗

Lateral Connections in Denoising Autoencoders Support Supervised Learningcs.LG

We show how a deep denoising autoencoder with lateral connections can be used as an auxiliary unsupervised learning task to support supervised learning. The proposed model is trained to minimize simultaneously the sum of supervised and unsupervised cost functions by back-propagation, avoiding the need for layer-wise pr…

2015-04-30abs ↗pdf ↗

Representation Learning via Invariant Causal Mechanismscs.LG

Paper analyzes self-supervised learning using causal methods and proposes a new objective.

problem Lack of theoretical understanding of self-supervised learning success.

method Uses a causal framework to enforce invariance constraints on proxy classifiers.

result ReLIC objective improves generalization guarantees and outperforms existing methods.

self-supervised learning representation learning causal inference invariance contrastive learning

2020-10-15abs ↗pdf ↗

Unsupervised Discovery of Interpretable Directions in the GAN Latent Spacecs.LG

Unsupervised method discovers interpretable directions in GAN latent space.

problem Discovering interpretable directions in GAN latent space without supervision.

method Model-agnostic procedure to identify directions corresponding to semantic manipulations.

result Findings include directions for background removal and competitive saliency detection performance.

GAN latent space unsupervised learning semantic manipulations interpretable directions weakly-supervised saliency detection

2020-02-10abs ↗pdf ↗

The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shiftcs.LG

Study shows pretraining and finetuning can effectively tackle covariate shift in linear regression.

problem Linear regression under covariate shift where source and target distributions differ but conditional distribution remains similar.

method Pretraining on source data and finetuning on target data using online SGD.

result Transfer learning with

O(N^2)

source data is as effective as supervised learning with

N

target data.

covariate shift linear regression pretraining finetuning online SGD

2022-08-03abs ↗pdf ↗

Evaluating Self-Supervised Learning via Risk Decompositioncs.LG

Researchers propose a new SSL risk decomposition method to evaluate and improve self-supervised learning models.

problem Self-supervised learning evaluation is limited to a single metric, providing little insight into model performance and improvement.

method Proposes an SSL risk decomposition that considers four error components: approximation, representation usability, probe generalization, and encoder generalization.

result Analysis of 169 SSL vision models reveals the main sources of error and provides insights for improving SSL models in specific settings.

self-supervised learning risk decomposition image classification representation learning probe generalization

2023-02-06abs ↗pdf ↗

Layer by layer, module by module: Choose both for optimal OOD probing of ViTcs.CV

Paper analyzes why deeper layers of ViTs perform worse on out-of-distribution tasks.

problem Performance degradation of intermediate layers in ViTs under distribution shift.

method Extensive linear probing experiments across various benchmarks and fine-grained analysis of transformer modules.

result Probing feedforward network activations yields best performance under significant distribution shift.

ViT Vision Transformers Intermediate Layers Distribution Shift OOD Probing

2026-03-05abs ↗pdf ↗

Ladder Networks for Semi-Supervised Hyperspectral Image Classificationcs.CV

We used the Ladder Network [Rasmus et al. (2015)] to perform Hyperspectral Image Classification in a semi-supervised setting. The Ladder Network distinguishes itself from other semi-supervised methods by jointly optimizing a supervised and unsupervised cost. In many settings this has proven to be more successful than o…

2018-12-04abs ↗pdf ↗

Multi-view self-supervised learning for multivariate variable-channel time seriesstat.ML

A method for self-supervised learning of multivariate time series data across varying channels.

problem Labeling multivariate biomedical time series data is laborious and expensive.

method Proposes a multi-view self-supervised learning approach using a message passing neural network to extract a single representation across channels.

result Our method, combined with the TS2Vec loss, outperforms all other methods in most settings.

self-supervised learning multivariate time series message passing neural network TS2Vec loss multi-view

2023-07-13abs ↗pdf ↗

Pretraining Deep Actor-Critic Reinforcement Learning Algorithms With Expert Demonstrationscs.AI

Pretraining with expert demonstrations have been found useful in speeding up the training process of deep reinforcement learning algorithms since less online simulation data is required. Some people use supervised learning to speed up the process of feature learning, others pretrain the policies by imitating expert dem…

2018-01-31abs ↗pdf ↗

Evaluating Protein Transfer Learning with TAPEcs.LG

Protein modeling is an increasingly popular area of machine learning research. Semi-supervised learning has emerged as an important paradigm in protein modeling due to the high cost of acquiring supervised protein labels, but the current literature is fragmented when it comes to datasets and standardized evaluation tec…

2019-06-19abs ↗pdf ↗

On the Provable Advantage of Unsupervised Pretrainingstat.ML

This paper proves the theoretical advantage of unsupervised pretraining for machine learning tasks.

problem Understanding why unsupervised pretraining helps in machine learning tasks.

method A generic framework using Maximum Likelihood Estimation (MLE) for unsupervised pretraining and Empirical Risk Minimization (ERM) for downstream tasks.

result Proves an excess risk of

ilde{\mathcal{O}}(\sqrt{\mathcal{C}_Φ/m} + \sqrt{\mathcal{C}_Ψ/n})

for downstream tasks under mild conditions.

unsupervised pretraining maximum likelihood estimation empirical risk minimization latent variable models downstream tasks

2023-03-02abs ↗pdf ↗

Contrastive Self-supervised Learning for Graph Classificationcs.LG

Two CSSL-based methods improve graph classification with limited labeled data.

problem Limited labeled data for graph classification leads to overfitting.

method Contrastive self-supervised learning (CSSL) for graph encoders pretraining and regularization.

result CSSL methods reduce overfitting and improve graph classification accuracy.

graph classification self-supervised learning contrastive loss data augmentation unsupervised learning

2020-09-13abs ↗pdf ↗

Supervised Fine-Tuning vs. In-Context Learning: An Equilibrium Analysis of LLM Personalization under Congestioncs.LG

New framework analyzes LLM personalization trade-offs under congestion.

problem Tension between personalization and resource sharing in LLMs.

method Developed a statistical-economic framework to model user incentives.

result Congestion can flip rankings of SFT and ICL, and offers both methods never hurt profits.

Large Language Models Personalization Resource Sharing Supervised Fine-Tuning In-Context Learning

2026-07-15abs ↗pdf ↗

DeepHeart: Semi-Supervised Sequence Learning for Cardiovascular Risk Predictioncs.LG

We train and validate a semi-supervised, multi-task LSTM on 57,675 person-weeks of data from off-the-shelf wearable heart rate sensors, showing high accuracy at detecting multiple medical conditions, including diabetes (0.8451), high cholesterol (0.7441), high blood pressure (0.8086), and sleep apnea (0.8298). We compa…

2018-02-07abs ↗pdf ↗

Contextures: The Mechanism of Representation Learningcs.LG

The dissertation establishes a contexture theory to mathematically characterize representation learning.

problem The lack of a scientific understanding of representation learning in foundation models.

method Introduces the contexture theory as a unified framework for analyzing representation learning methods.

result Representation learning is optimal when the association between input and context is neither too strong nor too weak.

representation learning contexture theory downstream tasks pretraining methods statistical learning bounds

2025-04-28abs ↗pdf ↗

Supervised level-wise pretraining for recurrent neural network initialization in multi-class classificationcs.LG

Recurrent Neural Networks (RNNs) can be seriously impacted by the initial parameters assignment, which may result in poor generalization performances on new unseen data. With the objective to tackle this crucial issue, in the context of RNN based classification, we propose a new supervised layer-wise pretraining strate…

2019-11-04abs ↗pdf ↗

Flaky Performances when Pretraining on Relational Databasescs.LG

Flaky performance found in GNN SSL on RDBs, leading to worse linear evaluation.

problem Downstream task performances of GNN SSL on RDBs are poor.

method Proposed InfoNode to maximize mutual information between initial and final node representations.

result InfoNode improves GNN SSL performance on RDBs, supporting conjecture of conflict between SSL and GNN message passing.

Graph Neural Networks Relational Databases Self-Supervised Learning Contrastive Loss Mutual Information

2022-11-09abs ↗pdf ↗

Self-supervised representation learning from electroencephalography signalscs.LG

The supervised learning paradigm is limited by the cost - and sometimes the impracticality - of data collection and labeling in multiple domains. Self-supervised learning, a paradigm which exploits the structure of unlabeled data to create learning problems that can be solved with standard supervised approaches, has sh…

2019-11-13abs ↗pdf ↗

Semi-Supervised Learning with Noisy Proxy Covariates: Generalization Bounds and Distribution Regressioncs.LG

Study semi-supervised learning with noisy proxy covariates, deriving bounds and showing gains.

problem Learning from noisy proxy covariates with scarce labels.

method Two-stage estimator learning kernel eigenfeatures from all proxy covariates and fitting a ridge predictor on labeled data.

result Finite sample bounds show fast labeled sample rates and consistent gains over supervised and semi-supervised baselines.

semi-supervised learning proxy covariates generalization bounds distribution regression

2026-05-30abs ↗pdf ↗

These Are Not All the Features You Are Looking For: A Fundamental Bottleneck in Supervised Pretrainingcs.LG

New research shows deep models learn sparse features, limiting transfer learning; ensembling improves performance.

problem Sparse feature learning in deep models limits transfer learning performance.

method Developed a theoretical framework and proposed an ensembling strategy to aggregate multiple models.

result Ensembling yields a 9% improvement in transfer accuracy without extra pretraining cost.

transfer learning sparse representations model ensembles deep learning feature aggregation

2025-06-23abs ↗pdf ↗

Pretrained Time-Series Foundation Models for Financial Return Forecastingq-fin.MF

Pretrained time-series models outperform train-from-scratch baselines in financial return forecasting.

problem Financial return forecasting

method Pretrained time-series foundation models

result Pretrained TSFMs dominate the ranking distribution, accounting for 8 of 10 task-level wins.

financial return forecasting pretrained models time-series foundation models TSFMs financial markets

2026-06-25abs ↗pdf ↗

A General Class of Transfer Learning Regression without Implementation Coststat.ML

Unified framework for transfer learning regression without extra cost.

problem Transfer learning for regression models without additional implementation cost.

method Introduces a density-ratio reweighting function estimated via Bayesian framework.

result Unified integration of three TL methods with no extra cost.

transfer learning regression density-ratio Bayesian supervised learning

2020-06-23abs ↗pdf ↗

Towards Few-Annotation Learning in Computer Vision: Application to Image Classification and Object Detection taskscs.CV

Develops methods for training models with few annotations in computer vision tasks.

problem Training models with limited labeled data in computer vision.

method Theoretical, algorithmic, and experimental contributions for Meta-Learning and Semi-Supervised Learning.

result Improves Contrastive Learning and proposes a semi-supervised method for transformer-based object detectors.

Few-Annotation Learning Computer Vision Image Classification Object Detection Transformer Architecture

2023-11-08abs ↗pdf ↗

In-Context Positive-Unlabeled Learningstat.ML

In-context learning solves PU classification without iterative optimization.

problem Binary classification with only labeled positives and unlabeled samples.

method Pretrained transformer (PUICL) that learns from synthetic PU datasets.

result Outperforms four standard PU learning baselines on 20 benchmarks.

in-context learning PU learning transformer semi-supervised binary classification

2026-05-07abs ↗pdf ↗

When Does Self-Supervision Help Graph Convolutional Networks?cs.LG

Self-supervision improves GCNs' generalizability and robustness.

problem Improving graph convolutional networks' performance.

method Three mechanisms of self-supervision, multi-task learning, and graph adversarial training.

result Self-supervision enhances GCNs' robustness and generalizability.

graph convolutional networks self-supervision multi-task learning graph adversarial training generalizability

2020-06-16abs ↗pdf ↗

Forecasting Cryptocurrency Returns from Sentiment Signals: An Analysis of BERT Classifiers and Weak Supervisionq-fin.ST

Study improves cryptocurrency price prediction using unlabeled text data.

problem Predicting cryptocurrency returns from unlabelled text data.

method Introduced weak learning approach to finetune BERT on unlabeled text data.

result Finetuning pretrained NLP models with weak labels enhances forecast accuracy.

cryptocurrency sentiment analysis BERT weak supervision forecasting

2022-04-06abs ↗pdf ↗

Revisiting Theory of Contrastive Learning for Domain Generalizationstat.ML

New bounds for contrastive learning handle domain shifts and generalization.

problem Domain shifts and generalization challenges in downstream tasks.

method Novel generalization bounds accounting for both domain shift and generalization.

result Performance of contrastively learned representations depends on statistical discrepancy between pretraining and downstream distributions.

contrastive learning domain shifts domain generalization latent class space statistical discrepancy

2025-12-02abs ↗pdf ↗

Training Subset Selection for Weak Supervisionstat.ML

Subset selection improves weak supervision performance.

problem Optimizing the use of weakly-labeled data.

method Combining pretrained data representations with the cut statistic for subset selection.

result Subset selection improves weak supervision performance by up to 19%.

subset selection weak supervision weak labels data representation cut statistic

2022-06-06abs ↗pdf ↗

Unsupervised Pretraining for Fact Verification by Language Model Distillationcs.CL

SFAVEL distills features from language models for fact verification without annotations.

problem Fact verification with semantically meaningful and compact features.

method Self-supervised pretraining using contrastive loss with language models.

result Achieved state-of-the-art results on FB15k-237 and FEVER.

fact verification unsupervised learning language model distillation contrastive loss

2023-09-28abs ↗pdf ↗

The SSL Interplay: Augmentations, Inductive Bias, and Generalizationstat.ML

SSL theory improves representation learning from raw data.

problem Challenges in SSL, including instability and collapse.

method Precise analysis of generalization performance with a theory-friendly setup.

result Insights for SSL practitioners on data augmentation, network architecture, and training algorithm.

self-supervised learning representation learning raw data generalization data augmentation

2023-02-06abs ↗pdf ↗

PanRep: Graph neural networks for extracting universal node embeddings in heterogeneous graphscs.LG

PanRep learns universal node embeddings for heterogeneous graphs.

problem Learning universal node embeddings for heterogeneous graphs.

method Graph Neural Network (GNN) model with four decoders capturing different properties.

result PanRep outperforms unsupervised and supervised methods in node classification and link prediction.

Graph Neural Networks Universal Node Embeddings Heterogeneous Graphs Node Classification Link Prediction

2020-07-20abs ↗pdf ↗