This work studies the impact of intra-/inter-class diversity on pre-training datasets and finds a balance for optimal performance.
problem The impact of intra-/inter-class diversity on supervised pre-training datasets and their effect on downstream tasks.
method Empirical study and theoretical analysis of the relationship between diversity types and downstream performance.
result The optimal class-to-sample ratio is invariant to the size of the pre-training dataset and can be predicted.
Self-training outperforms pre-training on COCO object detection and segmentation datasets.
problem The effectiveness of pre-training in improving object detection and segmentation models is limited.
method Investigated self-training as an alternative method to utilize additional data.
result Self-training consistently improves model performance across various dataset sizes and data augmentation levels.
Improved tabular models learn better from real-world data.
problem Tabular models perform poorly on real-world datasets when trained only on synthetic data.
method Continued pre-training on a curated set of real-world datasets.
result Real-TabPFN achieves superior predictive accuracy on 29 datasets.
This paper ranks pre-trained DNNs using a novel SI measure.
problem Optimizing pre-trained DNN selection for transfer learning.
method Automated ranking via Separation Index (SI) on target datasets.
result Ranked pre-trained DNNs improve classification performance.
We present a novel approach to leverage large unlabeled datasets by pre-training state-of-the-art deep neural networks on randomly-labeled datasets. Specifically, we train the neural networks to memorize arbitrary labels for all the samples in a dataset and use these pre-trained networks as a starting point for regular…
GCC pre-trains graph neural networks to transfer across diverse datasets.
problem Non-transferable graph models trained for specific datasets.
method Graph Contrastive Coding (GCC) framework using contrastive learning.
result GCC pre-trained models achieve competitive performance across multiple datasets.
Foundation models trained on collider data improve jet generation tasks.
problem Improving foundation models for jet generation tasks.
method Pre-training OmniJet-α model on AspenOpenJets dataset. result Pre-trained model improves performance on jet generation tasks with domain shift.
The study finds a theoretical bound for pre-training iterations needed for pruning to yield good subnetwork performance.
problem Discovering efficient subnetworks within pre-trained dense networks.
method Mathematical analysis of a two-layer, fully-connected network, validating with a multi-layer perceptron trained on MNIST.
result A logarithmically dependent threshold on dataset size for successful pruning.
Study builds a Japanese financial-specific LLM through continual pre-training.
problem Lack of domain-specific Japanese financial LLMs.
method Continual pre-training on Japanese financial-focused datasets using a base Japanese LLM.
result Tuned model outperforms original model on Japanese financial benchmarks.
Pre-trained model from healthy ADLs improves gait pattern classification for Parkinson's disease.
problem Limited training data for deep learning models in healthcare applications.
method Used convolutional autoencoder to extract features from healthy ADLs data and trained a multi-layer perceptron model for Parkinson's disease classification.
result Features extracted from healthy ADLs data can be used to train an effective classification model for Parkinson's disease.
New measure assesses time series pre-training data quality without labels.
problem Challenges in collecting diverse pre-training datasets for time series classification.
method Contrastive-learning-based foundation model and contrastive accuracy measure.
result Contrastive accuracy correlates with model performance on downstream tasks.
Many applications of machine learning require a model to make accurate pre-dictions on test examples that are distributionally different from training ones, while task-specific labels are scarce during training. An effective approach to this challenge is to pre-train a model on related tasks where data is abundant, and…
Researchers develop method to protect against 'weight poisoning' attacks on pre-trained models.
problem The security threat of downloading untrusted pre-trained weights that can be manipulated after fine-tuning.
method RIPPLe regularization method and Embedding Surgery initialization procedure.
result Demonstrated that weight poisoning attacks are possible even with limited knowledge of the dataset and fine-tuning procedure.
Named Entity Recognition (NER) is a fundamental Natural Language Processing (NLP) task to extract entities from unstructured data. The previous methods for NER were based on machine learning or deep learning. Recently, pre-training models have significantly improved performance on multiple NLP tasks. In this paper, fir…
CLIP dataset helps extract action items from hospital discharge notes.
problem Lost action items in long discharge notes hinder information sharing.
method Created CLIP dataset of annotated discharge notes, used multi-aspect extractive summarization, trained models on pre-trained language models and context.
result Best models improved by incorporating context and pre-trained language models.
The paper predicts loss scaling across different datasets and compute scales.
problem Predicting loss scaling across different datasets and compute scales.
method Derive shifted power law relationships between train and test losses.
result Shifted power law relationships hold for various datasets and tasks, improving prediction accuracy.
Method curates cost-effective, high-quality datasets using AI models.
problem Costly manual labeling of datasets.
method Probably Approximately Correct Labels (PACL) method.
result Curates high-quality datasets with low overall labeling error.
Paper improves molecular property prediction using denoising autoencoders.
problem Limited data for molecular property prediction from 3D structures.
method Pre-training via denoising for learning molecular force fields.
result Achieves new state-of-the-art performance on QM9 dataset.
Pre-trained GPs improve Bayesian optimization efficiency.
problem Optimizing complex machine learning models requires domain knowledge but is challenging.
method Developed HyperBO framework using pre-trained Gaussian processes.
result HyperBO achieves up to 3x better efficiency in hyperparameter tuning.
Delphyne improves financial time series models with pre-trained language models.
problem Lack of financial data and negative transfer effect in existing time-series pre-trained models.
method Delphyne is a pre-trained model for financial time series that addresses the lack of financial data and negative transfer effect.
result Delphyne achieves competitive performance and superior performances on various financial tasks.
Mimetic initialization improves Transformer training on small datasets.
problem Difficulty in training Transformers on small datasets.
method Initialize self-attention layers to look like pre-trained models.
result Vanilla Transformers trained with mimetic initialization achieve higher accuracy.
In drug-discovery-related tasks such as virtual screening, machine learning is emerging as a promising way to predict molecular properties. Conventionally, molecular fingerprints (numerical representations of molecules) are calculated through rule-based algorithms that map molecules to a sparse discrete space. However,…
Unsupervised pre-training improves model generalization, but lacks theoretical understanding.
problem Lack of theoretical understanding of unsupervised pre-training's impact on model generalization.
method Introduces a novel theoretical framework to analyze and enhance generalization.
result Enhances understanding of unsupervised pre-training and fine-tuning, proposing a new regularization method.
In this work we focus on fine-tuning a pre-trained BERT model and applying it to patent classification. When applied to large datasets of over two millions patents, our approach outperforms the state of the art by an approach using CNN with word embeddings. In addition, we focus on patent claims without other parts in …
Ensemble pre-trained models for low data transfer learning.
problem Training good models from scratch in low data regime.
method Fine-tune nearest-neighbour ranked pre-trained models to create ensembles.
result Achieves state-of-the-art performance with lower inference budget.
Our method upweights easy samples to mitigate forgetting in fine-tuning.
problem Catastrophic forgetting in fine-tuning pre-trained models.
method Sample weighting based on pre-trained model's losses.
result Our method reduces forgetting by up to 0.8% on MetaMathQA while preserving more accuracy on pre-training datasets.
Personalizes pre-trained models for nonparametric regression with limited data.
problem Improving data efficiency in nonparametric regression with few samples.
method Develops a theoretical framework and algorithms for few-shot personalization of black-box models.
result Achieves minimax optimal rate for personalization in nonparametric regression.
TSFMs improve financial forecasting from diverse datasets.
problem Challenges in forecasting financial time series due to noisy, non-stationary, and heterogeneous data.
method Empirical study of TSFMs in global financial markets, evaluating zero-shot inference, fine-tuning, and pre-training from scratch.
result Pre-trained TSFMs on financial data achieve substantial forecasting and economic improvements, highlighting the value of domain-specific adaptation.
Paper develops large time series models using pre-trained transformers.
problem Performance bottlenecks in small models on data-scarce scenarios.
method Large-scale pre-training, unified time series format, GPT-style architecture.
result Generative pre-trained Time Series Transformer (Timer) for diverse tasks.
Deep Neural Networks (DNNs) have become increasingly popular in computer vision, natural language processing, and other areas. However, training and fine-tuning a deep learning model is computationally intensive and time-consuming. We propose a new method to improve the performance of nearly every model including pre-t…
Two things seem to be indisputable in the contemporary deep learning discourse: 1. The categorical cross-entropy loss after softmax activation is the method of choice for classification. 2. Training a CNN classifier from scratch on small datasets does not work well. In contrast to this, we show that the cosine loss fun…
Pre-training improves Bayesian optimization efficiency.
problem Optimizing complex machine learning models with hyperparameters.
method Pre-training a tighter Gaussian process prior from similar functions.
result 3 times more efficient hyperparameter tuning on average.
Recent state-of-the-art language models utilize a two-phase training procedure comprised of (i) unsupervised pre-training on unlabeled text, and (ii) fine-tuning for a specific supervised task. More recently, many studies have been focused on trying to improve these models by enhancing the pre-training phase, either vi…
Post-hoc uncertainty quantification improves on pre-trained neural networks without underfitting.
problem Uncertainty quantification in neural networks is underfitting or computationally demanding.
method Gaussian Process Activation function (GAPA) for neuron-level uncertainty, with two methods: GAPA-Free and GAPA-Variational.
result GAPA-Variational outperforms Laplace approximation on most datasets in uncertainty quantification metrics.
Simple method disentangles content and style from pre-trained vision models.
problem Learning interpretable features in visual representations.
method Probabilistic linear entanglement model and simple disentanglement algorithm.
result Method provably disentangles content and style features.
Researchers show how to secretly train models with hidden data, detect usage with high confidence.
problem Protecting training data from traceability in large language models.
method Gradient-based optimization to learn secret sequences absent from training data.
result Secret sequences can be learned by models without performance degradation, detectable with high confidence.
Paper proposes a unified time series forecasting model with adaptive transfer.
problem General forecasting models for diverse time series data.
method Unified representations through Decomposed Frequency Learning and adaptive domain-specific features via Time Series Register.
result State-of-the-art forecasting performance on seven real-world benchmarks.
Two methods are proposed for high-dimensional shape-constrained regression and classification. These methods reshape pre-trained prediction rules to satisfy shape constraints like monotonicity and convexity. The first method can be applied to any pre-trained prediction rule, while the second method deals specifically w…
Kronos improves financial time series analysis with a pre-trained model.
problem Limited application of large-scale models to financial candlestick data.
method Unified, scalable pre-training framework for financial K-line modeling.
result Kronos excels in financial tasks like price forecasting and volatility prediction.
This work attempts to find the most optimal parameter setting of a deep artificial neural network (ANN) for Bengali digit dataset by pre-training it using stacked denoising autoencoder (SDA). Although SDA based recognition is hugely popular in image, speech and language processing related tasks among the researchers, i…
Paper provides conditions for reliable use of pre-trained embeddings in econometrics.
problem Uncertainty in using pre-trained embeddings for econometric tasks.
method Derives sufficient conditions and convergence rates for machine learning models with pre-trained embeddings.
result Establishes theoretical foundations for reliable use of pre-trained embeddings in econometrics.
Proposes efficient deep causal generative models for high-dimensional causal inference.
problem Inefficient training of deep generative models for high-dimensional data.
method Modular training of deep causal generative models using adversarial training and pre-trained models.
result First algorithm that provably samples from any identifiable causal query in the presence of latent confounders.
A RL-based method adds conditional controls to pre-trained diffusion models.
problem Precise control over generated samples in diffusion models.
method Formulates the task as an RL problem, using a classifier and KL divergence as reward functions.
result Produces soft-optimal policies that maximize reward functions, enabling conditional sampling.
Study compares deep feature methods for anomaly detection in limited data scenarios.
problem Handling limited data in industrial inspection applications.
method Three approaches (KNN, Mahalanobis, PaDiM) using pre-trained deep features with data augmentation.
result Data augmentation significantly improves performance in small data regimes.
Method transfers knowledge without label overlap, source data, or target architecture consistency.
problem Difficulties in transfer learning due to label mismatch, restricted source data, and specialized target architectures.
method Uses deep generative models in two stages: pseudo pre-training and pseudo semi-supervised learning.
result Outperforms scratch training and knowledge distillation methods.
Simplified image clustering achieves competitive results without text-based embeddings.
problem Complexity and resource requirements of state-of-the-art clustering methods.
method SCP: trains a small cluster head using pre-trained vision model features and positive data pairs.
result SCP achieves highly competitive performance on various benchmark datasets.
As a mental disorder progresses, it may affect brain structure, but brain function expressed in brain dynamics is affected much earlier. Capturing the moment when brain dynamics express the disorder is crucial for early diagnosis. The traditional approach to this problem via training classifiers either proceeds from ha…
System helps engineers with concept recognition for SEVA.
problem Recognizing systems engineering concepts for SEVA.
method Token classification task, domain expert labeling, pre-trained model fine-tuning, essential datasets creation, knowledge graph construction.
result System successfully recognizes systems engineering concepts.