Paper develops a theory explaining contrastive pre-training for multimodal AI.
problem Limited theoretical understanding of contrastive pre-training for multi-modal AI.
method Introduces approximate sufficient statistics and Joint Generative Hierarchical Model.
result Near-minimizers of contrastive loss are approximately sufficient, enabling diverse downstream tasks.
HyperBO+ pre-trains a universal prior for Bayesian optimization across different domains.
problem Bayesian optimization requires domain-specific priors, limiting its applicability.
method Two-step pre-training method for hierarchical Gaussian processes.
result HyperBO+ achieves lower regrets on unseen search spaces.
Improved simulation of phase transitions using hierarchical autoregressive networks.
problem Simulating phase transitions in complex systems.
method Hierarchical Autoregressive Neural (HAN) network sampling algorithm.
result Significant improvement in statistical uncertainty compared to the Wolff cluster algorithm.
Unsupervised pre-training preserves multiple views from input distribution.
problem Learning from multiple views of supervision in unsupervised settings.
method Investigates unsupervised pre-training under information theory, focusing on disentanglement and multiple views.
result Unsupervised pre-training helps conserve multiple views from input distribution, improving model performance.
Unified QuesNet learns comprehensive representations for diverse test questions.
problem Lack of labeled data for test questions in online learning systems.
method Unified framework and two-level hierarchical pre-training algorithm for unsupervised learning of heterogeneous question representations.
result QuesNet effectively learns comprehensive question representations and outperforms existing methods.
Self-supervised learning improves few-shot classification and segmentation on point clouds.
problem Efficiently learn from limited labeled data in point cloud applications.
method Hierarchical cover-tree partitioning for self-supervised pre-training; restricted to support set for few-shot learning.
result Self-supervised learning significantly improves downstream classification and segmentation accuracy.
Proposes EM-HRNN model for better hierarchical language representation.
problem Learning hierarchical representations in natural language processing.
method Introduces latent indicator layer and EM algorithm for training.
result EM-HRNN model outperforms other RNN-based models in document classification.
Wind energy forecasting helps to manage power production, and hence, reduces energy cost. Deep Neural Networks (DNN) mimics hierarchical learning in the human brain and thus possesses hierarchical, distributed, and multi-task learning capabilities. Based on aforementioned characteristics, we report Deep Belief Network …
Enhances topic models to better handle polysemous words.
problem Lack of polysemy handling in Gaussian latent Dirichlet allocation.
method Introduces a hierarchical structure to capture polysemy in Gaussian latent Dirichlet allocation.
result Significantly improves polysemy detection and provides more parsimonious topic representations.
Paper proposes a new hierarchical attention mechanism for multi-scale data.
problem Challenges in applying neural attention to multi-scale, multi-modal data.
method Developed a mathematical framework for multi-modal, multi-scale data and derived optimal neural attention mechanics.
result Proposed hierarchical attention mechanism improves transformer performance in multi-scale, multi-modal settings.
Framework harmonizes EHR data across institutions for better analysis.
problem Heterogeneity of medical codes and terminologies hinder EHR data analysis.
method MASH (Multi-source Automated Structured Hierarchy) uses neural optimal transport and learned hyperbolic embeddings to align and structure EHR data.
result MASH generates interpretable hierarchical graphs for unstructured local laboratory codes.
Bayesian nonparametric models improve OOD detection, especially with complex covariance structures.
problem Improving out-of-distribution detection methods, especially in complex scenarios.
method Proposes Bayesian nonparametric mixture models with hierarchical priors that generalize the Mahalanobis distance score.
result Bayesian nonparametric methods outperform existing OOD methods, especially in complex scenarios.
MPHD transfers knowledge across different domains for Bayesian optimization.
problem Optimizing functions with unknown or diverse domains.
method MPHD uses neural nets to map domain-specific contexts to GP specifications, enabling transfer learning across heterogeneous search spaces.
result MPHD improves black-box function optimization performance on diverse domains.
Two methods improve 10-K item segmentation using large language models.
problem Challenges in extracting specific items from 10-K reports due to variations in document formats and item presentation.
method Two advanced item segmentation methods: GPT4ItemSeg and BERT4ItemSeg.
result BERT4ItemSeg achieves a macro-F1 of 0.9825, surpassing other methods.
GP-TS optimizes TLM pre-training hyperparameters efficiently.
problem Resource inefficiency in TLM pre-training.
method Bayesian optimization with Thompson sampling and Gaussian process.
result GP-TS achieves lower MLM loss in fewer epochs.
We introduce a new RL problem where the agent is required to generalize to a previously-unseen environment characterized by a subtask graph which describes a set of subtasks and their dependencies. Unlike existing hierarchical multitask RL approaches that explicitly describe what the agent should do at a high level, ou…
SPF uses a hierarchical approach to efficiently emulate climate changes.
problem Slow and unstable climate emulation for long horizons.
method Spatiotemporal Pyramid Flows (SPF) model data hierarchically across spatial and temporal scales.
result SPF outperforms flow matching baselines and pre-trained models on ClimateBench.
Self-training outperforms pre-training on COCO object detection and segmentation datasets.
problem The effectiveness of pre-training in improving object detection and segmentation models is limited.
method Investigated self-training as an alternative method to utilize additional data.
result Self-training consistently improves model performance across various dataset sizes and data augmentation levels.
He et al. (2018) have called into question the utility of pre-training by showing that training from scratch can often yield similar performance to pre-training. We show that although pre-training may not improve performance on traditional classification metrics, it improves model robustness and uncertainty estimates. …
RoBERTa outperforms other pre-trained models in NER tasks.
problem Improving Named Entity Recognition (NER) performance.
method Fine-tuning four pre-trained models (BERT, ERNIE, ERNIE2.0-tiny, RoBERTa) on NER task.
result RoBERTa achieved state-of-the-art results on MSRA-2006 dataset.
New framework explains how larger pre-trained models reduce downstream learning sample complexity.
problem Understanding why larger pre-trained models reduce sample complexity in downstream tasks.
method Introducing a novel framework called Caulking inspired by PEFT methods.
result Improved pre-trained models provably decrease downstream task sample complexity.
Pre-training on different modalities improves Transformer performance in offline reinforcement learning.
problem Improving Transformer-based models for offline reinforcement learning tasks.
method Empirical investigation of pre-training on language and vision data, analysis of internal representations, and fine-tuning with no context.
result Pre-trained Transformers with language data can utilize context-like information to solve reinforcement learning tasks efficiently.
The study finds a theoretical bound for pre-training iterations needed for pruning to yield good subnetwork performance.
problem Discovering efficient subnetworks within pre-trained dense networks.
method Mathematical analysis of a two-layer, fully-connected network, validating with a multi-layer perceptron trained on MNIST.
result A logarithmically dependent threshold on dataset size for successful pruning.
metabeta uses neural networks to speed up Bayesian mixed-effects regression.
problem Bayesian mixed-effects regression is computationally expensive.
method metabeta is a neural network model that pre-trains to estimate posterior distributions.
result metabeta achieves comparable performance to MCMC at a fraction of the time.
Study identifies COVID-19 pneumonia from chest X-rays.
problem Identifying COVID-19 pneumonia from other types and healthy lungs using CXR images.
method Proposed a multi-class and hierarchical classification schema using CXR images, texture descriptors, and a pre-trained CNN model. Employed resampling algorithms and early/late fusion techniques.
result Achieved macro-avg F1-Score of 0.65 and F1-Score of 0.89 for COVID-19 identification in hierarchical classification scenario.
Unsupervised pre-training improves model generalization, but lacks theoretical understanding.
problem Lack of theoretical understanding of unsupervised pre-training's impact on model generalization.
method Introduces a novel theoretical framework to analyze and enhance generalization.
result Enhances understanding of unsupervised pre-training and fine-tuning, proposing a new regularization method.
New strategy improves GNN pre-training for graph datasets.
problem Effective pre-training for graph neural networks on graph datasets.
method Expressive pre-training of GNNs at both node and graph levels.
result Significant improvement in generalization across downstream tasks.
Paper shows pre-training and transfer learning reduce sample complexity for neural networks.
problem Training high-dimensional supervised learning with limited labeled data.
method Study of single-layer neural networks via online stochastic gradient descent, considering concept shift.
result Pre-training and transfer learning reduce sample complexity by polynomial factors under general assumptions.
Fine-tuning with pre-training data improves performance.
problem Limited training data for tasks.
method Theoretical analysis of excess risk bound and selection of pre-training data subset.
result Improvement in generalization performance with pre-training data.
This paper connects masked pre-training to Bayesian model selection.
problem Understanding the success of masked pre-training and its generalization.
method The paper shows masked pre-training corresponds to maximizing the marginal likelihood.
result Masked pre-training with a suitable scoring function maximizes the marginal likelihood.
New method improves ensemble quality by exploring the pre-train basin more effectively.
problem Limited diversity in ensembles trained from a single pre-trained checkpoint.
method Proposed StarSSE modification of Snapshot Ensembles for transfer learning.
result Stronger ensembles and uniform model soups achieved.
PLUS pre-trains protein sequences with structural info, improving performance.
problem Lack of labeled protein sequences for training models.
method PLUS combines masked language modeling with same-family prediction for pre-training.
result PLUS-RNN outperforms other models in protein biology tasks.
New attack reveals memorization patterns in pre-trained LLMs.
problem Determining if a data point was part of a pre-trained LLM's training set.
method Adapts MIA statistical tests to LLM's perplexity dynamics of subsequences.
result Significantly outperforms prior approaches in membership inference attacks.
This work studies the impact of intra-/inter-class diversity on pre-training datasets and finds a balance for optimal performance.
problem The impact of intra-/inter-class diversity on supervised pre-training datasets and their effect on downstream tasks.
method Empirical study and theoretical analysis of the relationship between diversity types and downstream performance.
result The optimal class-to-sample ratio is invariant to the size of the pre-training dataset and can be predicted.
Ensemble pre-trained models for low data transfer learning.
problem Training good models from scratch in low data regime.
method Fine-tune nearest-neighbour ranked pre-trained models to create ensembles.
result Achieves state-of-the-art performance with lower inference budget.
Pre-trains GNN for generic graph features, reducing data need.
problem Lack of labeled data and expressive features for GNN models.
method Synthetic graph tasks: link reconstruction, centrality ranking, cluster preservation.
result Pre-trained GNN achieves high performance with less labeled data.
The paper develops an asymptotic theory of self-supervised pre-training.
problem Sharpness of current rates in self-supervised pre-training and their accuracy.
method Two-stage M-estimation and tools from Riemannian geometry.
result Characterization of the limiting distribution of the downstream test risk.
New analysis shows diverse classes in pre-training boost NLP performance.
problem Improving sample efficiency in downstream NLP tasks.
method Proved that diverse classes in pre-training lead to better performance, using a large last linear layer singular value.
result Transfer learning excess risk improves with large ildeν and $O\left(\frac{1}{ ildeν \sqrt{n}}
ight)$ rate. Pre-training is crucial for learning deep neural networks. Most of existing pre-training methods train simple models (e.g., restricted Boltzmann machines) and then stack them layer by layer to form the deep structure. This layer-wise pre-training has found strong theoretical foundation and broad empirical support. Howe…
Personalizes pre-trained models for nonparametric regression with limited data.
problem Improving data efficiency in nonparametric regression with few samples.
method Develops a theoretical framework and algorithms for few-shot personalization of black-box models.
result Achieves minimax optimal rate for personalization in nonparametric regression.
Behavior Transfer improves reinforcement learning by leveraging pre-trained policies.
problem Efficient transfer of knowledge in reinforcement learning.
method Behavior Transfer (BT) that uses pre-trained policies for exploration.
result BT combined with pre-training leads to better solutions than without pre-training.
Adaptive kernel density estimation improves accuracy in high dimensions.
problem Challenges in high-dimensional density estimation with traditional methods.
method Pre-training a neural network to recommend location-adaptive kernels.
result Effective density estimation in high dimensions with improved accuracy.
This paper ranks pre-trained DNNs using a novel SI measure.
problem Optimizing pre-trained DNN selection for transfer learning.
method Automated ranking via Separation Index (SI) on target datasets.
result Ranked pre-trained DNNs improve classification performance.
Study builds a Japanese financial-specific LLM through continual pre-training.
problem Lack of domain-specific Japanese financial LLMs.
method Continual pre-training on Japanese financial-focused datasets using a base Japanese LLM.
result Tuned model outperforms original model on Japanese financial benchmarks.
Improved tabular models learn better from real-world data.
problem Tabular models perform poorly on real-world datasets when trained only on synthetic data.
method Continued pre-training on a curated set of real-world datasets.
result Real-TabPFN achieves superior predictive accuracy on 29 datasets.
GCC pre-trains graph neural networks to transfer across diverse datasets.
problem Non-transferable graph models trained for specific datasets.
method Graph Contrastive Coding (GCC) framework using contrastive learning.
result GCC pre-trained models achieve competitive performance across multiple datasets.
Downscaled models outperform larger ones on GLUE tasks.
problem Difficulty in attributing performance changes to specific factors in large language models.
method Pre-trained down-scaled versions of Transformer-based architectures on a common corpus, benchmarked on GLUE tasks.
result MLM + NSP (BERT-style) consistently outperforms other objectives.
Improved LLM pre-training performance through better weight and variance control.
problem Improper weight and variance control in LLM pre-training affects downstream task performance.
method Introduced Layer Index Rescaling (LIR) and Target Variance Rescaling (TVR) techniques.
result Substantial improvements in downstream task performance (up to 4.6%) and reduced extreme activation values.