New study finds best language model architecture and pretraining objective for zero-shot tasks.
problem Evaluating which language model architectures and pretraining objectives best enable zero-shot generalization.
method Compared three model architectures and two pretraining objectives across 170 billion tokens, with and without finetuning.
result Causal decoder-only models trained on autoregressive language modeling exhibit strongest zero-shot generalization.
Pretraining models improves text classification accuracy, but diminishing returns are observed with large datasets.
problem Improving text classification accuracy with pretrained models.
method Examined the benefits of pretrained models on text classification tasks with varying amounts of training data.
result As the number of training examples grows into the millions, the accuracy gap between pretrained BERT-based models and vanilla LSTM narrows to within 1%.
Improving in-context learning for latent space Bayesian optimization by adapting pretraining on molecular latent space.
problem Improving in-context learning for latent space Bayesian optimization.
method Adapting pretraining on molecular latent space.
result Achieving strong performance on held-out molecular optimization benchmarks.
Deep Convolutional features extracted from a comprehensive labeled dataset, contain substantial representations which could be effectively used in a new domain. Despite the fact that generic features achieved good results in many visual tasks, fine-tuning is required for pretrained deep CNN models to be more effective …
Multi-role dialogue understanding comprises a wide range of diverse tasks such as question answering, act classification, dialogue summarization etc. While dialogue corpora are abundantly available, labeled data, for specific learning tasks, can be highly scarce and expensive. In this work, we investigate dialogue cont…
TAMO optimizes multiple objectives in-context using transformers.
problem Balancing competing objectives in expensive, black-box problems.
method Fully amortized, transformer-based policy for multi-objective optimization.
result Significant speedup in proposal time with improved Pareto quality.
Unified framework for ICL in causal and masked models.
problem Understanding ICL in masked language models and comparing it to causal models.
method Developed a statistical learning framework representing context by empirical measure and predicting using context and query.
result Upper bounds for masked and autoregressive objectives under Wasserstein-type regularity conditions.
PULSE learns representations from physiological time series.
problem Lack of effective pretraining objectives for physiological time series.
method Proposes a pretraining framework exploiting dynamical systems generative model.
result PULSE learns representations that distinguish semantic classes and improve transfer learning.
Develops methods for training models with few annotations in computer vision tasks.
problem Training models with limited labeled data in computer vision.
method Theoretical, algorithmic, and experimental contributions for Meta-Learning and Semi-Supervised Learning.
result Improves Contrastive Learning and proposes a semi-supervised method for transformer-based object detectors.
PMI-Masking improves MLM pretraining by masking correlated spans efficiently.
problem Uniform token masking leads to inefficient and suboptimal performance in MLMs.
method PMI-Masking uses Pointwise Mutual Information to mask n-grams with high collocation.
result PMI-Masking reaches half the training time and improves performance.
We introduce a pretraining technique called Selfie, which stands for SELFie supervised Image Embedding. Selfie generalizes the concept of masked language modeling of BERT (Devlin et al., 2019) to continuous data, such as images, by making use of the Contrastive Predictive Coding loss (Oord et al., 2018). Given masked-o…
The dissertation establishes a contexture theory to mathematically characterize representation learning.
problem The lack of a scientific understanding of representation learning in foundation models.
method Introduces the contexture theory as a unified framework for analyzing representation learning methods.
result Representation learning is optimal when the association between input and context is neither too strong nor too weak.
A method merges two pretrained diffusion experts to improve image quality and likelihood.
problem Trade-off between image quality and data likelihood in diffusion models.
method Combining two pretrained diffusion experts by switching between them along the denoising trajectory.
result The merged model consistently matches or outperforms its base components, improving or preserving both likelihood and sample quality.
Transformers learn low-dimensional target functions efficiently in-context.
problem Efficiently learning nonlinear target functions in-context using transformers.
method Nonlinear MLP layer in transformers optimized by gradient descent, focusing on single-index target functions.
result Transformers can learn target functions with low-dimensional structures efficiently in-context.
Paper analyzes why deeper layers of ViTs perform worse on out-of-distribution tasks.
problem Performance degradation of intermediate layers in ViTs under distribution shift.
method Extensive linear probing experiments across various benchmarks and fine-grained analysis of transformer modules.
result Probing feedforward network activations yields best performance under significant distribution shift.
AFTER technique improves NLP models by preventing overfitting to task-specific domains.
problem Standard fine-tuning degrades pretraining domain representations.
method Complements task-specific loss with adversarial objective.
result AFTER leads to improved performance on various NLP tasks.
A new approach to protein language models combines latent space prediction with masked language modeling.
problem Improving protein language models by predicting amino acid identities at masked positions.
method A variant of masked language modeling that predicts latent targets only at masked positions, retaining the MLM cross-entropy.
result The new approach outperforms pure masked language modeling on 11 out of 16 downstream tasks.
Task-agnostic data augmentation shows little benefit for pretrained transformers.
problem Evaluating the effectiveness of task-agnostic data augmentation on pretrained transformers.
method Conducted a systematic examination of two data augmentation techniques (Easy Data Augmentation and Back-Translation) across 5 tasks, 6 datasets, and 3 pretrained transformer models.
result Data augmentation techniques previously effective for non-pretrained models fail to consistently improve performance for pretrained transformers, even with limited training data.
SEMASIA provides a large dataset of latent representations for model comparison.
problem Difficulty in comparing semantic structures across different neural network models.
method Collection of latent representations from 1700 pretrained models across various benchmarks.
result Consistent semantic organization across models and datasets.
New framework selects high-quality pretraining data without training LLMs.
problem Slow progress in understanding pretraining data due to costly experiments.
method Statistical framework based on perplexity-benchmark correlations.
result Approach outperforms existing methods on multiple benchmarks.
Funnel-Transformer reduces computation by compressing sequence data.
problem Redundant token-level representations in language processing.
method Gradually compresses sequence of hidden states to a shorter one.
result Funnel-Transformer outperforms standard Transformer with fewer FLOPs.
Paper studies how few pretraining tasks are needed for a linear model to solve new tasks.
problem How many pretraining tasks are needed for a linear model to solve new tasks?
method Pretrained a linear attention model for linear regression with a Gaussian prior.
result Effective pretraining requires a small number of independent tasks, and the model closely matches Bayes optimal.
Fine-grained pretraining improves neural network's ability to learn rare features.
problem Improving generalization in deep learning models.
method Introducing a hierarchical multi-view structure to confine input data distribution.
result Fine-grained pretraining leads to better accuracy on hard downstream test samples.
Method constructs finance LLMs without instruction data using pretraining and model merging.
problem Developing domain-specific LLMs for finance is resource-intensive.
method Continual pretraining on financial data + model merging of instruction-tuned and domain-specific pretrained vectors.
result Successfully constructs instruction-tuned LLMs for finance without additional instruction data.
Mask-reconstruction pretraining helps in downstream tasks by capturing more semantic features.
problem How mask-reconstruction pretraining helps in downstream tasks and why it surpasses supervised learning.
method Theoretical analysis and experimental validation of mask-reconstruction pretraining (MRP) on auto-encoders.
result MRP provably captures more semantic features than supervised learning, leading to better performance in downstream tasks.
Paper analyzes self-supervised learning using causal methods and proposes a new objective.
problem Lack of theoretical understanding of self-supervised learning success.
method Uses a causal framework to enforce invariance constraints on proxy classifiers.
result ReLIC objective improves generalization guarantees and outperforms existing methods.
Paper introduces a multi-stage influence function to track model predictions.
problem Improving natural language processing and computer vision performance.
method Develops a multi-stage influence function to track predictions from finetuned models back to pretraining data.
result Identifies pretraining examples contributing most to finetuning task predictions.
Synthetic continued pretraining enhances model performance with synthetic data.
problem Data inefficiency in pretrained models when adapting to domain-specific documents.
method Synthetic data augmentation using EntiGraph to create a large synthetic corpus.
result Language models can answer questions and follow instructions without access to domain-specific documents.
WeatherFormer learns robust weather features from small datasets.
problem Modeling complex weather dynamics from limited data.
method Pretrained transformer encoder on large satellite dataset, with spatiotemporal encoding.
result State-of-the-art performance in county-level soybean yield prediction and influenza forecasting.
Recurrent Neural Networks (RNNs) can be seriously impacted by the initial parameters assignment, which may result in poor generalization performances on new unseen data. With the objective to tackle this crucial issue, in the context of RNN based classification, we propose a new supervised layer-wise pretraining strate…
Theoretical analysis of data quality and synergies in LLMs.
problem Understanding why different training methods require different amounts of data.
method Theoretical analysis of transformers trained on a weight prediction task for linear regression.
result SFT excels on smaller datasets challenging for the pretrained model, while RL benefits from large, not overly difficult data.
It has been shown recently that deep convolutional generative adversarial networks (GANs) can learn to generate music in the form of piano-rolls, which represent music by binary-valued time-pitch matrices. However, existing models can only generate real-valued piano-rolls and require further post-processing, such as ha…
Denoised smoothing defends pretrained classifiers against adversarial attacks.
problem Adversarial attacks on pretrained classifiers.
method Prepending a denoiser to any off-the-shelf classifier using randomized smoothing.
result Guaranteed ℓp-robustness to adversarial examples without modifying the pretrained classifier. Framework adds invariance to pretrained networks without fine-tuning.
problem Adding invariance to pretrained networks without altering original behavior.
method Post-training augmentation invariance framework with Markov-Wasserstein minimization and Wasserstein correlation maximization losses.
result Adapter networks improve classification accuracy on rotated and noisy images.
Language models help text classification tasks by predicting next words.
problem Lack of theoretical understanding of why language models perform well on downstream tasks.
method Mathematical study of the connection between next word prediction and text classification, formalizing it and quantifying the benefit.
result Language models that are ε-optimal in cross-entropy learn features that can solve classification tasks with linear approximation.
Study shows how pretraining robustness transfers to downstream tasks.
problem Understanding how robustness is transferred from pretraining to downstream tasks.
method Theoretical analysis and practical validation of robustness constraints.
result Robustness of a linear predictor on downstream tasks can be constrained by the robustness of its underlying representation.
This study examines how the size and alignment of pretraining data affect the performance of large language models on downstream tasks.
problem Understanding how the size and alignment of pretraining data impact the performance of large language models on downstream tasks.
method Investigated the scaling behavior of large language models in a transfer learning setting, focusing on machine translation tasks.
result The size of the finetuning dataset and the distribution alignment between pretraining and downstream data significantly influence the scaling behavior of downstream performance.
New research reveals how the pretraining distribution affects in-context learning in large language models.
problem Understanding how the pretraining distribution influences in-context learning in large language models.
method Developed a theoretical framework to characterize the relationship between pretraining distribution properties and in-context learning performance.
result Characterized a fundamental trade-off between robust task selection and generalization in ICL due to the pretraining distribution's statistical properties.
Exact posterior score estimation for solving linear inverse problems
problem Solving linear inverse problems
method Derive the exact posterior score and use it as a denoising training objective
result EPS outperforms training-free and training-based baselines on various metrics
Research improves federated text models for next word prediction.
problem Training models on distributed devices efficiently and effectively.
method Employ transfer learning in federated learning for next word prediction.
result Enhancements to current baselines with pretrained embeddings and whole model pretraining.
Transformers learn to make decisions in new contexts from offline data.
problem Understanding when and how transformers can perform in-context reinforcement learning.
method Theoretical framework analyzing supervised pretraining for ICRL, including algorithm distillation and decision-pretrained transformers.
result Transformers can efficiently approximate optimal reinforcement learning algorithms for various environments.
This paper enhances language models with knowledge awareness.
problem Understanding how much knowledge pretrained language models grasp.
method Inserting explicit knowledge layers into pretraining without changing transformer architecture.
result Significantly more knowledge packed into transformer parameters.
Training of deep models for classification tasks is hindered by local minima problems and vanishing gradients, while unsupervised layer-wise pretraining does not exploit information from class labels. Here, we propose a new regularization technique, called diversifying regularization (DR), which applies a penalty on hi…
Training 3D object detectors for autonomous driving has been limited to small datasets due to the effort required to generate annotations. Reducing both task complexity and the amount of task switching done by annotators is key to reducing the effort and time required to generate 3D bounding box annotations. This paper…
Publicly pretraining models on Web data may undermine differential privacy.
problem The use of large Web-scraped datasets in differential privacy models.
method Critical review of leveraging pretrained models on public datasets for differential privacy.
result Publicizing pretrained models as 'private' could harm trust and generalize poorly.
New algorithm proves convergence for MAP estimation with denoisers.
problem Proving convergence of MAP estimation methods using pretrained denoisers.
method A simple gradient descent algorithm on smoothed proximal objectives.
result Algorithm provably converges to the proximal operator under log-concavity.
Transformers can learn new tasks from diverse pretraining data but struggle with out-of-domain tasks.
problem Transformer models' ability to learn new tasks in-context is limited by their pretraining data coverage.
method Investigation of transformer models trained on (x,f(x)) pairs, comparing in-context learning capabilities across different task families. result Transformers can identify and learn within task families in their pretraining data but fail with out-of-domain tasks.
Enhances anomaly detection in high dimensions with pretrained networks.
problem Difficult to characterize anomaly in high-dimensional data.
method Residual adaptation to adjust pretrained networks for anomaly detection.
result Significantly outperforms existing methods on anomaly detection benchmarks.