Mask-reconstruction pretraining helps in downstream tasks by capturing more semantic features.
problem How mask-reconstruction pretraining helps in downstream tasks and why it surpasses supervised learning.
method Theoretical analysis and experimental validation of mask-reconstruction pretraining (MRP) on auto-encoders.
result MRP provably captures more semantic features than supervised learning, leading to better performance in downstream tasks.
This work studies the impact of intra-/inter-class diversity on pre-training datasets and finds a balance for optimal performance.
problem The impact of intra-/inter-class diversity on supervised pre-training datasets and their effect on downstream tasks.
method Empirical study and theoretical analysis of the relationship between diversity types and downstream performance.
result The optimal class-to-sample ratio is invariant to the size of the pre-training dataset and can be predicted.
This study examines how the size and alignment of pretraining data affect the performance of large language models on downstream tasks.
problem Understanding how the size and alignment of pretraining data impact the performance of large language models on downstream tasks.
method Investigated the scaling behavior of large language models in a transfer learning setting, focusing on machine translation tasks.
result The size of the finetuning dataset and the distribution alignment between pretraining and downstream data significantly influence the scaling behavior of downstream performance.
Self-supervised metric learning boosts downstream tasks in multi-view data.
problem Improving distance-based downstream tasks without labeled data.
method Developed a statistical framework to study self-supervised metric learning in multi-view data.
result Self-supervised metric learning improves target distances for various downstream tasks.
The paper predicts loss scaling across different datasets and compute scales.
problem Predicting loss scaling across different datasets and compute scales.
method Derive shifted power law relationships between train and test losses.
result Shifted power law relationships hold for various datasets and tasks, improving prediction accuracy.
FWC creates fair synthetic samples for machine learning tasks.
problem Addressing biases in machine learning models for fair decision-making.
method FWC uses an efficient majority minimization algorithm to minimize Wasserstein distance while enforcing demographic parity.
result FWC achieves a competitive fairness-utility tradeoff and reduces biases in predictions from large language models.
Theoretical analysis shows pretext-based self-supervised learning can be boosted by downstream data under certain conditions.
problem Theoretical analysis of pretext-based self-supervised learning and downstream data refinement.
method Theoretical analysis and experiments on synthetic and real-world datasets.
result Theoretical lower bounds and experiments show that downstream data refinement can boost or hurt performance depending on conditions.
A comprehensive benchmark of 15 scRNA-seq imputation methods across various datasets and analyses.
problem Imputation of single-cell RNA sequencing data to recover latent transcriptional signals.
method Evaluation of 15 imputation methods across 30 datasets and 6 downstream analyses.
result Traditional methods generally outperform DL-based methods in scRNA-seq data analysis.
Boxhead dataset tests autoencoder disentanglement in hierarchical data.
problem Evaluate disentanglement in hierarchical data.
method Introduced Boxhead dataset with hierarchically structured factors, evaluated autoencoder models.
result Hierarchical models outperform single-layer VAEs in disentangling factors.
This work optimizes alignment and uniformity of features on a hypersphere for better downstream performance.
problem Improving the performance of contrastive representation learning.
method Identifying and optimizing alignment and uniformity of features on a hypersphere.
result Directly optimizing alignment and uniformity leads to comparable or better performance than contrastive learning.
CIT and CIF improve feature selection for downstream prediction.
problem Feature selection bias in machine learning models.
method Conditional inference trees and forests with Bonferroni correction.
result CIF ranks top 3 among 18 regression methods and top 4 among 17 classification methods.
Improved tabular models learn better from real-world data.
problem Tabular models perform poorly on real-world datasets when trained only on synthetic data.
method Continued pre-training on a curated set of real-world datasets.
result Real-TabPFN achieves superior predictive accuracy on 29 datasets.
Enhances performance on downstream tasks using multi-domain data.
problem Improving performance on tasks with limited downstream data.
method Deep transfer learning framework leveraging shared and domain-specific features.
result Significantly improves convergence rate for learning Lipschitz functions.
Paper designs a lossy compression method for lossless prediction.
problem Ensuring high performance on predictive tasks with minimal data.
method Characterizes bit-rate requirements for invariant transformations, designs unsupervised objectives for neural compressors.
result Achieves substantial rate savings on ImageNet compared to JPEG without compromising classification performance.
New framework analyzes why more negative samples improve self-supervised learning performance.
problem Inconsistency between theoretical degradation and empirical improvement of downstream supervised tasks with more negative samples.
method Coupon collector's problem framework to analyze self-supervised representation learning with more negative samples.
result Bound can implicitly incorporate supervised loss in self-supervised loss by increasing negative samples.
New measure assesses time series pre-training data quality without labels.
problem Challenges in collecting diverse pre-training datasets for time series classification.
method Contrastive-learning-based foundation model and contrastive accuracy measure.
result Contrastive accuracy correlates with model performance on downstream tasks.
New method controls bias in training data for fair outcomes.
problem Ensuring equal treatment between different groups in machine learning.
method Contrastive information estimation to control mutual information between representations and protected attributes.
result Our method provides strong theoretical guarantees on the parity of any downstream algorithm.
Framework fine-tunes foundation models with semi-supervised learning for downstream tasks and latent spaces.
problem Training foundation models with limited labelled data.
method Mutual information decomposition for downstream and latent spaces, semi-supervised fine-tuning.
result Significant improvements in classification tasks under low-labelled conditions.
Contrastive learning performance doesn't degrade with more negative samples.
problem Theoretical and empirical evidence of negative samples hurting performance in contrastive learning.
method Simple theoretical setting and empirical support on CIFAR-10 and CIFAR-100 datasets.
result Contrastive learning performance does not degrade with the number of negative samples.
Object-centric learning improves generalization and robustness in multi-object scenes.
problem Improving generalization and robustness in neural networks for scenes with multiple objects.
method Training state-of-the-art unsupervised models on multi-object datasets and evaluating segmentation metrics and downstream tasks.
result Object-centric representations are useful for downstream tasks and generally robust to most distribution shifts affecting objects, but less so for less structured shifts.
SMAI framework tests and integrates single-cell data alignability.
problem Lack of a rigorous statistical test for alignability and distortion during alignment.
method Spectral manifold alignment and inference (SMAI) framework.
result SMAI outperforms existing methods in alignability testing and integration.
HEAR benchmark evaluates audio representations for diverse tasks.
problem Developing a general-purpose audio representation for various tasks.
method Evaluated 29 models across 19 tasks using 16 datasets.
result No single audio representation performs holistically.
Generative Adversarial Networks (GANs) have made releasing of synthetic images a viable approach to share data without releasing the original dataset. It has been shown that such synthetic data can be used for a variety of downstream tasks such as training classifiers that would otherwise require the original dataset t…
Paper proposes a framework to detect distribution shifts using embedding space geometry.
problem Detecting distribution shifts in candidate datasets to improve model generalizability.
method Non-parametric framework using embedding space geometry for two tests: robustness boundary and in-distribution/out-of-distribution classification.
result Both tests successfully detect distribution shifts in various scenarios for both synthetic and real-world datasets.
New method optimizes experimental design for specific applications.
problem Inability to adapt causal inference methods to specific downstream applications.
method Task-specific experimental design and sampling strategies.
result Significantly reduces data requirements for achieving RCT performance.
Paper improves molecular property prediction using denoising autoencoders.
problem Limited data for molecular property prediction from 3D structures.
method Pre-training via denoising for learning molecular force fields.
result Achieves new state-of-the-art performance on QM9 dataset.
Paper proposes a unified time series forecasting model with adaptive transfer.
problem General forecasting models for diverse time series data.
method Unified representations through Decomposed Frequency Learning and adaptive domain-specific features via Time Series Register.
result State-of-the-art forecasting performance on seven real-world benchmarks.
Study improves healthcare time series imputation by considering structured missingness.
problem Structured missingness in clinical data impacts time series imputation models.
method Analysis of different masking strategies on imputation methods using PhysioNet Challenge 2012 dataset.
result Masking choices significantly affect imputation accuracy and clinical prediction.
This paper improves cross-domain learning using random forests for manifold alignment.
problem Improving cross-domain learning and feature integration.
method Semi-supervised manifold alignment using random forest proximities.
result Random forest proximities enhance downstream classification accuracy.
Reweighting training data to better represent new tasks.
problem Deploying machine learning models to new tasks is challenging due to training data distribution.
method Formulate an exponential tilt distribution shift model and learn train data importance weights to minimize KL divergence.
result The learned train data weights improve target performance evaluation, fine-tuning, and model selection.
BDC compresses both sample size and dimensionality of large datasets.
problem Large datasets in both sample size and dimensionality.
method Two-stage framework using Decoded MMD, Reconstruction MMD, and Encoded MMD.
result BDC achieves comparable or superior performance with lower cost and higher compression rates.
Fine-tunes GNNs by preserving generative patterns to improve transferability.
problem Vanilla fine-tuning fails due to structural divergence between pre-training and downstream graphs.
method G-Tuning, which reconstructs the generative patterns of the downstream graph using graphon bases.
result G-Tuning achieves an average improvement of 0.5% and 2.6% on in-domain and out-of-domain transfer learning experiments.
Study shows uncertainty of deep learning models can be measured from their embeddings.
problem Uncertainty in contrastive learning models for critical applications.
method Estimating the distribution of training data in embedding space and accounting for local consistency.
result Uncertainty of an embedding vector correlates strongly with downstream accuracy.
Geospatial ML models need special evaluation methods due to their unique challenges.
problem Evaluating geospatial machine learning models is challenging due to their specific characteristics.
method Delineated unique challenges and proposed concrete takeaways for improving geospatial model evaluations.
result Concrete takeaways for improving evaluations of geospatial model performance.
Contrastive learning harms minority group representations, affecting downstream tasks.
problem Representation harm in contrastive learning, especially affecting minority groups.
method Causal mediation analysis and stochastic block model explanation.
result Representation harm in contrastive learning is partly responsible for allocation harm in downstream tasks.
A new robust scaling approach improves downstream metabolomics analysis.
problem Challenges in choosing scaling techniques for metabolomics data.
method Introduces a weighted scaling approach robust to outliers.
result The proposed method outperforms traditional scaling techniques in both outlier-free and outlier-present datasets.
Adversarially robust models transfer better than standard models in image classification.
problem Improving transfer learning performance in image classification.
method Focused on adversarially robust ImageNet classifiers, compared to standard models.
result Adversarially robust models yield improved accuracy on downstream classification tasks.
A new metric FRD improves comparing medical images.
problem Comparing medical images for distribution or domain differences.
method Developed a new metric FRD using standardized radiomic features.
result FRD outperforms other metrics in various medical imaging applications.
MCRAGE generates synthetic data to balance healthcare datasets.
problem Imbalanced datasets in healthcare lead to biased model performance for minority groups.
method Generative modeling to create synthetic data for underrepresented classes.
result MCRAGE improves model performance on minority groups.
A central goal of unsupervised learning is to acquire representations from unlabeled data or experience that can be used for more effective learning of downstream tasks from modest amounts of labeled data. Many prior unsupervised learning works aim to do so by developing proxy objectives based on reconstruction, disent…
Many applications of machine learning require a model to make accurate pre-dictions on test examples that are distributionally different from training ones, while task-specific labels are scarce during training. An effective approach to this challenge is to pre-train a model on related tasks where data is abundant, and…
URL benchmark evaluates uncertainty quantification in pretrained models.
problem Need for reliable uncertainty estimates in transferable pretrained models.
method Proposes URL benchmark to measure transferability of representations and uncertainty estimates.
result Transferable uncertainty quantification remains challenging but not contradictory to traditional goals.
This paper tackles noise in raw datasets to improve representation learning efficiency.
problem Noise in real-world datasets degrades representation learning quality.
method Proposes denoising Cosine-Similarity (dCS) loss to learn robust representations.
result Empirical results show the dCS loss outperforms baseline objective functions.
Paper establishes a comprehensive benchmark for ECG time-series analysis.
problem Incomplete understanding of ECG signal properties and limitations in evaluation metrics.
method Categorization of downstream applications, identification of limitations, introduction of a novel metric, benchmarking of time-series models.
result Validation of the effectiveness of the proposed metric and model architecture.
Improves contrastive learning invariance with novel training objectives and feature averaging.
problem Contrastive learning's implicit invariance is insufficient for robust performance.
method Introduces a novel training objective and feature averaging approach to enforce invariance.
result Improved performance and robustness to transformations on downstream tasks.
PCA-Guided Quantile Sampling preserves data structure in large datasets.
problem Preserving data structure in large-scale subsampling.
method PCA QS uses leading principal components to guide stratified sampling.
result PCA QS maintains statistical and geometric structure without distorting semantics.
Method preserves correlations in synthetic data.
problem Preserving dependence structure of original data.
method Orthogonal Procrustes problem for restoring Pearson correlation.
result Restores Pearson correlation structure while preserving feature distributions and downstream tasks performance.
Bayesian graph contrastive learning improves uncertainty quantification for graph analytics.
problem Uncertainty quantification for node representations in graph contrastive learning.
method Proposes a Bayesian framework to learn stochastic encoders representing nodes as distributions, providing uncertainty estimates.
result Significant improvement in performance on benchmark datasets compared to existing methods.