Survey of financial foundation models for diverse applications.
problem Challenges in applying general-purpose FMs to financial tasks.
method Review of financial foundation models (FFMs) in three modalities.
result Emergence of FFMs designed specifically for finance.
Revises Bayesian model averaging for foundation models.
problem Ensemble pre-trained and lightly-finetuned foundation models for improved classification performance.
method Introduces trainable linear classifiers and computationally cheaper model averaging scheme (OMA).
result Ensembled models can better predict on various datasets.
Time series foundation models are well-calibrated, improving over baseline models.
problem Calibration of time series foundation models for practical applications.
method Systematic evaluations of five time series foundation models and two baselines, assessing calibration, prediction heads, and long-term forecasting.
result Time series foundation models are consistently better calibrated than baseline models and do not show over- or under-confidence.
Introduces foundation priors for using model-generated data in empirical research.
problem Using model-generated data as real observations in empirical research.
method Introduces foundation priors as an exponential-tilted, generalized Bayesian update of the user's primitive prior.
result Synthetic data reflects both model patterns and user's priors, enabling principled use in empirical work.
Combines foundation models with weak supervision to improve NLP and video tasks.
problem Leveraging weak supervision with foundation models without labeled data.
method Liger, a combination of foundation model embeddings and weak supervision techniques.
result Liger outperforms existing weak supervision methods by 14.1 points on benchmark NLP and video tasks.
Foundation models trained on collider data improve jet generation tasks.
problem Improving foundation models for jet generation tasks.
method Pre-training OmniJet-α model on AspenOpenJets dataset. result Pre-trained model improves performance on jet generation tasks with domain shift.
New measure assesses time series pre-training data quality without labels.
problem Challenges in collecting diverse pre-training datasets for time series classification.
method Contrastive-learning-based foundation model and contrastive accuracy measure.
result Contrastive accuracy correlates with model performance on downstream tasks.
Paper explores foundation models for dynamical systems using synthetic data.
problem Lack of synthetic data for dynamical systems training.
method Pretrained transformer model on synthetic dynamics functions sampled from RKHS.
result Pretrained model generalizes across various dynamical systems in simulations and hardware.
GAMformer bridges tabular models and interpretability, offering a single-pass approach.
problem Lack of interpretability in tabular foundation models like TabPFN.
method In-context learning for GAM shape functions, training on synthetic data.
result GAMformer performs comparably to other leading GAMs across various classification benchmarks.
Paper speeds up large foundation models for time series data.
problem Resource-intensive foundation models limit accessibility.
method Dimensionality reduction techniques, including PCA and neural network adapters.
result Up to 10x speedup and 4.5x more datasets fit on a single GPU.
New method reduces fine-tuning cost for reused models.
problem Repeating fine-tuning costs with outdated foundation models.
method Portable Reward Tuning (PRT) trains a reward model to maximize the same loss function as fine-tuning.
result PRT achieves comparable accuracy to inference-time tuning with less inference cost.
Foundation models outperform supervised methods in time series forecasting across various operational regimes.
problem Lack of domain-specific training and ongoing maintenance in supervised learning for time series forecasting.
method Evaluation of foundation models against standard supervised approaches across four operational regimes: periodic, physically constrained, stochastic, and demand forecasting.
result Foundation models are optimal for cold-start or long-tail scenarios and perform well in domains with transferable periodic structures.
Foundation models improve wage gap decomposition by capturing omitted career history factors.
problem Estimating wage disparities using incomplete career history data.
method Fine-tuning foundation models to mitigate omitted variable bias and estimate wage gaps.
result Foundation models can decompose gender wage gaps more accurately than traditional econometric methods.
Paper proposes verifier engineering for improving foundation models.
problem Challenges in providing effective supervision signals for foundation models.
method Leverages automated verifiers to perform verification tasks and deliver feedback.
result Verifier engineering can enhance foundation models' capabilities.
Foundation models improve time series prediction reliability, especially with limited data.
problem Improving time series prediction reliability with limited data.
method Comparison of Time Series Foundation Models (TSFMs) with traditional methods in conformal prediction.
result TSFMs provide more reliable conformalized prediction intervals and more stable calibration with limited data.
Foundation models improve on econometric benchmarks for forecasting volatility, but vary widely across models.
problem Comparing pretrained time series foundation models to econometric benchmarks for volatility forecasting.
method Systematic comparison of nine zero-shot TSFMs against eight econometric specifications on 50 assets across 3 markets and 3 horizons.
result Tiny Time Mixers (TTM) is the only model that consistently beats the Log-HAR benchmark, but performance varies widely across models.
Foundation AI model outperforms traditional VaR methods in forecasting.
problem Forecasting Value-at-Risk (VaR) for financial returns.
method Time-series foundation AI model, pre-trained on diverse datasets, fine-tuned for specific quantiles.
result Fine-tuned foundation model consistently outperforms traditional methods in actual-over-expected ratios.
Mantis improves time series classification using a transformer model trained on synthetic data.
problem Insufficient application of foundation models to time series classification.
method Pre-trained transformer model on synthetic data, enhanced test-time methodology.
result Mantis achieves state-of-the-art performance across diverse datasets.
Tabular foundation models outperform other methods in conditional density estimation across various datasets.
problem Estimating the full conditional distribution of a response given tabular covariates, especially in settings with heteroscedasticity, multimodality, or asymmetric uncertainty.
method Benchmarked three tabular foundation model variants (TabPFN and TabICL) against six CDE baselines on 39 real-world datasets.
result Tabular foundation models achieve the best CDE loss, log-likelihood, and CRPS across all sample sizes, outperforming other methods.
Paper introduces TS-GPT for engineering time series forecasting.
problem Engineering time series require causal operations, unlike linguistic data.
method Innovations representation theory, Generative Pre-trained Transformer.
result TS-GPT effectively forecasts real-time locational marginal prices.
Study evaluates interpretability of time series foundation models' latent spaces.
problem Improving interpretability of latent spaces in time series models for visual analytics.
method Evaluated MOMENT family of transformer-based models on five datasets, fine-tuning for performance.
result Fine-tuning improved latent space clarity but limited interpretability remained.
Study shows cliff-learning in transfer learning from foundation models.
problem Data-scaling of transfer learning from foundation models in low data regimes.
method Investigation of cliff-learning phenomenon through foundation-model analysis and toy models.
result Cliff-learning reflects compatibility between priors and tasks.
Framework fine-tunes foundation models with semi-supervised learning for downstream tasks and latent spaces.
problem Training foundation models with limited labelled data.
method Mutual information decomposition for downstream and latent spaces, semi-supervised fine-tuning.
result Significant improvements in classification tasks under low-labelled conditions.
GIFT-Eval benchmarks time series forecasting models across diverse datasets.
problem Lack of comprehensive benchmarks for evaluating time series foundation models.
method Developed GIFT-Eval, a benchmark with 23 datasets, 177 million data points, and 144,000 time series.
result Promotes evaluation of foundation models across various domains and frequencies.
VMoER improves uncertainty quantification in MoE layers for scalable foundation models.
problem Uncertainty quantification in large-scale models like MoE layers.
method Structured Bayesian approach with amortized variational inference over routing logits and temperature parameter inference.
result Improves routing stability, reduces calibration error, and increases AUROC by 12%.
Unified approach to learn interpretable concepts from data.
problem Building interpretable machine learning models and highly-performing foundation models.
method Relating causal representation learning and foundation models, defining concepts and proving their recoverability.
result Provable recovery of human-interpretable concepts from diverse data.
EAGLE-Net enhances foundation models by integrating patch-level features for better tissue understanding.
problem Foundation models lack mechanisms for global tissue structure and local context in computational pathology.
method EAGLE-Net combines multi-scale spatial encoding, attention-guided loss functions, and background suppression to aggregate patch-level features into slide-level predictions.
result EAGLE-Net improves classification accuracy and concordance indices across multiple cancer types, producing biologically coherent attention maps.
FEET protocol evaluates foundation models across three scenarios.
problem Lack of standardized evaluation protocols for foundation models.
method Structured evaluation protocol across three use cases: frozen, few-shot, and fine-tuned embeddings.
result Demonstrates comprehensive assessment of foundation models' effectiveness.
Improved forecast accuracy for energy systems through decision-focused fine-tuning.
problem Challenges in integrating forecast values into time series models for diverse and specific instances.
method Decision-focused fine-tuning within time series foundation models for dispatchable feeder optimization.
result Improvement of 9.45% in average total daily costs.
TabDPT combines ICL and self-supervised learning to train tabular models on real data.
problem Slow generalization of tabular foundation models to unseen datasets.
method In-Context Learning (ICL) combined with self-supervised learning, using real data for pre-training.
result TabDPT achieves strong performance on regression and classification benchmarks.
UniShape improves time series classification by selecting relevant subsequences.
problem Classifying time series data requires capturing interpretable shapelets.
method UniShape uses a shape-aware adapter to aggregate multiscale subsequences into class tokens.
result UniShape achieves state-of-the-art classification performance.
CTSyn generates high-quality synthetic tabular data.
problem Challenges in generating high-quality synthetic tabular data.
method Diffusion-based generative foundation model with autoencoder and conditional latent diffusion.
result CTSyn outperforms existing table synthesizers on standard benchmarks.
Foundation models struggle with multi-turn exploration but can learn through regular summaries.
problem Foundation models struggle with multi-turn exploration in dynamic environments.
method Implemented a text-based version of the Alchemy environment to test multi-trial learning. Prompting models to summarize their observations at regular intervals enabled them to improve across trials and adapt to changes.
result Foundation models can improve through regular summaries, enabling multi-trial learning and adaptation.
Foundation models fail to preserve continuous geometry, identified as the Geometric Alignment Tax.
problem Continuous geometry is lost in foundation models due to discrete categorical bottlenecks.
method Controlled ablations on synthetic systems and evaluation of 14 biological models using rate-distortion theory and MINE.
result Replacing cross-entropy with a continuous head reduces geometric distortion by up to 8.5x.
Foundation models improve volatility forecasting in finance.
problem Improving volatility forecasting in financial markets.
method Evaluation of TimesFM model, incremental fine-tuning, comparison with econometric benchmarks.
result Incremental fine-tuning improves forecast accuracy and outperforms traditional models.
Paper proposes a method to use in silico experiments with foundation models to reduce sample size.
problem Costly and uncertain randomized experiments.
method Integrates predictions from multiple foundation models with experimental data.
result Estimator offers substantial precision gains, equivalent to a 20% reduction in sample size.
New graph foundation models respect symmetries for broader applicability.
problem Tailored graph machine learning architectures limit broader applicability.
method Investigates symmetries for label and feature permutations, proving network universal approximator.
result Universal approximator on multisets respecting node and feature permutations.
The paper proposes a new risk model for foundation models in finance.
problem Understanding how foundation models affect trading strategies' risk and return.
method An extension of the CAPM, separating systematic and idiosyncratic risks.
result Monte Carlo dropout measures the epistemic risk of foundation models.
CoLoRA leverages task similarity to boost fine-tuning efficiency.
problem Efficiently fine-tuning large foundation models with scarce labeled data.
method CoLoRA trains a shared adapter for task similarity and personalized adapters for user-specific tasks.
result CoLoRA significantly boosts fine-tuning performance when tasks are similar.
This study benchmarks transcriptomics models for perturbation analysis, finding scVI and PCA superior.
problem Limited evaluation of transcriptomics foundation models for perturbation analysis.
method Developed a novel evaluation framework using diverse public datasets from different sequencing techniques and cell lines.
result scVI and PCA identified as superior models for understanding biological perturbations.
Enhanced TSFMs improve time series forecasting accuracy and reliability.
problem Variance, bias, and uncertainty in TSFMs' predictions on real data.
method Statistical and ensemble techniques including bagging, stacking, residual modeling, and prediction intervals.
result Hybrid models consistently outperform standalone TSFMs across multiple horizons.
Conformal Alignment ensures trustworthy outputs from foundation models.
problem Ensuring outputs from foundation models align with human values in high-stakes tasks.
method A framework that trains an alignment predictor using reference data to select trustworthy outputs.
result Conformal Alignment accurately identifies trustworthy outputs via lightweight training over moderate reference data.
Foundation models alter medical data science workflow, challenging veridical data science principles.
problem Foundation models disrupt traditional data science practices in medicine.
method Critically examined the medical foundation model lifecycle and its deviation from veridical data science principles.
result Foundation models challenge veridical data science principles of predictability, computability, and stability.
CInA method uses attention to improve causal inference.
problem Challenges in causal inference, especially in complex tasks.
method CInA method utilizes self-supervised causal learning with multiple unlabeled datasets and transformer-type architecture.
result CInA effectively generalizes to out-of-distribution datasets and various real-world datasets.
AI model enhances grid monitoring with synchro-waveform tech.
problem Dynamic, stochastic, low-inertia future grids need advanced monitoring.
method AI Foundation Model with synchro-waveform tech.
result Significantly improved fault detection accuracy and speed.
Paper presents a copula-based method to efficiently generate correlated sample paths from multi-step time series models.
problem Generating realistic correlation structures in multi-step forecast sample paths is expensive and time-consuming.
method Copula-based approach to generate correlated sample paths in one forward pass.
result Improved sample path quality and significant speedup over autoregressive sampling.
Single model learns physics from diverse data.
problem Lack of universal physics models for diverse applications.
method General Physics Transformer (GPhyT) trained on diverse physics data.
result Single model achieves superior performance across multiple physics domains.
Framework tackles class imbalance and noisy labels in active learning.
problem Class imbalance and noisy labels in real-world datasets.
method Uses foundation model priors to select informative samples for active learning.
result Substantial annotation savings (over 50%) with preserved performance and robustness.