A new method for experimental design focuses on predicting downstream quantities of interest.
problem Designs that maximize parameter learning may not maximize downstream quantity prediction.
method Likelihood-free goal-oriented optimal experimental design (LF-GO-OED) using ABC density ratio estimation.
result LF-GO-OED maximizes the expected information gain for downstream quantities.
PRAGMA models financial event sequences for various banking tasks.
problem Handling diverse financial data for multiple applications.
method Pre-training a Transformer model on a large banking event corpus with a self-supervised objective.
result PRAGMA achieves superior performance across multiple financial domains from raw event sequences.
Bayesian neural networks (BNNs) allow us to reason about uncertainty in a principled way. Stochastic Gradient Langevin Dynamics (SGLD) enables efficient BNN learning by drawing samples from the BNN posterior using mini-batches. However, SGLD and its extensions require storage of many copies of the model parameters, a p…
This paper generalizes BO uncertainty measures using decision-theoretic entropies.
problem Efficiently inferring optima of expensive black-box functions.
method Introduces a generalized entropy measure from statistical decision theory to optimize Bayesian optimization.
result Demonstrates strong empirical performance across various sequential decision-making tasks.
Theory explains how AI models can predict unseen tasks without labeled data.
problem Understanding how AI models can generalize to unseen tasks.
method Developed a theoretical framework to analyze zero-shot prediction.
result Identified key quantities and independence relationships for generalization.
Linear regression is a classic method of data analysis. In recent years, sketching -- a method of dimension reduction using random sampling, random projections, or both -- has gained popularity as an effective computational approximation when the number of observations greatly exceeds the number of variables. In this p…
Unified framework for SSL methods linking contrastive and non-contrastive approaches.
problem Lack of theoretical foundations and design guidelines for SSL methods.
method Spectral manifold learning framework to unify SSL methods.
result Theoretical bridge between contrastive and non-contrastive methods.
Paper proposes a method to make learned models focus on task-relevant information.
problem Mismatch between model objective and downstream task objective.
method Direct prediction towards task-relevant information, self-supervised.
result Model more effectively models relevant parts of the scene conditioned on the goal.
ABCI infers causal models and queries simultaneously using Bayesian active learning.
problem Inference of causal models and effects in a two-stage process is inefficient and unnatural.
method Active Bayesian Causal Inference (ABCI) using Gaussian processes for sequentially designing experiments.
result ABCI is more data-efficient and accurate in learning causal queries from fewer samples.
This paper tackles inventory control with general arrival dynamics and post-processing, improving profitability.
problem Inventory control with arbitrary arrival dynamics and post-processing constraints.
method Formulated as an exogenous decision process, incorporating deep generative models for arrivals, and applying supervised learning techniques.
result Improves profitability over production baselines and real-world A/B test data.
Unified framework for hybrid learning and optimization via active inference.
problem Sequential decisions in black-box evaluations requiring both task improvement and uncertainty reduction.
method Pragmatic Curiosity (PraC) framework that evaluates queries by balancing information gain and pragmatic value.
result Unified approach reduces decision risk and improves coverage of critical regions without task-specific rules.
We propose a neural hybrid model consisting of a linear model defined on a set of features computed by a deep, invertible transformation (i.e. a normalizing flow). An attractive property of our model is that both p(features), the density of the features, and p(targets | features), the predictive distribution, can be co…
Theoretical analysis shows pretext-based self-supervised learning can be boosted by downstream data under certain conditions.
problem Theoretical analysis of pretext-based self-supervised learning and downstream data refinement.
method Theoretical analysis and experiments on synthetic and real-world datasets.
result Theoretical lower bounds and experiments show that downstream data refinement can boost or hurt performance depending on conditions.
This study examines how the size and alignment of pretraining data affect the performance of large language models on downstream tasks.
problem Understanding how the size and alignment of pretraining data impact the performance of large language models on downstream tasks.
method Investigated the scaling behavior of large language models in a transfer learning setting, focusing on machine translation tasks.
result The size of the finetuning dataset and the distribution alignment between pretraining and downstream data significantly influence the scaling behavior of downstream performance.
Paper defines and solves a problem in representation learning to ensure fairness with high confidence.
problem Learning fair representations with high confidence guarantees for all downstream tasks.
method Formally defines the problem, introduces FRG framework, proves high probability fairness, and demonstrates effectiveness empirically.
result FRG framework provides high-confidence guarantees for limiting unfairness across all downstream models and tasks.
This paper develops a bootstrap method to estimate errors in Random Fourier Features.
problem Inability to estimate the error of Random Fourier Features approximations.
method Develops a bootstrap approach to numerically estimate the errors of RFF approximations.
result Specific, flexible, and adaptive error estimates for RFF approximations.
Study shows how pretraining robustness transfers to downstream tasks.
problem Understanding how robustness is transferred from pretraining to downstream tasks.
method Theoretical analysis and practical validation of robustness constraints.
result Robustness of a linear predictor on downstream tasks can be constrained by the robustness of its underlying representation.
Self-supervised metric learning boosts downstream tasks in multi-view data.
problem Improving distance-based downstream tasks without labeled data.
method Developed a statistical framework to study self-supervised metric learning in multi-view data.
result Self-supervised metric learning improves target distances for various downstream tasks.
Study shows scaling up models doesn't always improve downstream tasks.
problem Understanding why scaling up models doesn't always improve downstream performance.
method Systematic study of 4800 experiments on various models, analyzing performance on 20 downstream tasks.
result Performance on downstream tasks saturates as model size increases, revealing a nonlinear relationship.
We identify spectral conditions for reliable neural probe interpretation.
problem Unreliable performance of linear probes in interpreting neural representations.
method Formalized Spectral Identifiability Principle (SIP) based on eigengap and Fisher error.
result Reliability of neural probes depends on the eigengap relative to Fisher estimation error.
Analysis of pretrained models' effectiveness in downstream tasks.
problem Understanding why pretrained models perform well in NLP tasks.
method Analyzed head and prompt tuning approaches using latent variable models.
result Prompt tuning provides stronger guarantees than head tuning.
This paper characterizes VAE training pathologies and their effects on tasks.
problem Characterizing VAE training pathologies and their impact on downstream tasks.
method Concretely characterizing conditions for VAE training pathologies and their connection to specific downstream tasks.
result Connects VAE training pathologies to specific downstream tasks like learning compressed and disentangled representations, adversarial robustness, and semi-supervised learning.
New framework explains how larger pre-trained models reduce downstream learning sample complexity.
problem Understanding why larger pre-trained models reduce sample complexity in downstream tasks.
method Introducing a novel framework called Caulking inspired by PEFT methods.
result Improved pre-trained models provably decrease downstream task sample complexity.
End-to-end approach for weak supervision improves downstream model performance.
problem Data-labeling bottleneck in machine learning applications.
method Directly learning the downstream model by maximizing its agreement with probabilistic labels generated from weak supervision sources.
result Improved performance over prior work in terms of downstream model performance and robustness.
Kernel embeddings help estimate causal effects from observational data.
problem Estimating causal effects from observational data with confounding variables.
method Kernel embeddings in reproducing kernel Hilbert spaces (RKHS).
result Robust nonparametric framework for causal inference.
UBM transfers bias mitigation from upstream to downstream tasks efficiently.
problem Bias in fine-tuned language models across various tasks.
method Apply bias mitigation to an upstream model, then fine-tune a downstream model on this mitigated model.
result UBM effects transfer to new downstream tasks, creating less biased models.
Mask-reconstruction pretraining helps in downstream tasks by capturing more semantic features.
problem How mask-reconstruction pretraining helps in downstream tasks and why it surpasses supervised learning.
method Theoretical analysis and experimental validation of mask-reconstruction pretraining (MRP) on auto-encoders.
result MRP provably captures more semantic features than supervised learning, leading to better performance in downstream tasks.
Enhances performance on downstream tasks using multi-domain data.
problem Improving performance on tasks with limited downstream data.
method Deep transfer learning framework leveraging shared and domain-specific features.
result Significantly improves convergence rate for learning Lipschitz functions.
Deep neural networks provide meaningful uncertainty estimates for large-scale simulations.
problem Uncertainty estimates for deep neural network predictions from large-scale simulations.
method General variational inference approach to calibrate Bayesian uncertainties.
result Calibrated Bayesian uncertainties preserved physics-correlations in predicted quantities.
This work optimizes alignment and uniformity of features on a hypersphere for better downstream performance.
problem Improving the performance of contrastive representation learning.
method Identifying and optimizing alignment and uniformity of features on a hypersphere.
result Directly optimizing alignment and uniformity leads to comparable or better performance than contrastive learning.
The paper finds a surprising positive correlation between upstreamness and downstreamness in global value chains.
problem The puzzling positive correlation between upstreamness and downstreamness in industries and countries.
method Analysis of a simple model of random Input/Output tables and experiments on empirical data.
result Upstreamness and downstreamness of the same industrial sector/country are positively correlated with a slope close to +1.
We simplify SSL by approximating redundant structural components with low-rank factorization.
problem Improving self-supervised learning performance with limited labeled data.
method Low-rank approximation of structural redundancy, introducing ε_s to measure approximation quality.
result The proposed method enhances SSL performance, as shown by theoretical and experimental validations.
Optimal feature transfer identified through bias-variance analysis.
problem Optimizing feature transfer in transfer learning.
method Simple linear model with fine-grained bias-variance decomposition.
result Optimal pretrained feature transform is naturally sparse.
New framework minimizes model complexity for improved few-shot learning.
problem Empirical benefits of pre-training scale with data size but lack theoretical explanation.
method Complexity Minimization framework for meta-representation learning.
result Theoretical analysis shows error rate improves with more meta-training data.
Optimizes ad pruning in sponsored search systems using reinforcement learning.
problem How to efficiently select top K ads from N candidates to maximize revenue.
method Model-free reinforcement learning approach considering downstream as a black-box environment.
result Remarkable improvements in revenue achieved through reinforcement learning.
Study finds conserved quantities for two types of curves on conformal sphere.
problem Identifying conserved quantities for specific types of curves on a conformal sphere.
method Used parallel tractor and Lagrangian formalism to compute conserved quantities.
result Found relation between conserved quantities of two curve types.
New method predicts wind farm power and wakes using weather patterns.
problem Inefficient and computationally intensive wind energy resource assessment.
method Unsupervised clustering of ERA5 data on wind velocity, WRF simulations at cluster centers, and post-processing.
result Accurate long-term predictions of power and wakes with reduced computational time.
Ecker's and Huisken's quantities agree for ancient mean curvature flows.
problem Understanding the finiteness of integral quantities for ancient mean curvature flows.
method Comparison of Ecker's and Huisken's integral quantities.
result Finiteness of Ecker's integral quantity implies finiteness of entropy at infinity.
FWC creates fair synthetic samples for machine learning tasks.
problem Addressing biases in machine learning models for fair decision-making.
method FWC uses an efficient majority minimization algorithm to minimize Wasserstein distance while enforcing demographic parity.
result FWC achieves a competitive fairness-utility tradeoff and reduces biases in predictions from large language models.
New insights into contrastive learning reveal how projectors affect downstream performance.
problem Understanding how projectors in contrastive learning impact downstream linear classification accuracy.
method Identified and modeled two effects: expansion and shrinkage induced by contrastive loss.
result Linear projectors operating in the shrinkage regime hinder downstream classification accuracy.
This work proposes a new pre-processing method for supervised learning to improve fairness without sacrificing utility.
problem Improving fairness in supervised learning without compromising model performance.
method Task-tailored pre-processing approach that balances fairness and utility.
result The proposed method preserves consistent trade-offs among multiple downstream models and improves fairness in computer vision tasks.
The paper explores how word embeddings affect the stability of downstream NLP models.
problem Small changes in training data can cause significant changes in model predictions.
method Empirical and theoretical analysis of embedding instability, including the introduction of eigenspace instability measure.
result Increasing embedding memory can reduce the disagreement in predictions by 5% to 37%.
Identifies optimal base features for zero-shot adaptation in reinforcement learning.
problem Unclear what constitutes a good set of base features for a wide range of downstream tasks.
method Identifies optimal base features based on downstream performance, without assuming downstream tasks are linear.
result Optimal base features are the same across three task families, differing from Laplacian eigenfunctions.
A new framework designs experiments for better decision-making.
problem Suboptimal experimental designs for downstream decision-making.
method Amortized decision-aware Bayesian Experimental Design (BED) with Transformer Neural Decision Process (TNDP).
result TNDP effectively designs experiments and facilitates accurate decision-making.
This work investigates how neural collapse improves transfer learning for large-scale models.
problem Improving transfer learning for large-scale models with limited labeled data.
method Investigates neural collapse and develops a fine-tuning method using skip-connections.
result Feature collapse on downstream data correlates with higher transfer accuracy.
Object-centric learning improves generalization and robustness in multi-object scenes.
problem Improving generalization and robustness in neural networks for scenes with multiple objects.
method Training state-of-the-art unsupervised models on multi-object datasets and evaluating segmentation metrics and downstream tasks.
result Object-centric representations are useful for downstream tasks and generally robust to most distribution shifts affecting objects, but less so for less structured shifts.
New framework analyzes why more negative samples improve self-supervised learning performance.
problem Inconsistency between theoretical degradation and empirical improvement of downstream supervised tasks with more negative samples.
method Coupon collector's problem framework to analyze self-supervised representation learning with more negative samples.
result Bound can implicitly incorporate supervised loss in self-supervised loss by increasing negative samples.
Proposes a new metric to quantify the difference between neural network representations based on downstream task performance.
problem The lack of a consistent metric to measure the difference between neural network representations.
method Introduced the Transferred Discrepancy (TD) metric, which evaluates the difference between representations based on their performance on downstream tasks.
result TD provides fine-grained information for various downstream tasks and can evaluate the effectiveness of different training strategies.