AI bias arises from human-defined goals, not algorithmic flaws.
problem AI bias due to human-defined goals in LLMs.
method Purpose-conditioned cognition and revealing downstream use of LLM outputs.
result AI bias can be reduced by purpose-aware prompting but not fully by regularization.
HEAR benchmark evaluates audio representations for diverse tasks.
problem Developing a general-purpose audio representation for various tasks.
method Evaluated 29 models across 19 tasks using 16 datasets.
result No single audio representation performs holistically.
PRAGMA models financial event sequences for various banking tasks.
problem Handling diverse financial data for multiple applications.
method Pre-training a Transformer model on a large banking event corpus with a self-supervised objective.
result PRAGMA achieves superior performance across multiple financial domains from raw event sequences.
Kernel-spectral embedding learns low-dim. structures from noisy data.
problem Learning low-dimensional nonlinear structures from high-dimensional noisy data.
method Adaptive bandwidth spectral embedding using integral operators.
result Convergence to noiseless embeddings and eigenfunctions of integral operators.
New method selects relevant dimensions for better prediction in mixtures.
problem Learning mixtures with limited components for prediction tasks.
method Prediction-focused modeling for mixtures.
result Improves prediction performance compared to non-focused models.
Large, pre-trained generative models have been increasingly popular and useful to both the research and wider communities. Specifically, BigGANs a class-conditional Generative Adversarial Networks trained on ImageNet---achieved excellent, state-of-the-art capability in generating realistic photos. However, fine-tuning …
Paper presents AETN for efficient user modeling from mobile app usage.
problem Efficient user modeling from mobile app usage with reduced manual effort.
method AutoEncoder-coupled Transformer Network (AETN).
result AETN achieves effective user embeddings with reduced manual effort.
Bayesian imputation optimizes bias-variance tradeoff in time-series data.
problem Look-ahead bias in imputation of missing time-series data.
method Wasserstein interpolation for Bayesian posterior consensus distribution.
result Optimal control of look-ahead bias and variance in imputation.
dboost optimizes prediction models for convex cone problems.
problem Optimizing prediction models for decision-making.
method Gradient boosting with implicit differentiation for convex quadratic cone programming.
result dboost reduces out-of-sample decision regret.
We explore self-supervised models that can be potentially deployed on mobile devices to learn general purpose audio representations. Specifically, we propose methods that exploit the temporal context in the spectrogram domain. One method estimates the temporal gap between two short audio segments extracted at random fr…
In recent years, deep learning models have shown great potential in source code modeling and analysis. Generally, deep learning-based approaches are problem-specific and data-hungry. A challenging issue of these approaches is that they require training from starch for a different related problem. In this work, we propo…
New theory explains when pre-trained models can improve downstream tasks.
problem Lack of theoretical understanding of task similarity for transfer learning.
method Feature-centric viewpoint, theoretical results on transferability phase diagram.
result Transfer learning outperforms training from scratch when target task is well represented in feature space.
New framework ensures valid uncertainty estimates for any data stream changes.
problem Challenges of distribution shifts and adversarial actors in real-world data streams.
method Leveraging Blackwell approachability from game theory, the framework guarantees calibrated uncertainties for any compact space.
result Improves calibration and decision-making for energy systems.
Theoretical analysis shows pretext-based self-supervised learning can be boosted by downstream data under certain conditions.
problem Theoretical analysis of pretext-based self-supervised learning and downstream data refinement.
method Theoretical analysis and experiments on synthetic and real-world datasets.
result Theoretical lower bounds and experiments show that downstream data refinement can boost or hurt performance depending on conditions.
This study examines how the size and alignment of pretraining data affect the performance of large language models on downstream tasks.
problem Understanding how the size and alignment of pretraining data impact the performance of large language models on downstream tasks.
method Investigated the scaling behavior of large language models in a transfer learning setting, focusing on machine translation tasks.
result The size of the finetuning dataset and the distribution alignment between pretraining and downstream data significantly influence the scaling behavior of downstream performance.
Bayesian imputation optimizes bias-variance trade-off in time-series data.
problem Look-ahead bias in imputation of missing time-series data.
method Bayesian consensus posterior that fuses multiple posteriors to optimize bias and variance trade-off.
result Benefit of imputation for portfolio allocation with missing returns demonstrated.
Paper defines and solves a problem in representation learning to ensure fairness with high confidence.
problem Learning fair representations with high confidence guarantees for all downstream tasks.
method Formally defines the problem, introduces FRG framework, proves high probability fairness, and demonstrates effectiveness empirically.
result FRG framework provides high-confidence guarantees for limiting unfairness across all downstream models and tasks.
Study shows how pretraining robustness transfers to downstream tasks.
problem Understanding how robustness is transferred from pretraining to downstream tasks.
method Theoretical analysis and practical validation of robustness constraints.
result Robustness of a linear predictor on downstream tasks can be constrained by the robustness of its underlying representation.
Self-supervised metric learning boosts downstream tasks in multi-view data.
problem Improving distance-based downstream tasks without labeled data.
method Developed a statistical framework to study self-supervised metric learning in multi-view data.
result Self-supervised metric learning improves target distances for various downstream tasks.
Study shows scaling up models doesn't always improve downstream tasks.
problem Understanding why scaling up models doesn't always improve downstream performance.
method Systematic study of 4800 experiments on various models, analyzing performance on 20 downstream tasks.
result Performance on downstream tasks saturates as model size increases, revealing a nonlinear relationship.
Analysis of pretrained models' effectiveness in downstream tasks.
problem Understanding why pretrained models perform well in NLP tasks.
method Analyzed head and prompt tuning approaches using latent variable models.
result Prompt tuning provides stronger guarantees than head tuning.
New method estimates model parameters from incomplete data.
problem Estimating model parameters from incomplete data.
method Variational Gibbs Inference (VGI)
result Competitive or better performance compared to existing methods.
This paper characterizes VAE training pathologies and their effects on tasks.
problem Characterizing VAE training pathologies and their impact on downstream tasks.
method Concretely characterizing conditions for VAE training pathologies and their connection to specific downstream tasks.
result Connects VAE training pathologies to specific downstream tasks like learning compressed and disentangled representations, adversarial robustness, and semi-supervised learning.
Recent advancements in graph representation learning have led to the emergence of condensed encodings that capture the main properties of a graph. However, even though these abstract representations are powerful for downstream tasks, they are not equally suitable for visualisation purposes. In this work, we merge Mappe…
New framework explains how larger pre-trained models reduce downstream learning sample complexity.
problem Understanding why larger pre-trained models reduce sample complexity in downstream tasks.
method Introducing a novel framework called Caulking inspired by PEFT methods.
result Improved pre-trained models provably decrease downstream task sample complexity.
End-to-end approach for weak supervision improves downstream model performance.
problem Data-labeling bottleneck in machine learning applications.
method Directly learning the downstream model by maximizing its agreement with probabilistic labels generated from weak supervision sources.
result Improved performance over prior work in terms of downstream model performance and robustness.
UBM transfers bias mitigation from upstream to downstream tasks efficiently.
problem Bias in fine-tuned language models across various tasks.
method Apply bias mitigation to an upstream model, then fine-tune a downstream model on this mitigated model.
result UBM effects transfer to new downstream tasks, creating less biased models.
Mask-reconstruction pretraining helps in downstream tasks by capturing more semantic features.
problem How mask-reconstruction pretraining helps in downstream tasks and why it surpasses supervised learning.
method Theoretical analysis and experimental validation of mask-reconstruction pretraining (MRP) on auto-encoders.
result MRP provably captures more semantic features than supervised learning, leading to better performance in downstream tasks.
Enhances performance on downstream tasks using multi-domain data.
problem Improving performance on tasks with limited downstream data.
method Deep transfer learning framework leveraging shared and domain-specific features.
result Significantly improves convergence rate for learning Lipschitz functions.
SOBER framework optimizes Bayesian optimization tasks efficiently.
problem Challenges in parallel Bayesian optimization.
method Probabilistic Lifting with Kernel Quadrature.
result Versatile and flexible batch Bayesian optimization.
This work optimizes alignment and uniformity of features on a hypersphere for better downstream performance.
problem Improving the performance of contrastive representation learning.
method Identifying and optimizing alignment and uniformity of features on a hypersphere.
result Directly optimizing alignment and uniformity leads to comparable or better performance than contrastive learning.
The paper finds a surprising positive correlation between upstreamness and downstreamness in global value chains.
problem The puzzling positive correlation between upstreamness and downstreamness in industries and countries.
method Analysis of a simple model of random Input/Output tables and experiments on empirical data.
result Upstreamness and downstreamness of the same industrial sector/country are positively correlated with a slope close to +1.
The paper highlights how machine learning calibrations can be biased by training data.
problem Machine learning calibrations can be biased by the training data, affecting downstream analyses.
method The paper examines simulation-based and data-based calibrations, highlighting their prior dependence and proposing solutions.
result A recently proposed Gaussian Ansatz approach can avoid some biases in simulation-based calibrations.
Optimal feature transfer identified through bias-variance analysis.
problem Optimizing feature transfer in transfer learning.
method Simple linear model with fine-grained bias-variance decomposition.
result Optimal pretrained feature transform is naturally sparse.
New framework minimizes model complexity for improved few-shot learning.
problem Empirical benefits of pre-training scale with data size but lack theoretical explanation.
method Complexity Minimization framework for meta-representation learning.
result Theoretical analysis shows error rate improves with more meta-training data.
New framework for variational coresets simplifies Bayesian inference for complex models.
problem Efficient Bayesian inference for complex models like neural networks.
method Black-box variational inference for coresets that handle intractable posterior distributions.
result Principled application of variational coresets to Bayesian neural networks.
Optimizes ad pruning in sponsored search systems using reinforcement learning.
problem How to efficiently select top K ads from N candidates to maximize revenue.
method Model-free reinforcement learning approach considering downstream as a black-box environment.
result Remarkable improvements in revenue achieved through reinforcement learning.
New method predicts wind farm power and wakes using weather patterns.
problem Inefficient and computationally intensive wind energy resource assessment.
method Unsupervised clustering of ERA5 data on wind velocity, WRF simulations at cluster centers, and post-processing.
result Accurate long-term predictions of power and wakes with reduced computational time.
FWC creates fair synthetic samples for machine learning tasks.
problem Addressing biases in machine learning models for fair decision-making.
method FWC uses an efficient majority minimization algorithm to minimize Wasserstein distance while enforcing demographic parity.
result FWC achieves a competitive fairness-utility tradeoff and reduces biases in predictions from large language models.
New insights into contrastive learning reveal how projectors affect downstream performance.
problem Understanding how projectors in contrastive learning impact downstream linear classification accuracy.
method Identified and modeled two effects: expansion and shrinkage induced by contrastive loss.
result Linear projectors operating in the shrinkage regime hinder downstream classification accuracy.
This work proposes a new pre-processing method for supervised learning to improve fairness without sacrificing utility.
problem Improving fairness in supervised learning without compromising model performance.
method Task-tailored pre-processing approach that balances fairness and utility.
result The proposed method preserves consistent trade-offs among multiple downstream models and improves fairness in computer vision tasks.
Identifies optimal base features for zero-shot adaptation in reinforcement learning.
problem Unclear what constitutes a good set of base features for a wide range of downstream tasks.
method Identifies optimal base features based on downstream performance, without assuming downstream tasks are linear.
result Optimal base features are the same across three task families, differing from Laplacian eigenfunctions.
A new framework designs experiments for better decision-making.
problem Suboptimal experimental designs for downstream decision-making.
method Amortized decision-aware Bayesian Experimental Design (BED) with Transformer Neural Decision Process (TNDP).
result TNDP effectively designs experiments and facilitates accurate decision-making.
This work investigates how neural collapse improves transfer learning for large-scale models.
problem Improving transfer learning for large-scale models with limited labeled data.
method Investigates neural collapse and develops a fine-tuning method using skip-connections.
result Feature collapse on downstream data correlates with higher transfer accuracy.
Object-centric learning improves generalization and robustness in multi-object scenes.
problem Improving generalization and robustness in neural networks for scenes with multiple objects.
method Training state-of-the-art unsupervised models on multi-object datasets and evaluating segmentation metrics and downstream tasks.
result Object-centric representations are useful for downstream tasks and generally robust to most distribution shifts affecting objects, but less so for less structured shifts.
New framework analyzes why more negative samples improve self-supervised learning performance.
problem Inconsistency between theoretical degradation and empirical improvement of downstream supervised tasks with more negative samples.
method Coupon collector's problem framework to analyze self-supervised representation learning with more negative samples.
result Bound can implicitly incorporate supervised loss in self-supervised loss by increasing negative samples.
Proposes a new metric to quantify the difference between neural network representations based on downstream task performance.
problem The lack of a consistent metric to measure the difference between neural network representations.
method Introduced the Transferred Discrepancy (TD) metric, which evaluates the difference between representations based on their performance on downstream tasks.
result TD provides fine-grained information for various downstream tasks and can evaluate the effectiveness of different training strategies.
Action-BED: Task-Driven Bayesian Experimental Design
problem Bayesian experimental design with doubly intractable objectives
method Formulating BED in terms of expected future loss (EFL) and optimising it with stochastic gradients
result Simplified and task-driven framework for BED