Method diagnoses model performance under distribution shifts.
problem Understanding and improving model performance under distribution shifts.
method DIstribution Shift DEcomposition (DISDE) method to attribute performance drop to distribution shifts.
result Shows how model performance can be improved across different distribution shifts.
This work introduces a method to attribute model performance drops to distribution shifts.
problem Attributing performance drops of machine learning models to distribution shifts.
method Formulated as a cooperative game, value of a set of distributions is defined as the change in model performance when only that set of distributions changes. Importance weighting method for computing the value of an arbitrary set of distributions is derived. Quantifying the contribution of each distribution as its Shapley value.
result Demonstrated the effectiveness of the method on various case studies.
Monotonic relationship found between in-distribution and out-of-distribution performance.
problem Understanding performance of machine learning models under distribution shifts.
method Analyzing ridge-regularized models and linear inverse problems under covariate shift.
result Monotonic relationship between in-distribution and out-of-distribution performance for certain models.
Survey of performative prediction, a machine learning setup causing distribution shifts.
problem Machine learning models causing shifts in the environment they predict.
method Classification of performative prediction settings based on distribution map information.
result Introduction of new solution concepts and theoretical analyses.
A new framework for performative prediction robust to distributional misspecification.
problem Performative prediction models can be influenced by their own predictions, leading to suboptimal outcomes.
method Introduces distributionally robust performative prediction (DRPO) to approximate the true performative optimum (PO) robustly.
result DRPO provides provable guarantees as a robust approximation to the true PO when the nominal distribution map is misspecified.
This research examines how model explanations change under distribution shifts in tabular data.
problem Detecting distribution shifts in tabular data affecting model performance and explanations.
method Investigates the relationship between model performance and explanation characteristics under distribution shifts.
result Explanation shifts are a better indicator for detecting predictive performance changes than traditional distribution shift techniques.
The study improves model performance prediction on unseen distributions.
problem Improving model performance prediction on unseen distributions.
method Connecting domain adaptation and predictive uncertainty techniques, investigating distributional distances and DoC.
result Difference of confidences (DoC) successfully estimates classifier performance change over various distribution shifts.
The paper shows strong correlation between in-distribution and out-of-distribution performance in various machine learning models.
problem Understanding reliability of machine learning systems in unseen environments.
method Empirical analysis of various models and distribution shifts on CIFAR-10, ImageNet, and other datasets.
result Out-of-distribution performance is strongly correlated with in-distribution performance across different models and distribution shifts.
Training on mixed distributions improves test performance even when components are unrelated.
problem Improving test performance with mismatched training and test distributions.
method Analyzing mixture distributions with different training and test proportions.
result Distribution shift can be beneficial, improving test performance even when components are unrelated.
In this work, we perform an exploratory study on synthesizing deep neural networks using biological synaptic strength distributions, and the potential influence of different distributions on modelling performance particularly for the scenario associated with small data sets. Surprisingly, a CNN with convolutional layer…
Random variables of the generalized Pareto distribution, can be transformed to that of the Pareto distribution. Explicit expressions exist for the maximum likelihood estimators of the parameters of the Pareto distribution. The performance of the estimation of the shape parameter of generalized Pareto distributed using …
Partially performative prediction studies how predictive models influence future data.
problem Distribution shift in predictive models due to endogenous and exogenous factors.
method Generalizing performative prediction to capture both endogenous and exogenous sources of distribution shift.
result Developed online analogues of performative stability and optimality for partially performative environments.
In this article, we focus on distributed Apriori-based frequent itemsets mining. We present a new distributed approach which takes into account inherent characteristics of this algorithm. We study the distribution aspect of this algorithm and give a comparison of the proposed approach with a classical Apriori-like dist…
Estimates model performance under distribution shift using domain-invariant predictors.
problem Poor performance of models on test distributions different from training distributions.
method Uses domain-invariant predictors as a proxy for unknown target labels.
result Shows that the complexity of latent representations influences target risk.
New framework models algorithmic decisions affecting data distribution, enabling efficient learning.
problem Algorithmic decisions can alter data distribution, affecting model performance.
method Model performative effects as push-forward measures, estimating gradients under shift operators.
result Prove convexity of performative risk, allowing more accurate models to be harder to classify.
Study on distributed linear regression performance, focusing on generalization error.
problem Performance of distributed learning in large-scale linear regression.
method Statistical learning approach, focusing on generalization error.
result Generalization error of distributed solution can be higher than centralized solution.
DRL agents perform poorly at high decision frequencies, but a new algorithm improves performance.
problem DRL agents struggle at high decision frequencies, leading to poor performance.
method Proved that DRL agents' action-conditioned return distributions collapse to their policy's return distribution as decision frequency increases. Defined superiority as a probabilistic generalization of advantage for high-frequency value-based RL.
result Proper modeling of superiority distribution improves performance of controllers at high decision frequencies.
There is emerging interest in performing regression between distributions. In contrast to prediction on single instances, these machine learning methods can be useful for population-based studies or on problems that are inherently statistical in nature. The recently proposed distribution regression network (DRN) has sh…
Paper analyzes impact of PRM on binary random variables and distribution shifts.
problem Impact of performative risk minimization on binary random variables and distribution shifts.
method Formulated two measures of impact, derived explicit formulas for full information, and provided estimators for partial information.
result PRM can have amplified side effects compared to methods that do not model data shift.
The true distribution parameterizations of commonly used image datasets are inaccessible. Rather than designing metrics for feature spaces with unknown characteristics, we propose to measure GAN performance by evaluating on explicitly parameterized, synthetic data distributions. As a case study, we examine the performa…
The paper analyzes and improves the learning rates of distributed kernel ridge regression.
problem Generalization performance and learning rates of distributed kernel ridge regression.
method The paper derives optimal learning rates for DKRR in expectation and probability, proposes a communication strategy to improve learning performance, and evaluates these through theory and experiments.
result The communication strategy significantly improves the learning performance of DKRR, as demonstrated by both theoretical assessments and numerical experiments.
Chunking is a significant CL problem, accounting for half of performance drop, and current methods don't address it.
problem Chunking of data in continual learning.
method Analyzing and addressing the chunking sub-problem in continual learning.
result Current CL algorithms perform poorly on chunking, only as well as plain SGD training when there is no distribution shift.
The paper explores how machine learning models can be learnable despite label shifts.
problem Learnability of binary classification models in the presence of label shifts.
method Developed a performative empirical risk function that is an unbiased estimate of the true risk on the shifted distribution.
result PAC-learnable hypothesis spaces remain PAC-learnable for performative scenarios.
New distributed clustering algorithms show resilience to initialization issues.
problem Resilience of distributed gradient-based clustering algorithms to center initialization.
method Distributed gradient-based clustering algorithms with novel center initialization.
result The algorithms are more resilient to initialization compared to baseline methods.
Bayesian approach models match and non-match score distributions over continuous covariates.
problem Complex evaluation of model performance over continuous covariates in biometric verification.
method Generative model of score distributions, mixture models, local basis functions, Bayesian inference.
result Accurate and effective method for studying model performance over continuous covariates.
Paper proposes a new method for designing materials using deep learning.
problem Designing high-performance material distributions from given distributions.
method Iterative process of selecting, generating, and merging material distributions using a deep generative model.
result The method improves material performance through iterative refinement.
Develops robust MDPs for unknown disturbances with performance guarantees.
problem Unknown disturbance distribution in MDPs.
method Empirical distribution, sublevel set of distance function, weak convergence, concentration inequality.
result Robust optimal value function converges to true optimal value function with increasing sample sizes.
A new policy for contextual bandits adapts to reward vector shifts.
problem Learning under reward vector shifts with ordered rewards.
method Adaptive-discretization and optimistic elimination policy.
result Established upper bounds on preference-based regret.
Develops a minimax optimal estimator for system stability under distribution shift.
problem Ensuring system reliability under changes in the underlying environment.
method Minimax optimal estimation of stability defined in terms of acceptable performance degradation.
result Characterizes the minimax convergence rate and demonstrates practical utility.
The aim of this paper is to propose distributed strategies for adaptive learning of signals defined over graphs. Assuming the graph signal to be bandlimited, the method enables distributed reconstruction, with guaranteed performance in terms of mean-square error, and tracking from a limited number of sampled observatio…
Paper tackles performative prediction without convexity assumptions.
problem Performative prediction where data distribution changes with model deployment.
method Reparameterization framework to transform non-convex objective into convex one.
result Provably sublinear regret guarantees for learnable model.
Heavy-tailed distributions are widely used in robust mixture modelling due to possessing thick tails. As a computationally tractable subclass of the stable distributions, sub-Gaussian α-stable distribution received much interest in the literature. Here, we introduce a type of expectation maximization algorithm that e…
PerfGD solves model-induced data shifts by finding optimal points.
problem Model-induced data shifts where model choice changes data distribution.
method Performative Gradient Descent (PerfGD) which explicitly captures model-data interactions.
result PerfGD provably converges to performatively optimal point.
A common goal in statistics and machine learning is to learn models that can perform well against distributional shifts, such as latent heterogeneous subpopulations, unknown covariate shifts, or unmodeled temporal effects. We develop and analyze a distributionally robust stochastic optimization (DRO) framework that lea…
Federated learning studies separate client data and distribution gaps.
problem Understanding performance differences in federated learning across different datasets.
method Proposed a framework to disentangle out-of-sample and participation gaps.
result Dataset synthesis strategy is crucial for realistic simulations of federated learning generalization.
This paper extends performative prediction to nonlinear cases.
problem Performative prediction's effectiveness is limited by linear assumptions in real-world applications.
method Formulated a maximum margin approach loss function and extended it to nonlinear spaces using kernel methods.
result Derived conditions for performative stability in both linear and nonlinear cases.
New method learns distribution shifts caused by predictive models in social computing.
problem Learning distribution shifts due to predictive models in social computing.
method Reverse causal model with microfoundation for agents' actions.
result Effective in minimizing performative prediction risk.
Stochastic algo learns from evolving data, achieving optimal performance.
problem Performative prediction and multiplayer extensions.
method Stochastic approximation with decision-dependent distributions.
result Asymptotic normality and optimality of the algorithm's performance.
We investigate and compare the fundamental performance of several distributed learning methods that have been proposed recently. We do this in the context of a distributed version of the classical signal-in-Gaussian-white-noise model, which serves as a benchmark model for studying performance in this setting. The resul…
Algorithm minimizes regret in predictive models influenced by their own predictions.
problem Finding near-optimal models under performativity with unknown shifts.
method Developed an algorithm that uses performative feedback to achieve low regret, scaling only with distribution shift complexity.
result Achieved regret bounds scaling with distribution shift complexity, not reward function complexity.
MLI is an Application Programming Interface designed to address the challenges of building Machine Learn- ing algorithms in a distributed setting based on data-centric computing. Its primary goal is to simplify the development of high-performance, scalable, distributed algorithms. Our initial results show that, relativ…
Paper tackles robust federated learning for affine distribution shifts.
problem Statistical heterogeneity and distribution shifts degrade model performance in federated learning.
method Develops a robust federated learning algorithm (FLRA) for affine distribution shifts.
result FLRA achieves significant performance gains against affine distribution shifts.
Unified framework for distribution shift estimation, explanation, and improvement.
problem Estimating, explaining, and improving model performance on target domains with distribution shift.
method Entropic Projection Alignment (EPA) aligns source and target distributions by matching moments and minimizing KL divergence.
result EPA consistently outperforms state-of-the-art baselines while offering computational efficiency.
Study addresses RTB model performance drops due to distribution shifts.
problem Distribution shifts between training and target environments in RTB markets.
method Applies Exponential Tilt Reweighting Alignment (ExTRA) algorithm to estimate and correct model weights.
result Demonstrates improved RTB model performance using ExTRA algorithm.
A new framework evaluates model performance on single input points, revealing insights into data and model structure.
problem Traditional evaluation methods in machine learning are insufficient for understanding model performance and data structure.
method Developed a pointwise framework to measure model performance on individual input points, analyzing the relationship between pointwise and average performance.
result Profiles of data points reveal different types of correlations between pointwise and average performance, challenging existing models of learning.
COCOA can learn new tasks without forgetting previously learned ones in a distributed setting.
problem Continual learning with tasks arriving sequentially.
method Distributed optimization with COCOA algorithm.
result COCOA can perform continual learning without forgetting.
Improves regression models' performance on covariate shift.
problem Out-of-distribution generalization for regression.
method Spectrally adapting the weights of a pre-trained neural regression model.
result Spectral adaptation improves out-of-distribution performance.
When the performance of a machine learning model varies over groups defined by sensitive attributes (e.g., gender or ethnicity), the performance disparity can be expressed in terms of the probability distributions of the input and output variables over each group. In this paper, we exploit this fact to reduce the dispa…