New methods handle both data and network heterogeneity in federated learning.
problem Challenges in federated learning due to data and network heterogeneity.
method Two novel client selection schemes that minimize theoretical runtime to convergence.
result Our methods are at least competitive to and up to 20 times better than existing baselines.
Energy distance measures feature heterogeneity in federated learning.
problem Heterogeneity across data sources hinders model aggregation in federated learning.
method Introduced Taylor approximations of energy distance for efficient computation.
result Taylor approximations accurately capture feature discrepancies, improving convergence.
SHIFT method optimally estimates heterogeneous discrete distributions with limited communication.
problem Collaborative learning of discrete distributions under heterogeneity and communication constraints.
method Two-stage method: First, users learn a central distribution; then, fine-tune this to estimate individual distributions.
result SHIFT is minimax optimal in the model of heterogeneity and under communication constraints.
Minibatch SGD outperforms Local SGD in heterogeneous distributed learning.
problem Optimizing a combined convex objective with stochastic gradient estimates from different machines.
method Analysis of Minibatch SGD and Local SGD in a heterogeneous distributed setting.
result Minibatch SGD dominates Local SGD in the heterogeneous distributed setting.
CCs learn high-dimensional distributions from heterogeneous data.
problem Learning high-dimensional distributions from heterogeneous data.
method Introducing characteristic circuits (CCs) that learn from data and use spectral domain.
result CCs outperform state-of-the-art density estimators on common benchmark data sets.
Paper shows local SGD outperforms mini-batch SGD under certain conditions.
problem Proving local SGD's superiority in distributed learning with heterogeneous data.
method New lower and upper bounds for local SGD under first-order heterogeneity assumptions.
result Local SGD is min-max optimal under certain conditions, resolving understanding of distributed optimization.
Two tests identify heterogeneous components in distributed learning.
problem Identifying parameter heterogeneity in distributed learning with minimal data transmission.
method Two tests: Wald and Extreme Contrast (ECT).
result ECT avoids bias accumulation and is robust to varying levels of sparsity.
Federated Learning is a distributed learning paradigm with two key challenges that differentiate it from traditional distributed optimization: (1) significant variability in terms of the systems characteristics on each device in the network (systems heterogeneity), and (2) non-identically distributed data across the ne…
Proposes a new method to handle data heterogeneity in causal inference.
problem Challenges of collaborating between different data centers due to heterogeneity.
method Collaborative inverse propensity score weighting estimator to adjust for distribution shift.
result Significant improvements over traditional meta-analysis methods when dealing with increased heterogeneity.
Fog learning distributes ML model training across heterogeneous devices and networks.
problem Challenges with conventional federated learning in heterogeneous networks.
method Intelligent distribution of ML model training across nodes from edge devices to cloud servers.
result Enhanced federated learning with multi-layer hybrid framework considering network, heterogeneity, and proximity.
Study shows data heterogeneity affects distributed learning's generalization error.
problem Effect of data heterogeneity on distributed learning performance.
method Established bounds on generalization error using information-theoretic rate-distortion theory.
result Data heterogeneity improves generalization error for distributed learning.
Distributed learning adapts to diverse devices, improving performance.
problem Training neural networks on devices with varying capabilities and resources.
method Each device trains a customized neural network, sharing parameters with others.
result Achieves higher rewards on more powerful devices without sacrificing weaker ones.
Study robust estimation under varying corruption probabilities in data.
problem Robust estimation in scenarios with heterogeneous corruption rates.
method Developed estimators for mean and regression under various corruption patterns.
result Optimal estimators can discard corrupted samples beyond a specific threshold.
A distributed SGD method for heterogeneous networks with hubs and workers.
problem Learning in heterogeneous multi-level networks with worker heterogeneity and varying communication.
method Multi-Level Local SGD: distributed SGD with hub-and-spoke paradigm and hub averaging.
result The method converges with error dependent on worker heterogeneity, hub network topology, and iterations.
Rescaled ASGD optimizes distributed learning under heterogeneous data.
problem Vanilla ASGD biases towards a frequency-weighted average of local objectives.
method Rescale worker stepsizes by their computation times.
result Rescaled ASGD converges to the correct global objective in fixed-computation model.
This paper evaluates CFL algorithms for handling data heterogeneity in federated learning.
problem Handling data heterogeneity among clients in federated learning.
method Comparative evaluation of two state-of-the-art CFL algorithms with a proposed taxonomy of data heterogeneities.
result Analysis of CFL performance across different heterogeneity scenarios using extrinsic clustering metrics.
Ringleader ASGD optimizes SGD for diverse edge devices with varying data and computation speeds.
problem Scalable distributed optimization with heterogeneous devices and data.
method Ringleader ASGD, an asynchronous SGD algorithm.
result Achieves optimal time complexity under data heterogeneity and arbitrary computation speeds.
New methods improve distributed optimization on non-iid data.
problem Communication bottleneck in distributed machine learning models.
method Two types of distributed gradient compression methods (D-QSGD and D-EF-SGD) analyzed for non-iid data.
result D-EF-SGD performs better than D-QSGD on non-iid data but can still slow down with high data skewness.
This research tackles sample complexity in causal graph recovery with temporal heterogeneity.
problem Recovering a unique causal graph from observational data with temporal heterogeneity.
method Integrates time-series dynamics and multi-environment heterogeneity to constrain the problem, enabling a rigorous analysis of statistical limits.
result Unified necessary identifiability conditions and explicit information-theoretic bounds quantify the sample complexity under different noise distributions.
Estimates population mean from user-level data with privacy, accounting for heterogeneity.
problem Heterogeneous user data with varying numbers of data points and distributions.
method Simple model of heterogeneous user data, differential privacy mechanism for estimation.
result Asymptotic optimality of the proposed estimator and general lower bounds on error.
HeteroFL trains diverse clients with varying capabilities efficiently.
problem Training models on clients with different computational and communication capacities.
method Proposes HeteroFL framework to adaptively distribute subnetworks based on client capabilities.
result Adaptive subnetwork distribution leads to efficient computation and communication.
Proposes MSS to identify causal structure from heterogeneous environments.
problem Distribution shifts between environments violate i.i.d. data assumption.
method Sparse mechanism shift hypothesis, score-based approach.
result Identifies entire causal structure with high probability.
The thesis clarifies when local updates outperform centralized methods in heterogeneous data environments.
problem Understanding when local updates are more effective than centralized or mini-batch methods in distributed optimization.
method Fine-grained consensus-error-based analysis framework, focusing on bounded second-order heterogeneity and third-order smoothness.
result Local updates outperform centralized or mini-batch methods under realistic models of data heterogeneity.
New algorithms tackle statistical heterogeneity in federated learning.
problem Statistical heterogeneity in distributed machine learning models.
method Introduces three novel methods: SuPerFed, AAggFF, and FedEvg.
result Mitigates statistical heterogeneity in federated learning.
Characterizes uncertainty in low-rank matrix completion with noisy data.
problem Uncertainty quantification in low-rank matrix completion with heterogeneous sub-exponential noise.
method Characterizes the distribution of estimated matrix entries under low-rank estimators with heterogeneous sub-exponential noise.
result Explicit formulas for the distribution of estimated matrix entries under Poisson and Binary noise.
A framework for federated learning with heterogeneous data.
problem Federated learning with data from clients using different data representations.
method FLIC framework that maps client data into a common feature space via local embedding functions, learned federally using Wasserstein barycenters and trained locally via distribution alignment.
result FLIC outperforms FL benchmarks with heterogeneous input feature spaces.
The aim of this paper is to propose a heterogeneous agent model of stock markets that develop complicated endogenous price fluctuations. We find occurrences of non-stationary chaos, or speculative bubble, are caused by the heterogeneity of traders' strategies. Furthermore, we show that the distributions of returns gene…
UNTIE learns representations of coupled categorical data.
problem Challenges in learning from unlabeled categorical data with complex couplings.
method UNTIE approach for unsupervised representation learning of heterogeneous couplings.
result UNTIE significantly improves categorical data representations on 25 diverse datasets.
Extends SW and GSW to compare heterogeneous joint distributions.
problem Limited applicability of SW and GSW to heterogeneous joint distributions.
method Introduces HHRT and PGRT to extend SW and GSW.
result H2SW distance for heterogeneous joint distributions.
The goal of this paper is to study organized flocking behavior and systemic risk in heterogeneous mean-field interacting diffusions. We illustrate in a number of case studies the effect of heterogeneity in the behavior of systemic risk in the system, i.e., the risk that several agents default simultaneously as a result…
RelaySum improves decentralized deep learning by uniformly distributing data across workers.
problem Handling data heterogeneity in decentralized deep learning.
method RelaySum uses spanning trees to distribute information exactly uniformly across all workers with finite delays.
result RelaySum is independent of data heterogeneity and scales to many workers, enabling highly accurate decentralized deep learning.
New Random Forest variants estimate heterogeneous treatment effects using Wasserstein distances.
problem Estimating heterogeneous treatment effects in complex situations.
method Proposes natural variants of Random Forests using Wasserstein distances.
result Natural variants of Random Forests are well-suited for estimating conditional distributions.
A new random forest method for multivariate distributions.
problem Estimating complex multivariate distributions with heterogeneity.
method A novel splitting criterion based on MMD for multivariate responses.
result Estimates full conditional distribution for arbitrary targets.
VSAE learns from missing heterogeneous data by modeling latent dependencies.
problem Learning from partially-observed heterogeneous data with missingness.
method Variational selective autoencoder (VSAE) models joint distribution of observed, unobserved, and missing data.
result VSAE improves over state-of-the-art models in data generation and imputation tasks.
CWAN tackles multi-source heterogeneous domain adaptation with conditional weighting.
problem Learning cross-domain samples from multiple heterogeneous domains.
method CWAN uses a feature transformer, label classifier, and domain discriminator to learn from multiple sources.
result CWAN outperforms state-of-the-art methods on four real-world datasets.
New test detects differences in heterogeneous datasets.
problem Detecting differences between two samples with unknown heterogeneity.
method Developed a nonparametric testing procedure that handles latent heterogeneity through a composite null.
result The test accurately detects differences in the presence of unknown heterogeneity.
MindFlayer SGD improves parallel SGD for heterogeneous, random compute times.
problem Minimizing nonconvex functions with heterogeneous, random compute times.
method MindFlayer SGD, designed for stochastic and heterogeneous delays.
result MindFlayer SGD outperforms existing methods in environments with heavy-tailed noise.
The paper introduces heterogeneous manifolds for better graph embeddings.
problem Graph embeddings in Euclidean spaces often fail to capture the curvature of real-world graphs.
method The authors propose heterogeneous rotationally-symmetric manifolds with a radial dimension to account for varying curvature.
result The method improves graph embeddings by better preserving high-order structures and heterogeneous random graphs.
New bucketing scheme improves Byzantine robustness for heterogeneous data.
problem Byzantine attacks on federated learning with heterogeneous data.
method Bucketing scheme to adapt robust algorithms to non-iid data.
result Bucketing scheme ensures convergence against Byzantine attacks.
New robust discriminant analysis for non-Gaussian data.
problem Classical discriminant analysis struggles with non-Gaussian distributions and contaminated datasets.
method Each data point follows its own ES distribution with arbitrary scale, leading to robust classification.
result Maximum-likelihood estimation and classification are simple, fast, and robust.
In regression tasks, aleatoric uncertainty is commonly addressed by considering a parametric distribution of the output variable, which is based on strong assumptions such as symmetry, unimodality or by supposing a restricted shape. These assumptions are too limited in scenarios where complex shapes, strong skews or mu…
This work improves Gaussian process regression for large, non-stationary data.
problem Scalability issues and performance degradation for non-stationary data.
method Combines variational free energy approximations with online expectation propagation and local splitting steps.
result Incremental adaptation to locality, heterogeneity, and non-stationarity in training data.
Proposes EDM algorithm to accelerate model training in distributed networks.
problem Hindered effectiveness of distributed stochastic optimization algorithms due to data heterogeneity and network sparsity.
method Introduces Exact-Diffusion with Momentum (EDM) algorithm, incorporating momentum techniques to mitigate bias and enhance convergence rate.
result EDM algorithm converges sub-linearly to the optimal solution, radius independent of data heterogeneity, for non-convex objective functions.
The paper tackles personalized policy learning from diverse data sources in a federated setting.
problem Learning personalized decision policies from observational bandit feedback across multiple heterogeneous data sources.
method Introduces a novel regret analysis for distinguishing global and local regret, and presents a federated policy learning algorithm using local policies trained with doubly robust offline policy evaluation strategies.
result Establishes finite-sample upper bounds on global and local regret, characterizing them by source heterogeneity and distribution shift.
CausalMix generates synthetic data with causal controls for mixed-type tables.
problem Synthetic data for causal inference with mixed-type and multimodal tabular data.
method CausalMix combines Gaussian latent priors with data-type-specific decoders for control over overlap, confounding, and treatment effect heterogeneity.
result CausalMix achieves state-of-the-art distributional metrics and stable causal control.
Unified analysis of federated learning with compression for various data distributions.
problem Communication overhead in federated learning with unreliable or limited communication.
method Periodic compressed communication and local gradient tracking schemes.
result Sharp convergence rates for various objective functions and data distributions.
TSVQR captures heterogeneous and asymmetric data using quantile regression.
problem Capturing heterogeneous and asymmetric information in modern data.
method Twin Support Vector Quantile Regression (TSVQR) with two nonparallel planes for quantile levels.
result TSVQR outperforms previous methods in capturing and learning from data.
Two novel methods identify influential features in CMABs for better reward distribution.
problem Suboptimal features degrade rewards, interpretability, and efficiency in CMABs.
method Heterogeneous Incremental Effect (HIE) and Heterogeneous Distribution Divergence (HDD) methods.
result Consistent ability to identify influential HTE features, enhancing CMAB performance.