MindFlayer SGD improves parallel SGD for heterogeneous, random compute times.
problem Minimizing nonconvex functions with heterogeneous, random compute times.
method MindFlayer SGD, designed for stochastic and heterogeneous delays.
result MindFlayer SGD outperforms existing methods in environments with heavy-tailed noise.
Ringleader ASGD optimizes SGD for diverse edge devices with varying data and computation speeds.
problem Scalable distributed optimization with heterogeneous devices and data.
method Ringleader ASGD, an asynchronous SGD algorithm.
result Achieves optimal time complexity under data heterogeneity and arbitrary computation speeds.
HeteroFL trains diverse clients with varying capabilities efficiently.
problem Training models on clients with different computational and communication capacities.
method Proposes HeteroFL framework to adaptively distribute subnetworks based on client capabilities.
result Adaptive subnetwork distribution leads to efficient computation and communication.
A federated minimax framework for heterogeneous clients.
problem Training with edge devices having different datasets and capabilities.
method Proposes a federated minimax optimization framework with normalized updates.
result Improves convergence and communication complexity for nonconvex functions.
We study the fundamental tradeoffs between statistical accuracy and computational tractability in the analysis of high dimensional heterogeneous data. As examples, we study sparse Gaussian mixture model, mixture of sparse linear regressions, and sparse phase retrieval model. For these models, we exploit an oracle-based…
Intratumor heterogeneity is often manifested by vascular compartments with distinct pharmacokinetics that cannot be resolved directly by in vivo dynamic imaging. We developed tissue-specific compartment modeling (TSCM), an unsupervised computational method of deconvolving dynamic imaging series from heterogeneous tumor…
Rescaled ASGD optimizes distributed learning under heterogeneous data.
problem Vanilla ASGD biases towards a frequency-weighted average of local objectives.
method Rescale worker stepsizes by their computation times.
result Rescaled ASGD converges to the correct global objective in fixed-computation model.
Hogwild! adapts to distributed data by varying mini-batch sizes.
problem Efficiently parallelize SGD over distributed local data sets.
method Asynchronous SGD with varying mini-batch sizes, aggregated by an aggregator.
result Improved convergence for heterogeneous data, reducing communication rounds.
Fog learning distributes ML model training across heterogeneous devices and networks.
problem Challenges with conventional federated learning in heterogeneous networks.
method Intelligent distribution of ML model training across nodes from edge devices to cloud servers.
result Enhanced federated learning with multi-layer hybrid framework considering network, heterogeneity, and proximity.
Energy distance measures feature heterogeneity in federated learning.
problem Heterogeneity across data sources hinders model aggregation in federated learning.
method Introduced Taylor approximations of energy distance for efficient computation.
result Taylor approximations accurately capture feature discrepancies, improving convergence.
Paper extends FFT-based differential privacy method to heterogeneous compositions.
problem Computing accurate differential privacy guarantees for mixed mechanisms.
method Uses Fast Fourier Transform (FFT) for error analysis and parameter selection.
result Provides tighter bounds for heterogeneous compositions compared to homogeneous cases.
Freya PAGE optimizes nonconvex optimization with heterogeneous, asynchronous workers.
problem Optimizing nonconvex finite-sum problems with varying worker processing times.
method Freya PAGE, a parallel method robust to stragglers and adaptive to slow computations.
result Freya PAGE offers improved time complexity guarantees compared to previous methods.
UNTIE learns representations of coupled categorical data.
problem Challenges in learning from unlabeled categorical data with complex couplings.
method UNTIE approach for unsupervised representation learning of heterogeneous couplings.
result UNTIE significantly improves categorical data representations on 25 diverse datasets.
Ringmaster ASGD improves Asynchronous SGD's efficiency under varying worker times.
problem Suboptimal performance of Asynchronous SGD under heterogeneous worker computation times.
method Ringmaster ASGD, a novel Asynchronous SGD method with optimal time complexity.
result Ringmaster ASGD achieves optimal time complexity under arbitrary worker heterogeneity.
Improves decentralized learning by optimizing graph mixing for data heterogeneity.
problem Data heterogeneity impacts convergence in decentralized learning, but existing methods ignore this.
method Characterized and quantified the relationship between graph mixing and data heterogeneity. Proposed an optimization approach to improve convergence.
result Our approach leads to improved test performance across various tasks.
This paper tackles objective inconsistency in federated optimization with heterogeneous clients.
problem Objective inconsistency due to heterogeneity in clients' datasets and computation speeds.
method General framework for analyzing federated heterogeneous optimization algorithms, including FedAvg and FedProx, and proposing FedNova.
result FedNova eliminates objective inconsistency while preserving fast error convergence.
Scalable model learns from mixed data groups.
problem Heterogeneous data affecting predictive models and interpretability.
method Joint learning of feature distributions, regression models, and latent group labels.
result Effective in high dimensions, combining data reduction and re-weighting.
A new method reduces communication costs in distributed learning.
problem Reduces communication bottlenecks in distributed learning.
method Local SGD with communication-computation overlap and delay-corrected sparse model averaging.
result Theoretical convergence guarantees for smooth non-convex objectives.
SOLVAR efficiently analyzes cryo-EM data's structural variability.
problem Analyzing continuous heterogeneity in cryo-EM data.
method Low-rank assumption on covariance matrix for tractable estimation.
result Accurately captures dominant components of structural variability.
Efficiently solves heterogeneous QPs by reducing variables using instance-specific projections.
problem Solving high-dimensional quadratic programming problems efficiently.
method Data-driven framework with a graph neural network generating projections tailored to each QP instance.
result Produces high-quality solutions with reduced computation time, outperforming existing methods.
This paper tackles computational bottlenecks in federated learning on mobile devices.
problem Computationally heterogeneous mobile devices hinder federated learning efficiency.
method Proposes efficient algorithms to schedule mobile devices based on data heterogeneity.
result Achieves up to 100x speedup and 7% accuracy gain in federated learning.
New bucketing scheme improves Byzantine robustness for heterogeneous data.
problem Byzantine attacks on federated learning with heterogeneous data.
method Bucketing scheme to adapt robust algorithms to non-iid data.
result Bucketing scheme ensures convergence against Byzantine attacks.
Extends SW and GSW to compare heterogeneous joint distributions.
problem Limited applicability of SW and GSW to heterogeneous joint distributions.
method Introduces HHRT and PGRT to extend SW and GSW.
result H2SW distance for heterogeneous joint distributions.
Federated learning on graphs tackles heterogeneity with efficient parameter estimation.
problem Parameter estimation in federated learning with data distribution and communication heterogeneity.
method Joint estimation of parameters using M-estimation framework with fused Lasso regularization, considering graph structure. result Our estimator achieves optimal rate under certain graph fidelity conditions, similar to centralized aggregation.
Polynomial-time test for detecting dense subgraphs in heterogeneous networks.
problem Detecting a planted community in heterogeneous networks.
method Proposes a polynomial-time test with a standard normal distribution null limiting distribution.
result The test is efficient and performs well in both simulations and real data.
Distributed learning adapts to diverse devices, improving performance.
problem Training neural networks on devices with varying capabilities and resources.
method Each device trains a customized neural network, sharing parameters with others.
result Achieves higher rewards on more powerful devices without sacrificing weaker ones.
Generative models unify heterogeneous data for multimodal fusion.
problem Learning effective representations of mixed numerical and categorical data.
method Bayesian approach with exponential family distributions and Laplace-Bernstein approximation.
result Generative models enable fusion of multimodal data for various machine learning tasks.
FedZKT enables resource-constrained devices to participate in federated learning with heterogeneous models.
problem Inequality in resource allocation hinders participation from resource-constrained devices in federated learning.
method Zero-shot knowledge transfer through a server-assigned distillation process.
result FedZKT effectively transfers knowledge across heterogeneous on-device models without requiring comparable local training efforts.
Study on incentivizing truthfulness in federated learning with heterogeneous data.
problem Manipulated updates in federated learning due to data heterogeneity.
method Formulated a game-theoretic approach to prevent clients from misreporting their gradient updates.
result Developed a payment rule that provably disincentivizes sending modified updates in federated learning.
ATA optimizes task allocation in distributed machine learning.
problem Greedy task allocation leads to inefficiencies in distributed machine learning.
method Adaptive Task Allocation (ATA) adapts to unknown computation time distributions.
result ATA identifies optimal task allocation without prior knowledge of computation times.
S2M optimizes mining for diverse data subpopulations.
problem Scalability and uniformity in training sets with many labels and diverse data.
method Doubly-stochastic mining (S2M) computes per-example and minibatch losses on hardest labels/examples.
result S2M ensures good performance across all data subpopulations.
DFFL tackles federated learning with heterogeneous objectives and constraints.
problem Federated learning with clients having different objectives and feasible regions.
method Derived heterogeneity bounds for cost-vector distances and support-function/shape-distance terms. Lifted pointwise bounds to local-versus-federated excess-risk comparison.
result Federation is beneficial when the statistical advantage of pooling exceeds a client-specific heterogeneity penalty.
Heterogeneity is often natural in many contemporary applications involving massive data. While posing new challenges to effective learning, it can play a crucial role in powering meaningful scientific discoveries through the understanding of important differences among subpopulations of interest. In this paper, we expl…
We discuss the stationary states of a model economy in which N heterogeneous adaptive consumers purchase commodity bundles repeatedly from P sellers. The system undergoes a transition from an inefficient to an efficient state as the number of consumers increases. In the latter phase, however, price fluctuations may…
New spectral clustering method for graphs with uneven node degrees.
problem Challenges in community detection for graphs with heterogeneous degree distributions.
method Spectral clustering on spherical coordinates with degree correction.
result Improved performance in representing computer networks.
Estimates heterogeneous treatment effects in panel data with a new method.
problem Estimating heterogeneous treatment effects in panel data with general treatment patterns.
method Partition observations into clusters with similar treatment effects using a regression tree, then estimate average treatment effects for each cluster.
result Our method achieves superior accuracy compared to alternative approaches.
Electronic health records (EHR) are rich heterogeneous collection of patient health information, whose broad adoption provides great opportunities for systematic health data mining. However, heterogeneous EHR data types and biased ascertainment impose computational challenges. Here, we present mixEHR, an unsupervised g…
A new validation scheme improves Federated Learning performance.
problem Learning a joint model from distributed, heterogeneous data.
method DVW scheme that uses distributed validation sets.
result DVW outperforms FedAvg in heterogeneous environments.
A new method detects communities in multi-relational networks.
problem Detect communities in heterogeneous multi-relational networks.
method Message passing based algorithm on a hierarchical structure of homogeneous networks.
result Effectiveness of the proposed method confirmed on synthetic and real-world networks.
Recent work has developed Bayesian methods for the automatic statistical analysis and description of single time series as well as of homogeneous sets of time series data. We extend prior work to create an interpretable kernel embedding for heterogeneous time series. Our method adds practically no computational cost co…
Study shows how diverse investors' learning and preferences shape financial markets.
problem Understanding how diverse investor behaviors and preferences affect market dynamics.
method Developed a multi-agent reinforcement learning framework with heterogeneous preferences and learning mechanisms.
result Diverse investors develop differentiated strategies through interaction, leading to realistic market dynamics.
This paper uses machine learning to select kernels for machine learning models on various devices.
problem Traditional kernel auto-tuning is limited for machine learning research with changing network topologies and hyperparameters.
method Combines auto-tuning and machine learning to select kernels for SYCL on various devices.
result Initial results show high performance kernel selection with little developer effort.
Ringmaster LMO accelerates training in distributed systems by asynchronously updating neural networks.
problem Asynchronous training in distributed systems where workers compute gradients at different speeds.
method Introduces an asynchronous LMO-based momentum method for unconstrained stochastic nonconvex optimization.
result Establishes convergence guarantees and time complexity bounds for asynchronous LMO-based updates.
Memory bandwidth bottleneck is a major challenges in processing machine learning (ML) algorithms. In-memory acceleration has potential to address this problem; however, it needs to address two challenges. First, in-memory accelerator should be general enough to support a large set of different ML algorithms. Second, it…
Bayesian framework for model uncertainty identifies complex heterogeneity without strong assumptions.
problem Identifying complex heterogeneity in factorial data with varying covariates.
method Rashomon Partition Sets (RPS) using l0 prior for robust model uncertainty.
result RPS provides a robust set of models capturing complex heterogeneity without strong assumptions.
In recent years, dynamically growing data and incrementally growing number of classes pose new challenges to large-scale data classification research. Most traditional methods struggle to balance the precision and computational burden when data and its number of classes increased. However, some methods are with weak pr…
The paper develops a causal machine learning framework to optimize aid allocation.
problem Optimizing aid allocation to reduce new HIV infections in poor countries.
method The framework uses a balancing autoencoder, counterfactual generator, and inference model to predict heterogeneous treatment effects.
result The framework predicts a reduction of up to 3.3% in new HIV infections, saving 50,000 lives.
ie-HGCN addresses HIN challenges by efficiently learning node representations.
problem Lack of flexibility in exploring meta-paths and high computational complexity in HIN GCN methods.
method Hierarchical aggregation architecture that automatically extracts useful meta-paths and reduces computational cost.
result ie-HGCN outperforms state-of-the-art methods on real network datasets.