Mixed-precision CA-SGD for generalized linear models on GPUs
problem SGD communication bottleneck
method Mixed-precision CA-SGD
result Matches FP32 SGD loss within 0.5% on various problems
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
169,341 papers · 148 categories
Trend · papers per month
48 results for “finite precision communication”
Study on distributed coordinate descent with quantized updates for finite precision communication.
problem Finite precision communication limits the accuracy of updates in distributed coordinate descent.
method Introduced a randomized distributed coordinate descent algorithm with quantized updates, derived convergence conditions, and validated with experiments.
result Algorithm with quantized updates converges under certain conditions on the quantization error.
New limits found for power, speed, and precision in physical communication.
problem Limits on power, speed, and precision in physical communication.
method Derivation involving a novel connection between friction and information geometry.
result Product of precision and speed is universally bounded by power dissipation.
Novel AMP framework for multi-environment transfer learning.
problem Characterizing risk of Lasso-based transfer learning estimators.
method Multi-Environment Generalized Long AMP (multi-environment GLAMP) framework.
result Precise characterization of the risk of three Lasso-based transfer learning estimators.
We develop a class of rules spanning the range between quadratic discriminant analysis and naive Bayes, through a path of sparse graphical models. A group lasso penalty is used to introduce shrinkage and encourage a similar pattern of sparsity across precision matrices. It gives sparse estimates of interactions and pro…
A new algorithm reduces communication in distributed SVD by factors.
problem Efficiently compute SVD in distributed systems.
method LocalPower algorithm with weighted aggregation and periodic decay of iterations.
result Reduces communication cost by a factor of .
Distributed Lion optimizes large model training by reducing communication costs.
problem Training large AI models efficiently with reduced communication costs.
method Adapted Lion optimizer for distributed training, using binary or lower-precision vectors for communication.
result Distributed Lion achieves comparable performance to standard optimizers but with significantly reduced communication bandwidth.
A new method reduces communication in distributed learning by 32x.
problem High communication costs in distributed machine learning.
method Distributed compressed SGD with Nesterov's momentum and blockwise compression.
result Achieves 32x reduction in communication cost and converges as fast as full-precision SGD.
DESTRESS optimizes decentralized nonconvex optimization with optimal IFO complexity and efficient communication.
problem Decentralized nonconvex finite-sum optimization in multi-agent systems.
method DESTRESS uses stochastic recursive gradient updates, gradient tracking, and careful hyper-parameter choices to achieve optimal IFO complexity with efficient communication.
result DESTRESS matches the optimal IFO complexity of centralized algorithms while maintaining communication efficiency.
This study analyzes communication constraints in MoE architectures using information theory.
problem Communication constraints in Mixture-of-Experts (MoE) architectures.
method Developed a rate-distortion characterization of finite-rate gating in MoE architectures using information theory.
result Yielded capacity-aware limits for communication-constrained MoE systems.
Improved sampling from high-dimensional Gaussians using smoothed scores.
problem Sampling from high-dimensional Gaussian distributions with gradient information.
method Using smoothed scores, which are gradients of the logarithms of Gaussian-convolved densities, to overcome approximation barriers.
result Improved sampling efficiency with a complexity of \(O\left(\left(\logκ+\log(e\sqrt d/δ_{
m TV})
ight)\log(e\sqrt d/δ_{
m TV})
ight)\) smoothed-score queries.
Finite-precision learning of networks is limited by the Monte Carlo rate.
problem Learning neural networks under finite precision
method Using iterated activations to construct localized bump functions
result No adaptive randomized algorithm can achieve higher convergence rate than Monte Carlo rate in finite precision
Conformal prediction ensures reliable weed spraying guarantees.
problem Lack of trust in deep learning-based precision weeding systems.
method Conformal prediction for providing trustworthy guarantees on black-box models.
result Formal guarantees on spraying at least 90% of weeds.
Paper proposes double quantization to reduce communication in distributed machine learning.
problem High communication overhead in synchronizing stochastic gradients and model parameters in distributed training.
method Proposes double quantization for model parameters and gradients, and three communication-efficient algorithms.
result Established performance guarantees and demonstrated effective bit reduction without performance degradation.
Moniqua improves SGD convergence with quantized communication.
problem Efficiently communicating in decentralized SGD with limited bandwidth.
method Modulo quantized communication in decentralized SGD.
result Moniqua converges at the same rate as full-precision communication with less bits.
Decentralized detection avoids sharing data, controls false discoveries.
problem Global false discovery rate control in decentralized novelty detection.
method Quantized surrogate models for low-precision sharing, preserving exchangeability.
result Quantized composite scores maintain competitive statistical power with reduced communication.
Finite Sample Guarantees for Distributed Online Parameter Estimation with Communication Costseess.SY
The paper provides bounds on estimation error in a distributed online learning setting.
problem Estimating an unknown parameter in a distributed and online manner with finite sample guarantees.
method Proposes a distributed online estimation algorithm that improves accuracy through communication, providing non-asymptotic bounds on estimation error.
result Demonstrates a trade-off between estimation error and communication costs, and determines a stopping time for communication based on desired accuracy.
Paper detects communities from graph signals using low-rank excitation modeling.
problem Detect communities in graphs from noisy signals.
method Model signals as graph filter outputs, apply spectral method to covariance matrix.
result Community structure can be retrieved directly from graph signals.
Finite precision RNNs have varying computational power, with LSTMs and ReLU-RNNs being more powerful.
problem Understanding the computational limits of finite precision RNNs for language recognition.
method Comparison of different RNN variants with finite precision and linear computation time.
result LSTMs and ReLU-RNNs are strictly stronger than other RNN variants in terms of computational power.
SparCML improves machine learning scalability by handling sparse data efficiently.
problem Scalability bottleneck in machine learning due to global allreduce operations.
method Design and implement communication protocols for sparse data, extending MPI.
result SparCML extends MPI to support sparse data, improving scalability.
Evaluating accuracy of community detection using the relative normalized mutual informationphysics.soc-ph
The Normalized Mutual Information (NMI) has been widely used to evaluate the accuracy of community detection algorithms. However in this article we show that the NMI is seriously affected by systematic errors due to finite size of networks, and may give a wrong estimate of performance of algorithms in some cases. We gi…
Efficiently addresses federated learning challenges with reduced communication and sample complexity.
problem Heterogeneity in data volumes and distributions at different clients compromises model generalization ability.
method Introduces algorithms for communication-efficient Federated Group Distributionally Robust Optimization (FGDRO).
result Communication complexity reduced to for FGDRO-CVaR and for FGDRO-KL.
This paper discusses the role of risk communication in macroprudential oversight and of visualization in risk communication. Beyond the soar in data availability and precision, the transition from firm-centric to system-wide supervision imposes vast data needs. Moreover, except for internal communication as in any orga…
New techniques improve distributed training with compressed gradients.
problem Gradient mismatch problem in local error feedback.
method Step-ahead error feedback and error averaging techniques.
result Our methods handle gradient mismatch and train faster than full-precision training.
Paper presents an algorithm for optimal regret in communicating Markov decision processes.
problem Achieving optimal regret in Markov decision processes with a communicating assumption.
method The algorithm explicitly tracks the constant K(M) to learn optimally, balancing exploration, co-exploration, and exploitation.
result The algorithm achieves asymptotically optimal regret for communicating Markov decision processes.
Paper proposes SCALLION and SCAFCOM for compressed FL with reduced communication.
problem Reducing communication overhead in Federated Learning with data heterogeneity and partial participation.
method Revisit and simplify stochastic controlled averaging, proposing SCALLION and SCAFCOM for unbiased and biased compression.
result SCALLION and SCAFCOM outperform existing methods in communication and computation complexities.
Study shows peers' graduation improves residents' success in TCs.
problem Identifying peer influence in therapeutic communities adjusting for latent homophily.
method Used data on affirmations and exit dates to form peer networks, modeled latent homophily, and proposed bias correction methods.
result Positive effect of peers' graduation on residents' graduation, varying by gender, race, and role model definition.
Improved community detection using effective resistance-based seed sets.
problem Low quality of seed nodes hinders community detection.
method Two-step process: effective resistance-based germination followed by PageRank.
result Improves precision and recall in community detection.
Study shows impossibility of clustering in a specific graph model.
problem Tackles the impossibility of clustering in a Degree-Corrected Stochastic Block Model.
method Analyzes the Degree-Corrected Stochastic Block Model with specific parameters and weights.
result Proves it is impossible to estimate clusters positively correlated with true community structure under certain conditions.
Targeting Underrepresented Populations in Precision Medicine: A Federated Transfer Learning Approachstat.ML
Proposes a federated transfer learning method to improve precision medicine models for underrepresented populations.
problem Underrepresentation of minorities in precision medicine research leads to underperforming risk prediction models.
method Two-way federated transfer learning strategy integrating diverse populations and healthcare institutions.
result Improves risk prediction models for underrepresented populations, reducing performance gaps.
APMSqueeze improves Adam for faster training with less communication.
problem Adam's inefficiency in parallel training due to gradient compression limitations.
method Error-compensated gradient compression for Adam-preconditioned Momentum SGD.
result Significantly reduces communication time per epoch without sacrificing accuracy.
We propose a new statistical model for computational linguistics. Rather than trying to estimate directly the probability distribution of a random sentence of the language, we define a Markov chain on finite sets of sentences with many finite recurrent communicating classes and define our language model as the invarian…
We develop a secure aggregation protocol for federated learning that reduces communication and computation costs.
problem Expensive communication and privacy concerns in federated learning.
method Adapting compression-based federated techniques to additive secret sharing.
result Our protocol achieves high accuracy with low communication costs and is more efficient than prior work.
Optimal algorithms found for distributed optimization in networks.
problem Efficiently solving optimization problems in distributed systems.
method Developed optimal centralized and decentralized algorithms for smooth and strongly convex optimization.
result Optimal convergence rates for distributed optimization in centralized and decentralized settings.
Secure communication system detects deletions and eavesdropping.
problem Reliable communication over deletion channels with eavesdropping.
method Probabilistic finite-state automata for encoding and decoding.
result Scheme ensures small error probability in decoding and tamper detection.
A new topology improves decentralized learning efficiency and accuracy.
problem Finding efficient decentralized learning topologies with fast consensus and low maximum degree.
method Proposed the Base- Graph topology for decentralized learning.
result The Base- Graph enables faster convergence and better communication efficiency than the exponential graph.
New method improves FL efficiency by shuffling data, balancing privacy and accuracy.
problem Balancing privacy, communication, and accuracy in federated learning.
method Developed communication-efficient schemes for private mean estimation, combining privacy amplification and shuffled data.
result Achieved same privacy, optimization performance with lower communication cost.
Model identifies urgent radiology reports with high accuracy.
problem Lack of annotated training data for text analysis.
method Self-supervised contextual language representation using BERT.
result Model achieved 97.0% precision, 93.3% recall, and 95.1% F-measure.
The paper introduces a method for detecting principal communities and embedding vertices.
problem Detecting and embedding vertices in graphs with community structure.
method Principal graph encoder embedding method that detects principal communities and produces vertex embeddings.
result The method successfully detects principal communities and produces accurate vertex embeddings.
Federated learning clusters EMRs to improve mortality and stay predictions.
problem Decentralized, non-IID EMRs complicate centralized machine learning algorithms.
method Introduced CBFL algorithm that clusters EMRs into communities for federated learning.
result CBFL outperforms baseline FL algorithm in ROC AUC, PR AUC, and communication cost.
RiskNet predicts penalties in unreliable communication networks using GNNs.
problem Predicting penalties in networks with unreliable resources.
method Graph Neural Network (GNN) based approach trained on random graphs.
result Precisely models penalties across various network topologies.
Communication-efficient Distributed Estimation and Inference for Transelliptical Graphical Modelsstat.ML
Efficiently estimates graph models across multiple machines.
problem Estimating graph models in high-dimensional data with limited communication.
method Distributed estimation and inference for transelliptical graphical models.
result Aggregated estimator achieves same statistical rate as centralized estimator.
Paper finds exact recovery threshold in general hypergraph model.
problem Exact recovery of communities in general hypergraph model.
method Developed a two-stage polynomial-time algorithm for exact recovery.
result Sharp threshold for exact recovery in terms of generalized Chernoff-Hellinger divergence.
New model estimates unknown number of users in asynchronous multiuser communication.
problem Estimating unknown number of users in asynchronous multiuser communication.
method Infinite factorial finite state machine model with Bayesian nonparametric approach.
result Effective recovery of data-generating process for various scenarios.
Two-stage TMLE reduces bias and improves efficiency in CRTs.
problem Differential outcome measurement and imbalance in baseline predictors in CRTs.
method Two-stage targeted minimum loss-based estimator (TMLE) to adjust for baseline covariates.
result Our approach nearly eliminates bias due to differential outcome measurement.
Privacy utility trade offs for parameter estimation in degree heterogeneous higher order networksstat.ML
Study privacy vs. utility in estimating network parameters with aggregated data.
problem Privacy-preserving estimation of network parameters from aggregated node degrees.
method β model, local and central differential privacy, minimax lower bounds, simple estimators.
result Achieved minimax-optimal risk bounds for parameter estimation under privacy constraints.
Paper uses EXIT analysis for community detection with side information.
problem Community detection in the presence of side information.
method Imported EXIT method from iterative decoding of error control codes.
result Predicts asymptotic phase transition and residual errors for community detection.
Community detection is the task of detecting hidden communities from observed interactions. Guaranteed community detection has so far been mostly limited to models with non-overlapping communities such as the stochastic block model. In this paper, we remove this restriction, and provide guaranteed community detection f…