High-dimensional data simplifies problems, contrary to the curse of dimensionality.
problem Exponential difficulty in high-dimensional problems.
method Analysis of high-dimensional datasets and their geometric properties.
result Generic high-dimensional datasets exhibit simple geometric properties.
New research shows CFG improves high-dimensional data generation.
problem Characterizing CFG's effect on high-dimensional distributions.
method High-dimensional analysis of CFG's impact on target distributions.
result CFG accurately reproduces the target distribution in high dimensions.
Ogburn et al. (2019, arXiv:1910.05438) discuss "The Blessings of Multiple Causes" (Wang and Blei, 2018, arXiv:1805.06826). Many of their remarks are interesting. But they also claim that the paper has "foundational errors" and that its "premise is...incorrect." These claims are not substantiated. There are no foundatio…
Regularization can improve both privacy and performance in machine learning models.
problem Privacy vs. Utility trade-off in machine learning models.
method The study uses logistic regression with ridge regularization and a leave-one-out analysis tool.
result Increasing the number of parameters can improve both privacy and performance when coupled with proper regularization.
High-dimensional models pose both safety benefits and risks.
problem Emergent problems in safety alignment due to high-dimensional representations.
method Detailed visualizations and lower-dimensional subspace projections.
result Dimensional reduction preserves safety alignment while avoiding linear jailbreaking.
In this paper we show that very large mixtures of Gaussians are efficiently learnable in high dimension. More precisely, we prove that a mixture with known identical covariance matrices whose number of components is a polynomial of any fixed degree in the dimension n is polynomially learnable as long as a certain non-d…
The power of neural networks lies in their ability to generalize to unseen data, yet the underlying reasons for this phenomenon remain elusive. Numerous rigorous attempts have been made to explain generalization, but available bounds are still quite loose, and analysis does not always lead to true understanding. The go…
Counterexamples show deconfounder fails to control multi-cause confounding.
problem Deconfounding method fails to handle multi-cause confounding.
method Incorrectly inferring independence and joint independence from conditional independence.
result Two simple counterexamples demonstrate deconfounder's failure.
This commentary has two goals. We first critically review the deconfounder method and point out its advantages and limitations. We then briefly consider three possible ways to address some of the limitations of the deconfounder method.
New MCMC methods map high-dimensional problems to spheres for better mixing.
problem Mixing issues in high-dimensional distributions, especially heavy-tailed ones.
method Stereographic Markov Chain Monte Carlo (MCMC) methods that map high-dimensional problems to spheres.
result Uniformly ergodic samplers for various distributions, including heavy-tailed ones, with faster convergence in higher dimensions.
New algorithm tackles multi-agent reinforcement learning issues.
problem Multi-agent reinforcement learning suffers from the curse of many agents.
method Proposes MF-FQI algorithm based on mean embeddings of distributions.
result Establishes a non-asymptotic analysis for MF-FQI algorithm.
(This comment has been updated to respond to Wang and Blei's rejoinder [arXiv:1910.07320].) The premise of the deconfounder method proposed in "Blessings of Multiple Causes" by Wang and Blei [arXiv:1805.06826], namely that a variable that renders multiple causes conditionally independent also controls for unmeasured mu…
This paper examines how adversarial perturbations affect model performance and equilibrium learning.
problem Adversarial perturbations and covariate shifts impact model performance and equilibrium learning.
method Characterizes the extrapolation region in regression and classification, analyzes dynamics of adversarial learning games.
result Establishes two directional convergence results: a blessing in regression and a curse in classification.
Unobserved confounding is a major hurdle for causal inference from observational data. Confounders---the variables that affect both the causes and the outcome---induce spurious non-causal correlations between the two. Wang & Blei (2018) lower this hurdle with "the blessings of multiple causes," where the correlation st…
We show that deep networks are better than shallow networks at approximating functions that can be expressed as a composition of functions described by a directed acyclic graph, because the deep networks can be designed to have the same compositional structure, while a shallow network cannot exploit this knowledge. Thu…
A novel method for common and individual feature analysis from exceedingly large-scale data is proposed, in order to ensure the tractability of both the computation and storage and thus mitigate the curse of dimensionality, a major bottleneck in modern data science. This is achieved by making use of the inherent redund…
Improved multi-task averaging reduces mean squared error in high-dimensional data.
problem Joint estimation of multiple distributions using independent data sets.
method Exploits similarities between tasks by shrinking naive estimators towards local averages.
result The method provides a significant reduction in mean squared error, especially in high-dimensional spaces.
Paper shows how noisy data can improve robust decision-making.
problem The challenge of noisy data in decision-making.
method Distributionally robust optimization (DRO) with a novel ambiguity set construction.
result Noisy data can lead to more robust and equitable decisions.
The interaction between transitivity and sparsity, two common features in empirical networks, implies that there are local regions of large sparse networks that are dense. We call this the blessing of transitivity and it has consequences for both modeling and inference. Extant research suggests that statistical inferen…
The aim of this comment (set to appear in a formal discussion in JASA) is to draw out some conclusions from an extended back-and-forth I have had with Wang and Blei regarding the deconfounder method proposed in "The Blessings of Multiple Causes" [arXiv:1805.06826]. I will make three points here. First, in my role as th…
BDDMs eliminate noise conditioning in diffusion models, simplifying training and sampling.
problem Noise conditioning in diffusion models is ad hoc and requires unprincipled noise embeddings.
method Introduce blind denoising diffusion models (BDDMs) that do not require noise conditioning.
result BDDMs simplify training and sampling by eliminating noise conditioning.
Stacking improves inference for multimodal Bayesian posterior distributions.
problem Difficulty of MCMC in moving between modes and underestimation of posterior uncertainty.
method Parallel runs of MCMC, variational, or mode-based inference, combined using Bayesian stacking.
result Stacking efficiently samples from multimodal posterior distributions and represents uncertainty better than variational inference.
This paper tackles nonsmooth optimization in machine learning.
problem Nonsmoothness in machine learning optimization problems.
method Identifying specific structures and leveraging them for practical applications.
result Compression, acceleration, and dimension reduction are possible with nonsmooth optimization.
CARE method estimates precision matrix for compositional data, achieving optimality in high dimensions.
problem Challenges in inferring conditional dependence relationships in high-dimensional compositional data.
method Composition adaptive regularized estimation (CARE) method for sparse basis precision matrix.
result CARE estimator achieves minimax optimality in high dimensions, performing as well as if the basis were observed.
Cluster analysis is a fundamental tool for pattern discovery of complex heterogeneous data. Prevalent clustering methods mainly focus on vector or matrix-variate data and are not applicable to general-order tensors, which arise frequently in modern scientific and business applications. Moreover, there is a gap between …
New optimization methods improve Cox Proportional Hazards model training for high-dimensional data.
problem Vanishing second order derivatives in Newton method prevent convergence for high-dimensional CPH model training.
method Construct and minimize surrogate functions exploiting hidden mathematical structures of CPH model.
result Global convergence and monotonic loss decrease, leading to sparse and high-quality models.
Federated Q-learning achieves linear speedup with heterogeneity, improving sample complexity.
problem Collaborative learning in distributed RL settings with limited data sharing.
method Analyzes synchronous and asynchronous federated Q-learning, proposes importance averaging.
result Achieves linear speedup with heterogeneity, robust to local trajectory heterogeneity.
CRL framework groups features for multivariate learning with sparse and dense problems.
problem Sparse and dense problems in supervised multivariate learning.
method Clustered reduced-rank learning (CRL) with joint matrix regularizations.
result CRL framework is more interpretable and relaxes sparsity assumption.
Kernel embeddings separate distinct probability distributions, simplifying testing.
problem Testing equality of non-atomic probability distributions.
method Kernel covariance embeddings and Gaussian measures in reproducing kernel Hilbert spaces.
result Testing for singularity between Gaussian measures is equivalent to testing for equality of non-atomic probability distributions.
This work proposes external correctors for quick AI error corrections without system modification.
problem Quick corrections of AI errors without modifying legacy systems.
method Special `external` devices with classifiers for high-dimensional data.
result Simple classifiers can correct small errors in high-dimensional data.
Causal inference from observational data often assumes "ignorability," that all confounders are observed. This assumption is standard yet untestable. However, many scientific studies involve multiple causes, different variables whose effects are simultaneously of interest. We propose the deconfounder, an algorithm that…
Method estimates shared and study-specific factors for multi-study data.
problem Covariance estimation for multi-study data with shared and study-specific components.
method Spectral decomposition for latent factors, surrogate Bayesian regressions for loadings and variances.
result Strong frequentist guarantees and superior performance in simulations and real data.
The objective of the paper is to study accuracy of multi-class classification in high-dimensional setting, where the number of classes is also large ("large L, large p, small n" model). While this problem arises in many practical applications and many techniques have been recently developed for its solution, to t…
Simple greedy algorithms can excel in multi-objective bandits with multiple good arms.
problem Optimizing multiple objectives in bandits is traditionally harder.
method Introduced greedy algorithms that exploit multiple good arms for multiple objectives.
result Simple greedy algorithms achieve strong performance in multi-objective bandits.
The paper improves bandit algorithms by incorporating random-effect models.
problem Improving statistical efficiency in multi-armed bandit problems with misspecified priors.
method Introduces a random-effect model to bandits, estimating arm means and designing a UCB algorithm ReUCB.
result Derives an upper bound on the Bayes regret of ReUCB, showing improved performance over Thompson sampling.
Motivated by the pursuit of a systematic computational and algorithmic understanding of Generative Adversarial Networks (GANs), we present a simple yet unified non-asymptotic local convergence theory for smooth two-player games, which subsumes several discrete-time gradient-based saddle point dynamics. The analysis rev…
New RL method tackles dynamic MDPs with evolving rewards and states.
problem Dynamic MDPs with evolving rewards and states.
method Sliding Window Upper-Confidence bound for Reinforcement Learning (SWUCRL2-CW) and Bandit-over-Reinforcement Learning (BORL).
result Achieves dynamic regret bound for non-stationary MDPs.
DeFi exploits lead to reduced CP spreads, contrary to contagion hypothesis.
problem Vulnerabilities in DeFi destabilize traditional short-term funding markets.
method Analysis of commercial paper spreads and regulatory segmentation.
result DeFi exploits lead to a 'Flight-to-Quality' pattern, narrowing rather than widening CP spreads.
Mitigates DeFi liquidations with reversible call options.
problem Systemic failures in DeFi due to liquidations.
method Introduces reversible call options to prevent liquidations.
result Reduces liquidated collateral by 89.82% in simulations.
Conditions for uniquely identifying parameters of deep ReLU networks.
problem Characterizing networks whose parameters can be uniquely identified.
method Conditions on deep fully-connected feedforward ReLU neural networks.
result Parameters of the network are uniquely identified under certain conditions.
A new method for speaker recognition on hyperspheres improves on PLDA's limitations.
problem Improving speaker recognition on hyperspheres with PLDA's limitations.
method Probabilistic Spherical Discriminant Analysis (PSDA) using Von Mises-Fisher distributions.
result PSDA scores are closed-form and can handle various trials, improving over PLDA.
New analysis shows diverse classes in pre-training boost NLP performance.
problem Improving sample efficiency in downstream NLP tasks.
method Proved that diverse classes in pre-training lead to better performance, using a large last linear layer singular value.
result Transfer learning excess risk improves with large ildeν and $O\left(\frac{1}{ ildeν \sqrt{n}}
ight)$ rate. The last decades have seen a surge of interests in distributed computing thanks to advances in clustered computing and big data technology. Existing distributed algorithms typically assume {\it all the data are already in one place}, and divide the data and conquer on multiple machines. However, it is increasingly ofte…
Proposes mutual information for regression without prior knowledge.
problem Regression tasks without prior model knowledge.
method Mutual information learning formulation, SGD convergence analysis.
result High dimensionality can be beneficial with a threshold.
New method identifies proxies for causal effects on multiple outcomes.
problem Estimating causal effects in scenarios with multiple outcomes and treatments.
method Causal discovery method leveraging multiple outcomes as proxies for each treatment effect.
result Parallel studies of multiple outcomes can assist in causal identification.
Deeper models have a more favorable optimization landscape, making them more robust to noise.
problem Characterizing the effect of depth on the optimization landscape of linear regression models.
method Robust and over-parameterized setting, simple sub-gradient method.
result A simple sub-gradient method converges to a balanced solution that is close to the ground truth and enjoys a flat local landscape.
Enhances random forest performance with exogenous randomness.
problem Improving random forest performance through exogenous randomness.
method Developed non-asymptotic MSE expansions for individual trees and forests, identified two types of randomness, and conducted simulations.
result Exogenous randomness, particularly feature subsampling, reduces both bias and variance of random forests.
New method preserves privacy while detecting communities in distributed networks.
problem Privacy-preserving community detection in locally distributed multi-layer networks.
method Privacy-preserving Distributed Spectral Clustering (ppDSC) using randomized response mechanism.
result Developed a novel algorithm that maintains community structure while protecting privacy.