Paper presents a copula-based method to efficiently generate correlated sample paths from multi-step time series models.
problem Generating realistic correlation structures in multi-step forecast sample paths is expensive and time-consuming.
method Copula-based approach to generate correlated sample paths in one forward pass.
result Improved sample path quality and significant speedup over autoregressive sampling.
Enhances labels from unlabeled data using sample correlations.
problem Lack of label distributions in real-world applications.
method Proposes LESC and gLESC methods to enhance label distributions.
result Improves performance of label enhancement through sample correlations.
CADGMM detects anomalies by capturing complex correlations in data.
problem Detecting anomalies in complex, unstructured data.
method CADGMM uses a graph structure to encode correlations, then a dual-encoder to learn low-dimensional latent space, followed by a Gaussian Mixture Model for anomaly detection.
result CADGMM effectively detects anomalies in real-world datasets.
We describe a method to determine the eigenvalue density of empirical covariance matrix in the presence of correlations between samples. This is a straightforward generalization of the method developed earlier by the authors for uncorrelated samples. The method allows for exact determination of the experimental spectru…
Paper addresses the disparity between sampled and mean representations in disentangled learning.
problem Disparity between sampled and mean representations in disentangled learning.
method Proposes a method to eliminate the disparity by proving and utilizing the relationship between total correlation of sampled and mean representations for multivariate normal distributions.
result Demonstrates that a factorized mean representation can have lower total correlation than the sampled representation.
Improved sample complexity for Gaussian Mixture Models using Pair Correlation Factor.
problem Understanding the sample complexity of Gaussian Mixture Models.
method Introducing Pair Correlation Factor (PCF) to measure clustering of component means and improving sample complexity bounds.
result The Pair Correlation Factor (PCF) more accurately determines the difficulty of parameter recovery in Gaussian Mixture Models.
Estimates covariance matrices with correlations between samples.
problem Estimating large-dimensional covariance matrices with correlated samples.
method Generalized Marcenko-Pastur equation and Ledoit-Peche shrinkage estimator using random matrix theory and free probability. Developed an efficient algorithm based on Ledoit-Wolf kernel estimation.
result Efficient algorithm for estimating large covariance matrices with correlations.
New tensor-based method for estimating stock correlation matrices.
problem Choosing a proper sample period for estimating correlation matrices.
method Slice-Diagonal Tensor (SDT) factorization technique.
result The new method produces a stable correlation matrix unaffected by the sample period.
We introduce a new approach to variable selection, called Predictive Correlation Screening, for predictor design. Predictive Correlation Screening (PCS) implements false positive control on the selected variables, is well suited to small sample sizes, and is scalable to high dimensions. We establish asymptotic bounds f…
ESS improves MCMC efficiency for correlated & multimodal distributions.
problem Slice Sampling's sensitivity to initial length scale and difficulty with correlated distributions.
method Adaptive tuning and parallel walkers for efficient sampling.
result ESS improves efficiency by more than an order of magnitude on correlated distributions.
Correlation matrices play a key role in many multivariate methods (e.g., graphical model estimation and factor analysis). The current state-of-the-art in estimating large correlation matrices focuses on the use of Pearson's sample correlation matrix. Although Pearson's sample correlation matrix enjoys various good prop…
The Epps effect varies under different sampling schemes, affecting correlation emergence rates.
problem Uncertainty in choosing time and sampling rates for financial systems.
method Comparison of Epps effect under calendar, volume, and trade time schemes using a Hawkes process model.
result Correlations emerge faster under trade time compared to calendar time, and linearly under volume time.
This paper treats the problem of screening for variables with high correlations in high dimensional data in which there can be many fewer samples than variables. We focus on threshold-based correlation screening methods for three related applications: screening for variables with large correlations within a single trea…
We study the problem of finding the most mutually correlated arms among many arms. We show that adaptive arms sampling strategies can have significant advantages over the non-adaptive uniform sampling strategy. Our proposed algorithms rely on a novel correlation estimator. The use of this accurate estimator allows us t…
Enhances sensitivity analysis for correlated inputs.
problem Estimating sensitivity indices in models with correlated inputs.
method Proposes an extension of Sobol' estimator using a linear correlation model.
result Improves accuracy in variance-based sensitivity analysis.
Machine learning often needs to model density from a multidimensional data sample, including correlations between coordinates. Additionally, we often have missing data case: that data points can miss values for some of coordinates. This article adapts rapid parametric density estimation approach for this purpose: model…
We study finite sample properties of estimators of power-law cross-correlations -- detrended cross-correlation analysis (DCCA), height cross-correlation analysis (HXA) and detrending moving-average cross-correlation analysis (DMCA) -- with a special focus on short-term memory bias as well as power-law coherency. Presen…
New insights into ridge regression with correlated data, improving risk prediction.
problem Understanding and predicting risk in ridge regression with correlated samples.
method Random matrix theory and free probability for asymptotic analysis; modified GCV estimator (CorrGCV) for unbiased prediction.
result GCV estimator fails for out-of-sample risk with correlated data; CorrGCV provides an unbiased estimator.
Paper develops efficient algorithms for learning rationalizable equilibria in multiplayer games.
problem Learning rationalizable behavior in multiplayer games under bandit feedback.
method New algorithms for finding rationalizable Coarse Correlated Equilibria and Correlated Equilibria with polynomial sample complexity.
result Achieved polynomial sample complexity for learning rationalizable equilibria, improving over existing exponential complexity.
Correlation matrices are omnipresent in multivariate data analysis. When the number d of variables is large, the sample estimates of correlation matrices are typically noisy and conceal underlying dependence patterns. We consider the case when the variables can be grouped into K clusters with exchangeable dependence; t…
Logit correction improves model performance by correcting spurious correlations.
problem Spurious correlations lead to poor model performance during inference.
method Proposes logit correction (LC) loss to mitigate spurious correlations.
result LC loss outperforms state-of-the-art solutions by 5.5% absolute improvement.
Generative Adversarial Networks create realistic financial correlation matrices.
problem Creating realistic financial correlation matrices for practical applications.
method Generative Adversarial Networks (GANs) to model correlation matrices.
result GANs can recover known stylized facts about empirical correlation matrices.
Paper proposes PSIPS for identifying Pareto set with correlated objectives.
problem Identifying the best answer among items with multiple conflicting metrics.
method Posterior sampling in stopping and sampling rules for structure and correlation.
result PSIPS is asymptotically optimal and demonstrates good empirical performance.
We study Principal Component Analysis (PCA) in a setting where a part of the corrupting noise is data-dependent and, as a result, the noise and the true data are correlated. Under a bounded-ness assumption on the true data and the noise, and a simple assumption on data-noise correlation, we obtain a nearly optimal samp…
Mean representations of VAEs are correlated but still useful for tasks.
problem Correlation between mean and sampled representations of VAEs.
method Selective posterior collapse to identify active and passive variables.
result Passive variables in mean representations are correlated but uncorrelated in sampled ones.
Understanding and developing a correlation measure that can detect general dependencies is not only imperative to statistics and machine learning, but also crucial to general scientific discovery in the big data age. In this paper, we establish a new framework that generalizes distance correlation --- a correlation mea…
Study sharpens threshold for matching correlated graphs without labels.
problem Matching latent vertex correspondences in correlated random graphs.
method Analyzes information-theoretic limits for correct vertex matching in sub-sampled graphs.
result Establishes a sharp information-theoretic threshold for vertex matching recovery.
When can reliable inference be drawn in the "Big Data" context? This paper presents a framework for answering this fundamental question in the context of correlation mining, with implications for general large scale inference. In large scale data applications like genomics, connectomics, and eco-informatics the dataset…
Mitigates spurious correlations without bias labels.
problem Spurious correlations bias model performance.
method Introduces a novel training objective and debiasing method DPR.
result DPR achieves state-of-the-art performance.
Given two data matrices X and Y, sparse canonical correlation analysis (SCCA) is to seek two sparse canonical vectors u and v to maximize the correlation between Xu and Yv. However, classical and sparse CCA models consider the contribution of all the samples of data matrices and thus cannot identify an unde…
Paper proposes a chi-square test for distance correlation.
problem Testing distance correlation is computationally expensive.
method Proposes a chi-square test for distance correlation, non-parametric, fast, applicable to various metrics.
result Chi-square test exhibits similar power to permutation test and can be valid and universally consistent for testing independence.
We study the sample complexity of canonical correlation analysis (CCA), \ie, the number of samples needed to estimate the population canonical correlation and directions up to arbitrarily small error. With mild assumptions on the data distribution, we show that in order to achieve ε-suboptimality in a properly define…
Improved best-arm identification in correlated multi-armed bandits.
problem Best-arm identification in multi-armed bandits with correlated rewards.
method Proposed C-LUCB algorithm that exploits upper bounds on conditional rewards.
result Significant reduction in sample complexity for best-arm identification.
New SMC sampler improves diffusion model sampling efficiency.
problem Sampling generative diffusion models efficiently.
method Constructs correlated observation paths and designs a sampler.
result Improved statistical efficiency, especially under outlier conditions.
Losaw improves FI scores by decorrelating features in ML models.
problem Feature correlation distorts feature importance scores in ML models.
method Losaw uses local sample weighting to decorrelate features.
result Losaw consistently improves feature importance scores and prediction accuracy.
A new algorithm removes unexpected correlations in biased data for better clustering.
problem Clustering with selection bias in data.
method Decorrelation regularized K-Means (DCKM) algorithm.
result DCKM achieves significant performance gains on real-world datasets.
VADD enhances discrete diffusion models by capturing inter-dimensional correlations, improving sample quality.
problem Limited modeling of inter-dimensional dependencies in MDMs degrades performance with few denoising steps.
method Introduces an auxiliary recognition model for latent variable modeling, enabling stable training via variational lower bounds maximization and amortized inference.
result VADD consistently outperforms MDM baselines in sample quality with few denoising steps.
Proposes meTS for efficient exploration in correlated bandits.
problem Efficiently explore in bandits with correlated actions.
method Mixed-effect model with Thompson Sampling.
result Bound on Bayes regret with two terms.
Diffusion models learn simple statistics before complex ones, revealing a sample complexity exponent.
problem Understanding the learning dynamics of diffusion models.
method Empirical observations and theoretical analysis of diffusion models and denoisers.
result Diffusion models learn simple statistics (pair-wise correlations) at linear sample complexity, while higher-order statistics (e.g., fourth cumulant) require cubic sample complexity.
We consider the problem of providing nonparametric confidence guarantees for undirected graphs under weak assumptions. In particular, we do not assume sparsity, incoherence or Normality. We allow the dimension D to increase with the sample size n. First, we prove lower bounds that show that if we want accurate infe…
CNN detects multipath in GNSS signals using correlator output.
problem Multipath contamination errors in GNSS signals.
method Deep Convolutional Neural Networks (CNN) applied to GNSS correlator output.
result CNN accurately detects multipath in GNSS signals.
This work explains how maximizing latent correlations across multiple data views helps in identifying shared and private components.
problem Understanding how to identify shared and private components in multiview data.
method An intuitive generative model of multiview data is adopted, and latent correlation maximization is shown to guarantee the extraction of shared components.
result Latent correlation maximization guarantees the extraction of shared components across views and disentangles private information.
Discovering a correlation from one variable to another variable is of fundamental scientific and practical interest. While existing correlation measures are suitable for discovering average correlation, they fail to discover hidden or potential correlations. To bridge this gap, (i) we postulate a set of natural axioms …
Correlations and other collective phenomena in a schematic model of heterogeneous binary agents (individual spin-glass samples) are considered on the complete graph and also on 2d and 3d regular lattices. The system's stochastic dynamics is studied by numerical simulations. The dynamics is so slow that one can meaningf…
Private method measures nonlinear correlations between data hosted across two entities.
problem Measuring nonlinear correlations between sensitive data hosted across multiple parties while preserving privacy.
method Differentially private estimator of distance correlation.
result First private estimator of nonlinear correlations in a multi-party setup.
Sparse GCA finds linear relationships in multiple datasets, using gradient descent.
problem Finding linear relationships across multiple datasets with sparse loading vectors.
method Formulated as generalized eigenvalue problems, used a thresholded gradient descent algorithm.
result Proposed algorithm yields tight estimation error bounds and demonstrates effectiveness on synthetic datasets.
Paper proposes universally consistent K-sample tests using any dependence measure.
problem Testing whether K groups of data points are drawn from the same distribution.
method Demonstrates the use of any dependence measure for K-sample testing.
result Achieves universally consistent K-sample testing using distance correlation and Hilbert-Schmidt independence criterion.
Neurons in the visual cortex are correlated in their variability. The presence of correlation impacts cortical processing because noise cannot be averaged out over many neurons. In an effort to understand the functional purpose of correlated variability, we implement and evaluate correlated noise models in deep convolu…