We propose a semiparametric approach, named nonparanormal skeptic, for estimating high dimensional undirected graphical models. In terms of modeling, we consider the nonparanormal family proposed by Liu et al (2009). In terms of estimation, we exploit nonparametric rank-based correlation coefficient estimators includin…
Recent methods for estimating sparse undirected graphs for real-valued data in high dimensional problems rely heavily on the assumption of normality. We show how to use a semiparametric Gaussian copula--or "nonparanormal"--for high dimensional inference. Just as additive models extend linear models by replacing linear …
The paper shows how to infer conditional independence from non-Gaussian data.
problem Inferring conditional independence from non-Gaussian distributions.
method Developed a method to recover conditional independence structure from the precision matrix of generalized nonparanormal data.
result The conditional independence structure can be inferred from the precision matrix of generalized nonparanormal data.
CNR uses convex optimization to estimate conditional distributions.
problem Estimating uncertainty in predictions and posterior conditional distributions.
method Convex optimization of a posterior defined via non-linear transformations on Gaussians.
result CNR can fit arbitrary conditional distributions, including multimodal and non-symmetric ones.
Estimates mutual information for nonparanormal distributions robustly.
problem Estimating mutual information for non-Gaussian distributions robustly.
method Proposes estimators for mutual information in nonparanormal models.
result Proposed estimators are robust and scalable with dimensionality.
New method for multivariate distribution regression using NPT metric.
problem Regression with multivariate distributional responses and Euclidean predictors.
method Fréchet regression with nonparanormal transport (NPT) metric.
result Efficient estimation and granular interpretation of predictor effects.
New method detects edges in time-varying networks without minimum signal strength.
problem Detecting edges in time-varying, heavy-tailed, nonparanormal networks.
method Time-varying nonparanormal graphical models, high-dimensional debiasing-free moment estimator, kernel smoothed Kendall's tau correlation matrix.
result Minimax optimal rate of convergence for estimating latent inverse Pearson correlation matrix.
In this paper, we propose a semiparametric approach, named nonparanormal skeptic, for efficiently and robustly estimating high dimensional undirected graphical models. To achieve modeling flexibility, we consider Gaussian Copula graphical models (or the nonparanormal) as proposed by Liu et al. (2009). To achieve estima…
SIMULE learns multiple sparse UGMs from aggregated data, identifying context-specific and shared interactions.
problem Jointly estimating multiple sparse UGMs from aggregated samples across different contexts.
method Constrained L1 minimization approach for multi-UGM learning.
result SIMULE achieves consistent results at rate O(log(Kp)/n_{tot}) and significantly improves over state-of-the-art methods.
This paper proposes a unified framework to quantify local and global inferential uncertainty for high dimensional nonparanormal graphical models. In particular, we consider the problems of testing the presence of a single edge and constructing a uniform confidence subgraph. Due to the presence of unknown marginal trans…
Diagonal transformations preserve independence structures in non-Gaussian distributions.
problem Preserving independence structures in non-Gaussian distributions.
method Diagonal nonlinear transformations of multivariate normal variables.
result Independence structures are preserved in non-Gaussian distributions under diagonal transformations.
Bayesian semi-supervised learning for multi-class classification.
problem Classification with missing labels using unlabeled data.
method Bayesian approach with B-splines and conjugate priors.
result Proposed method outperforms other methods in prediction accuracy.
New methods estimate brain connectivity from calcium imaging data with missing data.
problem Estimating functional neuronal connectivity from calcium imaging data with missing data.
method Two approaches for nonparanormal Graph Quilting based on the Gaussian copula graphical model.
result Our methods yield more meaningful functional connectivity estimates than existing Gaussian graph quilting methods.
Graphical models are commonly used tools for modeling multivariate random variables. While there exist many convenient multivariate distributions such as Gaussian distribution for continuous data, mixed data with the presence of discrete variables or a combination of both continuous and discrete variables poses new cha…
This paper tackles structure learning in indirect observations of Gaussian and non-Gaussian random vectors.
problem Learning the graphical structure of random vectors indirectly observed through a sensing matrix and corrupted noise.
method Parametric and non-parametric approaches for Gaussian and non-Gaussian distributions, respectively.
result Correct graphical structure can be recovered under indefinite sensing systems with insufficient samples.
Proposes RSP model for efficient big data analysis.
problem Efficiently partitioning big data sets for analysis.
method Random sample partition (RSP) data model and block-level sampling.
result RSP data blocks can estimate statistics and build models equivalent to whole data set.
Data preprocessing improves data quality for robust data mining.
problem Noisy and incomplete data hinders data mining models.
method Overview of data cleaning, transformation, and preprocessing methods.
result Preprocessing significantly affects data mining model performance.
A new method for handling imbalanced big data using ensembles and smart data.
problem Imbalanced data distribution in big data scenarios.
method Smart Data driven Decision Trees Ensemble (SD_DeTE) methodology.
result SD_DeTE outperforms Random Forest in handling imbalanced binary classification problems in big data.
Prevents sensitive data generation in diffusion models using labeled and unlabeled data.
problem Generating sensitive data in diffusion models using unlabeled data.
method Positive-Unlabeled Diffusion Models, approximating ELBO with labeled and unlabeled data.
result Prevents the generation of sensitive data without compromising image quality.
Study reveals Data Shapley's inconsistent performance in data selection tasks.
problem Inconsistency of Data Shapley's performance in data selection across different settings.
method Hypothesis testing framework and identification of utility functions.
result Data Shapley's performance is no better than random selection without specific constraints.
Survey on data collection challenges in machine learning.
problem Data scarcity and need for labeled data in machine learning.
method Comprehensive study of data acquisition, labeling, and improvement techniques.
result Identification of research challenges in data collection.
PRRO generates synthetic tabular data that improves SL performance and class distribution.
problem Low SL utility of synthetic data due to class imbalance and overlooked data relationships.
method Data pruning and column reordering to optimize SL utility.
result Synthetic data generated with PRRO enhances predictive performance and class distribution.
Defines data science as a natural ecosystem with challenges and missions.
problem Challenges and missions in data science due to 5D complexities and data life cycle phases.
method Systemic and data-centric view of data science as a fusion of data universe and its challenges, formalizing a general-purpose architecture.
result Essential data science as a natural ecosystem integrating specific disciplines and high-impact applications.
Synthetic data enhances analytics but requires careful volume management.
problem Accuracy of statistical methods on synthetic data vs. raw data.
method Synthetic Data Generation for Analytics framework using tabular diffusion models.
result Error rate decreases with more synthetic data but may stabilize or increase.
Data science redefines causal inference from observational data, classifying tasks into description, prediction, and counterfactual prediction.
problem Widespread misunderstandings about data science's role in causal inference from observational data.
method Organizing data science tasks into three classes: Description, prediction, and counterfactual prediction (including causal inference).
result The necessity of subject-matter expert knowledge for causal analyses in data science.
This paper evaluates how dirty data affects data mining and machine learning results.
problem Negative impacts of dirty data on data mining and machine learning results.
method Experimental comparison of missing, inconsistent, and conflicting data on classification and clustering algorithms.
result Guidelines for algorithm selection and data cleaning based on experimental findings.
DPASF stream preprocesses Big Data streams efficiently.
problem Efficient preprocessing of streaming Big Data.
method Implemented six preprocessing algorithms in Apache Flink.
result Preprocessing improves data accuracy in streaming Big Data.
This paper introduces C-DSL to improve data mining outcomes by considering context.
problem Data collection ambiguities, data imbalance, hidden biases, lack of domain info, and data incompleteness.
method Developed Context-Driven Data Science Lifecycle (C-DSL) to address data quality issues.
result Tangible improvements to data mining outcomes were achieved through C-DSL.
Proposes using probabilistic models for privacy-preserving synthetic data.
problem Designing high-quality synthetic data for privacy preservation.
method Formulate the problem through probabilistic modelling, choosing a model for the data.
result Statistical discoveries can be reliably reproduced from synthetic data.
Unlabeled data helps stop active learning better than labeled data.
problem Reducing the need for manual annotation in text classification.
method Compared stopping methods based on labeled, unlabeled, and training data.
result Stopping methods using unlabeled data are more effective.
New test ensures quality of shared data in machine learning.
problem Ensuring quality of external data in machine learning tasks.
method Distribution-free two-sample testing procedures grounded in conformal outlier detection.
result Identifies valuable external data agents for model personalization.
Paper creates fair synthetic data ensuring equal predictions across sensitive attributes.
problem Ensuring fair predictions across sensitive attributes in synthetic data.
method Equalizing target probability distributions across sensitive attributes in synthetic data generation.
result Synthetic data provides strong fair predictions, equal across all thresholds.
A new method classifies multiple correlated data streams simultaneously.
problem Classifying multiple correlated data streams in practical scenarios.
method Double-Coupling Support Vector Machines (DC-SVM) considers both internal and external correlations.
result The proposed method outperforms traditional methods on artificial and real-world data streams.
This paper improves neural machine translation training by selecting and denoising data.
problem Reduces negative impact of noisy data on neural machine translation training.
method Measures and selects domain data, applies denoising curriculum using online data selection.
result Significant effectiveness for training on noisy data.
DPA preserves data distribution in reduced dimensions.
problem Loss of data distribution in dimension reduction.
method DPA combines encoder and decoder to match data distribution.
result DPA successfully reconstructs data distribution.
Framework captures missing data in sparse data sets.
problem Capturing missing data in extremely sparse data sets.
method Coupled compound Poisson factorization with stochastic variational inference.
result Explicitly modeling missing data improves results in clustering, prediction, and matrix factorization.
Efficient synthetic data generation improves model performance on tabular data.
problem Improving model robustness and performance with scarce or low-quality data.
method Hardness characterization to identify high-value training points, generating synthetic data only from these points.
result Synthetic data generated from hardest points outperforms non-targeted methods on tabular datasets.
For most problems in science and engineering we can obtain data sets that describe the observed system from various perspectives and record the behavior of its individual components. Heterogeneous data sets can be collectively mined by data fusion. Fusion can focus on a specific target relation and exploit directly ass…
DAERNN models censored data using neural networks with data augmentation.
problem Handling censored data in expectile regression.
method Data augmentation based Expectile Regression Neural Networks (ERNNs).
result DAERNN outperforms existing censored ERNNs methods and achieves comparable predictive performance to fully observed data.
This paper quantifies uncertainty in Data Shapley using statistical inference.
problem Uncertainty in data valuation due to dynamic data distribution.
method Established relationship with U-statistics and quantified uncertainty using statistical inference.
result Confidence intervals for Data Shapley estimations are provided.
Generative Adversarial Networks create time series data from images.
problem Generating realistic time series data from images.
method Wasserstein GANs with gradient penalty for stability, synthesizing sinusoidal, PPG, and ECG data.
result Successfully generated time series data using image-based GANs.
DCoM uses deep neural networks to detect semantic data types from raw column values.
problem Detecting semantic data types from dirty and unseen data.
method DCoM employs multi-input NLP-based deep neural networks trained on 686,765 data columns.
result DCoM outperforms existing methods significantly on 78 different semantic data types.
Model refines coarse spatial data using diverse auxiliary data sets.
problem Tackles the challenge of refining coarse spatial data with varying auxiliary data granularities.
method Proposes a probabilistic model using Gaussian processes to hierarchically incorporate auxiliary data sets of various granularities.
result Can effectively refine coarse-grained spatial data using auxiliary data sets of different granularities.
GANs generate training data for machine learning tasks.
problem Imbalanced data sets and sensitive information.
method Generative Adversarial Networks (GANs) to create artificial training data.
result A Decision Tree classifier trained on GAN-generated data achieved similar or better accuracy and recall than on original data.
Task-agnostic data valuation without validation requirements.
problem Valuing data without specific task assumptions.
method Estimating data diversity and relevance through queries without raw data.
result Estimates capture the diversity and relevance of seller's data for the buyer.
Framework generates private synthetic data for unlabeled mixed-type data.
problem Generating private synthetic data for unlabeled mixed-type data.
method Combines autoencoders and GANs for differential privacy.
result Learned model generates synthetic data with similar statistical properties.
New algorithm improves data imputation for complex multimodal data sets.
problem Artifacts in imputation methods for multimodal distributions.
method Combines kNN and KDE for probabilistic estimates. result Lower imputation errors and higher likelihood estimates.
New algorithms for clustering and synthetic data generation of heterogeneous tabular datasets.
problem Clustering and generating synthetic data from heterogeneous tabular datasets with hidden cluster structure.
method Developed MMM and MMMsynth algorithms for clustering and synthetic data generation.
result MMMsynth algorithm outperforms other literature tabular-data generators and approaches real data performance.