This paper examines risks and uncertainties of changing data sources in machine learning for official statistics.
problem Risks and uncertainties associated with changing data sources in machine learning for official statistics.
method An overview of risks, causes, and repercussions of changing data sources, with a checklist of measures.
result Maintaining integrity, reliability, consistency, and relevance in official statistics.
Paper introduces data-dependent SSP for private linear and logistic regression.
problem Private linear and logistic regression with better performance.
method Data-dependent sufficient statistic perturbation (SSP) for linear and logistic regression.
result Data-dependent SSP outperforms state-of-the-art methods for linear and logistic regression.
Unified approach to private statistics from empirical to population data.
problem Divided focus on empirical vs population statistics in private statistics.
method Unified methods for both types of statistics.
result Methods for empirical statistics can be applied to population statistics.
This article provides the role of big idea statisticians in future of Big Data Science. We describe the `United Statistical Algorithms' framework for comprehensive unification of traditional and novel statistical methods for modeling Small Data and Big Data, especially mixed data (discrete, continuous).
This paper quantifies uncertainty in Data Shapley using statistical inference.
problem Uncertainty in data valuation due to dynamic data distribution.
method Established relationship with U-statistics and quantified uncertainty using statistical inference.
result Confidence intervals for Data Shapley estimations are provided.
Breiman discusses two statistical cultures, advocating for more research on 'before' and 'after' the black box.
problem Statistical modeling lacks exploration of processes before and after the 'black box'.
method Analyzes Breiman's visual metaphor of two statistical cultures.
result Promotes the importance of studying the 'before' and 'after' of data transformations.
This paper connects functional data analysis with machine learning techniques.
problem Lack of theoretical analysis for functional depths.
method Viewing functional depths as kernel mean embeddings in machine learning.
result Facilitates answers to open questions about functional depths.
Enhances power of covariance matrix tests for high-dimensional data.
problem Testing large covariance matrices in high-dimensional data.
method Proposes a new Fisher's combined probability test for quadratic form and maximum form statistics.
result Boosts power against more general alternatives.
Enhances machine learning models by preserving data structure, addressing statistical distortions.
problem Statistical distortions in synthetic data generated by Mixup.
method Proposes a generalized mixup method with a flexible weighting scheme to preserve data structure.
result Preserves statistical properties of original data while maintaining model performance.
Paper proposes a statistical test for feature selection pipelines using selective inference.
problem Assessing the significance of feature selection pipelines in data analysis.
method Selective inference technique applied to feature selection pipelines composed of various algorithms.
result The proposed statistical test controls false positive feature selection probabilities.
Enhances U-statistics for semi-supervised datasets using unlabeled data.
problem Efficiently utilizing unlabeled data in semi-supervised settings.
method Semi-supervised U-statistics enhanced by unlabeled data.
result Proposed method is asymptotically Normal and more efficient than classical U-statistics.
Paper stabilizes persistent homology rank functions for statistical inference.
problem Stability issues in persistent homology rank functions.
method Derive stability results for rank functions under FDA metrics.
result Rank functions stabilize, improving statistical inference.
Estimates peeking effects in p-values to correct bias.
problem Data peeking biases reported p-values downward.
method Develops mechanisms to estimate running extrema of test statistics.
result Corrects bias in p-values due to peeking.
Raising statistical hurdles may not be justified due to data bias.
problem Data bias leads to unobserved results that weaken identification of revised hurdles.
method Theoretical and empirical analysis of statistical hurdles and data bias.
result Statistics targeting only published findings can be strongly identified.
Paper uses learned summary statistics for Bayesian inference with difficult likelihood functions.
problem Difficult to obtain exact likelihood function for observation data and simulation model.
method Simulation-based inference with learned summary statistics, using Cressie-Read discrepancy criterion.
result Effective inference performed over selected sample sets of observation data.
Study proposes a statistical testing framework for evaluating clustering pipelines.
problem Quantifying the statistical reliability of clustering results from data analysis pipelines.
method Selective inference-based statistical testing framework for clustering pipelines.
result The proposed test controls the type I error rate and is effective in validating clustering results.
Big Data bring new opportunities to modern society and challenges to data scientists. On one hand, Big Data hold great promises for discovering subtle population patterns and heterogeneities that are not possible with small-scale data. On the other hand, the massive sample size and high dimensionality of Big Data intro…
New estimator for symmetric kernel expectations, robust to missing data.
problem Efficient estimation of symmetric kernel expectations with missing data.
method Median-of-Incomplete-U-Statistics (MIU) estimator.
result Established finite-sample concentration rate for MIU.
Boosting improves data fitting while maintaining fairness guarantees.
problem Ensuring fairness in data preprocessing.
method Boosting algorithm to learn sufficient statistics of exponential families.
result The learned distribution maintains fairness guarantees while fitting the data better.
Enhances statistical inference using synthetic data.
problem Limited labeled data for statistical inference.
method GESPI framework that combines synthetic and real data.
result Error rate remains below a user-specified bound and decreases with synthetic data quality.
The paper evaluates variational auto-encoders using model criticism methods.
problem Evaluating the quality of variational auto-encoders (VAEs).
method Statistical model criticism, focusing on reproducing statistics of unknown data generating processes.
result The proposed framework offers possibilities for model selection beyond intrinsic metrics.
Paper improves robust spectral clustering for noisy data.
problem Noisy data and heavy-tailed entries hinder traditional clustering methods.
method Robust spectral clustering with rank statistics for latent structure recovery.
result Provable recovery of latent block structure in large data matrices.
Proposes a method to partition univariate data into unimodal subsets.
problem Partitioning univariate multimodal data into unimodal subsets.
method Recursive splitting around valley points of the data density using properties of critical points on the convex hull of the ecdf plot.
result Obtains a hierarchical statistical model of the initial dataset as a mixture of UMMs.
The paper addresses the relevance problem in statistical inference.
problem The relevance problem in statistical inference from large-scale data.
method Not specified in the abstract, likely involves statistical methods and analysis of large-scale data.
result The relevance problem is a long-neglected topic in statistical inference.
New method reconstructs data subsets from limited published statistics.
problem Reconstructing tabular data from aggregate statistics when full datasets are not possible.
method Generates and verifies subsets of rows and columns that are guaranteed to be correct.
result Privacy violations can persist even with sparse published statistics.
In this paper we address the problem of performing statistical inference for large scale data sets i.e., Big Data. The volume and dimensionality of the data may be so high that it cannot be processed or stored in a single computing node. We propose a scalable, statistically robust and computationally efficient bootstra…
New geometric methods improve optimization and data science problems.
problem Improving optimization and data science problems.
method Geometric tools for high-dimensional optimization and statistical data science.
result New algorithms and statistical guarantees for optimization and data science.
Paper reviews synthetic data from AI models for statistical inference.
problem When can synthetic data be used reliably in statistical inference?
method Survey of generative models, statistical analysis of pitfalls.
result Principled use of synthetic data requires careful model specification.
Unified platform for statistical and machine learning in bioinformatics.
problem Workflow inefficiencies in using multiple tools for data analysis.
method Automated hyperparameter optimization, feature importance analysis, statistical tests.
result Accelerates biological discovery workflows with methodological soundness.
New method improves statistical inference using machine learning-imputed data.
problem Improving statistical inference with imputed data from machine learning.
method Two-phase sampling approach for Z-estimation with ML-imputed outcomes.
result Guaranteed efficiency matching or exceeding classical inference, regardless of prediction quality.
Modern technologies are generating ever-increasing amounts of data. Making use of these data requires methods that are both statistically sound and computationally efficient. Typically, the statistical and computational aspects are treated separately. In this paper, we propose an approach to entangle these two aspects …
For high dimensional data, some of the standard statistical techniques do not work well. So modification or further development of statistical methods are necessary. In this paper, we explore these modifications. We start with the important problem of estimating high dimensional covariance matrix. Then we explore some …
A quantum circuit designed for efficient statistical model preparation and training.
problem Challenges in preparing and learning statistical models on quantum processors.
method Utilizes the maximum entropy principle to design a statistics-informed parameterized quantum circuit (SI-PQC).
result Improves trainability and interpretability for learning quantum states and classical model parameters.
Twinning splits data into fast, statistically similar sets.
problem Creating statistically similar data splits for Big Data.
method Twinning is a method based on SPlit for fast, model-independent dataset splitting.
result Twinning is orders of magnitude faster than SPlit.
New method efficiently evaluates policies using trajectory data.
problem Statistically efficient policy evaluation with limited data.
method Trajectory-based approach for policy evaluation.
result Improved sample complexity for policy evaluation.
Hypothesis tests are a crucial statistical tool for data mining and are the workhorse of scientific research in many fields. Here we present a differentially private analogue of the classic Wilcoxon signed-rank hypothesis test, which is used when comparing sets of paired (e.g., before-and-after) data values. We present…
High-dimensional statistics advances in complex data domains.
problem Complex, rich datasets challenge traditional methods.
method Evolved to address sophisticated estimation and inference problems.
result Deepened connections with optimization, concentration, and information theory.
Adaptive data fusion boosts efficiency in multi-task optimization.
problem Multi-task non-smooth optimization in various fields.
method Adaptive data fusion approach leveraging commonalities among objectives.
result Significant improvements in sample efficiency with sharp statistical guarantees.
DPpack offers R tools for private data analysis and machine learning.
problem Ensuring privacy in statistical analysis and machine learning.
method Differential privacy mechanisms (Laplace, Gaussian, exponential).
result User-friendly implementation of privacy-preserving models.
This describes a statistical technique called "tonsuring" for exploratory data analysis in finance. Instead of rejecting "outlier" data that conflicts with the model, this strips out "inlier" data to get a clearer picture of how the market changes for larger moves.
New DP framework using data truncation for efficient estimation.
problem Differential privacy in unbounded data support.
method Data truncation, exponential family distributions, maximum likelihood estimation, DP stochastic gradient descent.
result Near-optimal sample complexity for Gaussian mean and covariance estimation.
Study on statistical inference for nonlinear stochastic approximation with Markovian data.
problem Statistical inference for nonlinear stochastic approximation algorithms with Markovian data.
method Established a functional central limit theorem for the partial-sum process of the target parameter estimate, providing asymptotic pivotal statistics for constructing confidence intervals.
result Valid and efficient asymptotic inference method for nonlinear stochastic approximation algorithms with Markovian data.
Survey of robust clustering methods for hotspot detection.
problem Detecting false positives in spatial hotspot mapping.
method Statistically rigorous clustering techniques.
result Survey of models and algorithms for robust clustering.
RobPy offers robust statistical methods in Python.
problem Lack of robust statistical methods in Python.
method Built on NumPy, SciPy, and scikit-learn, RobPy includes robust tools for various statistical tasks.
result RobPy enables more users to perform robust data analysis in Python.
Ridge regularization simplifies model complexity in data science.
problem Overfitting in statistical models.
method Adding a penalty on the magnitude of coefficients.
result Effective in reducing model complexity and improving generalization.
Detecting the emergence of an abrupt change-point is a classic problem in statistics and machine learning. Kernel-based nonparametric statistics have been used for this task which enjoy fewer assumptions on the distributions than the parametric approach and can handle high-dimensional data. In this paper we focus on th…
Machine learning and statistical modeling complement each other in healthcare analytics.
problem Choosing between machine learning and statistical modeling for analytics challenges.
method Choosing based on problem, data, and desired outcomes.
result Machine learning and statistical modeling are complementary, using similar principles but different tools.
The scalability of statistical estimators is of increasing importance in modern applications. One approach to implementing scalable algorithms is to compress data into a low dimensional latent space using dimension reduction methods. In this paper we develop an approach for dimension reduction that exploits the assumpt…