Method detects multi-timescale consumer spending patterns from receipts.
problem Understanding and managing consumer behavior in high-dimensional data.
method Non-negative tensor factorization (NTF) to extract multi-timescale expenditure patterns.
result Consumption patterns are characterized based on spending behavior over different timescales.
In this paper, we provide efficient estimators and honest confidence bands for a variety of treatment effects including local average (LATE) and local quantile treatment effects (LQTE) in data-rich environments. We can handle very many control variables, endogenous receipt of treatment, heterogeneous treatment effects,…
Paper creates transparent, safe synthetic data from coarsened margins.
problem Creating synthetic data that maintains original relationships and is safe from disclosure.
method Defining and curating margins, applying SDC, coarsening counts, and using IPF algorithm.
result Synthetic data derived from safe, coarsened margins maintains original relationships.
StatFEM uses low-rank approximations to scale Bayesian statFEM for high-dimensional problems.
problem Model misspecification and scalability in high-dimensional physical systems.
method Low-rank approximation of covariance matrix, Bayesian filtering, sparse data reconstruction.
result Reconstructs sparsely observed data-generating processes with minimal loss of information.
This paper provides a link between causal inference and machine learning techniques - specifically, Classification and Regression Trees (CART) - in observational studies where the receipt of the treatment is not randomized, but the assignment to the treatment can be assumed to be randomized (irregular assignment mechan…
American Depositary Receipts (ADRs) are exchange-traded certificates that rep- resent shares of non-U.S. company securities. They are major financial instruments for investing in foreign companies. Focusing on Asian ADRs in the context of asyn- chronous markets, we present methodologies and results of empirical analysi…
An integrated and extendable approach for stress-testing loan portfolios
problem Stress-testing loan portfolios
method Simulate completed portfolios, generate uncertain cash flow history, compute credit risk metrics
result Enhanced stress-testing practices within any bank
A new method for clearing liability networks using sheaves on directed hypergraphs.
problem Clearing in liability networks using a novel mathematical approach.
method Associate a liability sheaf on a directed hypergraph to a liability network, identifying clearing configurations as global sections of this sheaf.
result Clearing configurations are precisely the global sections of the sheaf, and the sheaf construction is functorial under change of coefficient category.
LineMVGNN improves AML detection by integrating multi-view graph learning.
problem Ineffective and scalable AML systems using rule-based methods.
method LineMVGNN combines multi-view graph neural networks with line-graph features.
result LineMVGNN outperforms state-of-the-art methods in detecting money laundering.
The paper proves an obstruction to immersed curves with specific curvature.
problem Existence of immersed curves with prescribed curvature.
method Proofs and corrections of submitted manuscript.
result An obstruction to the existence of immersed curves of prescribed curvature.
Improved IV estimates by weighting on compliance reduces noise in treatment effect estimation.
problem Noisy IV estimates in settings with non-random treatment receipt.
method Weighting observations by estimated compliance, leveraging machine learning for compliance estimation.
result Compliance weighting reduces IV variance, improving precision of treatment effect estimates.
Big data sets must be carefully partitioned into statistically similar data subsets that can be used as representative samples for big data analysis tasks. In this paper, we propose the random sample partition (RSP) data model to represent a big data set as a set of non-overlapping data subsets, called RSP data blocks,…
Prevents sensitive data generation in diffusion models using labeled and unlabeled data.
problem Generating sensitive data in diffusion models using unlabeled data.
method Positive-Unlabeled Diffusion Models, approximating ELBO with labeled and unlabeled data.
result Prevents the generation of sensitive data without compromising image quality.
Study reveals Data Shapley's inconsistent performance in data selection tasks.
problem Inconsistency of Data Shapley's performance in data selection across different settings.
method Hypothesis testing framework and identification of utility functions.
result Data Shapley's performance is no better than random selection without specific constraints.
PRRO generates synthetic tabular data that improves SL performance and class distribution.
problem Low SL utility of synthetic data due to class imbalance and overlooked data relationships.
method Data pruning and column reordering to optimize SL utility.
result Synthetic data generated with PRRO enhances predictive performance and class distribution.
Defines data science as a natural ecosystem with challenges and missions.
problem Challenges and missions in data science due to 5D complexities and data life cycle phases.
method Systemic and data-centric view of data science as a fusion of data universe and its challenges, formalizing a general-purpose architecture.
result Essential data science as a natural ecosystem integrating specific disciplines and high-impact applications.
Differences in data size per class, also known as imbalanced data distribution, have become a common problem affecting data quality. Big Data scenarios pose a new challenge to traditional imbalanced classification algorithms, since they are not prepared to work with such amount of data. Split data strategies and lack o…
Synthetic data enhances analytics but requires careful volume management.
problem Accuracy of statistical methods on synthetic data vs. raw data.
method Synthetic Data Generation for Analytics framework using tabular diffusion models.
result Error rate decreases with more synthetic data but may stabilize or increase.
New test ensures quality of shared data in machine learning.
problem Ensuring quality of external data in machine learning tasks.
method Distribution-free two-sample testing procedures grounded in conformal outlier detection.
result Identifies valuable external data agents for model personalization.
Differential privacy allows quantifying privacy loss resulting from accessing sensitive personal data. Repeated accesses to underlying data incur increasing loss. Releasing data as privacy-preserving synthetic data would avoid this limitation, but would leave open the problem of designing what kind of synthetic data. W…
Paper creates fair synthetic data ensuring equal predictions across sensitive attributes.
problem Ensuring fair predictions across sensitive attributes in synthetic data.
method Equalizing target probability distributions across sensitive attributes in synthetic data generation.
result Synthetic data provides strong fair predictions, equal across all thresholds.
DPA preserves data distribution in reduced dimensions.
problem Loss of data distribution in dimension reduction.
method DPA combines encoder and decoder to match data distribution.
result DPA successfully reconstructs data distribution.
Efficient synthetic data generation improves model performance on tabular data.
problem Improving model robustness and performance with scarce or low-quality data.
method Hardness characterization to identify high-value training points, generating synthetic data only from these points.
result Synthetic data generated from hardest points outperforms non-targeted methods on tabular datasets.
For most problems in science and engineering we can obtain data sets that describe the observed system from various perspectives and record the behavior of its individual components. Heterogeneous data sets can be collectively mined by data fusion. Fusion can focus on a specific target relation and exploit directly ass…
DAERNN models censored data using neural networks with data augmentation.
problem Handling censored data in expectile regression.
method Data augmentation based Expectile Regression Neural Networks (ERNNs).
result DAERNN outperforms existing censored ERNNs methods and achieves comparable predictive performance to fully observed data.
Data preprocessing techniques are devoted to correct or alleviate errors in data. Discretization and feature selection are two of the most extended data preprocessing techniques. Although we can find many proposals for static Big Data preprocessing, there is little research devoted to the continuous Big Data problem. A…
Data stream classification methods demonstrate promising performance on a single data stream by exploring the cohesion in the data stream. However, multiple data streams that involve several correlated data streams are common in many practical scenarios, which can be viewed as multi-task data streams. Instead of handli…
Data collection is a major bottleneck in machine learning and an active research topic in multiple communities. There are largely two reasons data collection has recently become a critical issue. First, as machine learning is becoming more widely-used, we are seeing new applications that do not necessarily have enough …
This paper quantifies uncertainty in Data Shapley using statistical inference.
problem Uncertainty in data valuation due to dynamic data distribution.
method Established relationship with U-statistics and quantified uncertainty using statistical inference.
result Confidence intervals for Data Shapley estimations are provided.
DCoM uses deep neural networks to detect semantic data types from raw column values.
problem Detecting semantic data types from dirty and unseen data.
method DCoM employs multi-input NLP-based deep neural networks trained on 686,765 data columns.
result DCoM outperforms existing methods significantly on 78 different semantic data types.
Causal inference from observational data is the goal of many data analyses in the health and social sciences. However, academic statistics has often frowned upon data analyses with a causal objective. The introduction of the term "data science" provides a historic opportunity to redefine data analysis in such a way tha…
Task-agnostic data valuation without validation requirements.
problem Valuing data without specific task assumptions.
method Estimating data diversity and relevance through queries without raw data.
result Estimates capture the diversity and relevance of seller's data for the buyer.
Data mining is about obtaining new knowledge from existing datasets. However, the data in the existing datasets can be scattered, noisy, and even incomplete. Although lots of effort is spent on developing or fine-tuning data mining models to make them more robust to the noise of the input data, their qualities still st…
New algorithm improves data imputation for complex multimodal data sets.
problem Artifacts in imputation methods for multimodal distributions.
method Combines kNN and KDE for probabilistic estimates. result Lower imputation errors and higher likelihood estimates.
New algorithms for clustering and synthetic data generation of heterogeneous tabular datasets.
problem Clustering and generating synthetic data from heterogeneous tabular datasets with hidden cluster structure.
method Developed MMM and MMMsynth algorithms for clustering and synthetic data generation.
result MMMsynth algorithm outperforms other literature tabular-data generators and approaches real data performance.
WeMix improves data augmentation by correcting bias in deep learning.
problem Data augmentation's effectiveness is limited by data bias.
method Developed AugDrop and MixLoss algorithms to correct data bias.
result WeMix improves data augmentation performance through bias correction.
A new method reduces data valuation variance for more trustworthy data trading.
problem Data valuation and trustworthy data trading in algorithmic prediction.
method Variance reduced Shapley value estimation using stratified sampling.
result VRDS method reduces estimation variance and improves data marketplace development.
Adapts data analysis for growing data, improving generalization guarantees.
problem Challenges of overfitting and statistical validity in adaptive workflows with growing data.
method Generalizes adaptive analysis on dynamic data, incorporating time-varying empirical accuracy bounds and mechanisms.
result First generalization bounds for adaptive analysis on dynamic data, matching prior works' improvement over data splitting.
This work redefines data-centric AI by unifying categorical and cochain notions.
problem The need to rethink data notions for data-centric AI.
method Proposes unifying principles from categorical and cochain notions of data.
result Unified definition of data impacts machine learning development, implementation, and utilization.
SMOTE-DP enhances synthetic data privacy without sacrificing utility.
problem Balancing privacy and utility in synthetic data generation.
method Integrating SMOTE with differential privacy mechanisms.
result SMOTE-DP produces synthetic data that is both private and useful.
VSAE learns from missing heterogeneous data by modeling latent dependencies.
problem Learning from partially-observed heterogeneous data with missingness.
method Variational selective autoencoder (VSAE) models joint distribution of observed, unobserved, and missing data.
result VSAE improves over state-of-the-art models in data generation and imputation tasks.
Improves anomaly detection with contaminated unlabeled data.
problem Weakness in existing semi-supervised anomaly detection methods when unlabeled data contain anomalies.
method Integrates positive-unlabeled learning with deep anomaly detection models.
result Achieves better detection performance on various datasets.
ArtificialReplay improves data efficiency in bandits using historical data.
problem Data inefficiency in warm-starting bandit algorithms.
method ArtificialReplay, a meta-algorithm for incorporating historical data into any bandit algorithm.
result ArtificialReplay uses only a fraction of historical data compared to a full warm-start approach, achieving identical regret.
Data stream clustering tackles real-time data processing challenges.
problem Real-time processing of data streams with less prior information.
method Review of data stream clustering algorithms and their characteristics.
result Comparison and analysis of data stream clustering algorithms.
Framework evaluates quality of synthetic data generated with differential privacy.
problem Ensuring synthetic data retains statistical quality after applying differential privacy.
method Developed a framework to evaluate synthetic data quality from a practical researcher's viewpoint.
result Synthetic data can be evaluated against training data or underlying populations, and for specific tasks like inference or prediction.
As data becomes the fuel driving technological and economic growth, a fundamental challenge is how to quantify the value of data in algorithmic predictions and decisions. For example, in healthcare and consumer markets, it has been suggested that individuals should be compensated for the data that they generate, but it…
Framework for pricing data products in data-poor markets.
problem Challenges in pricing advanced data products due to lack of transaction data.
method Prior-predictive Monte Carlo framework for generating probabilistic price bands.
result Stable probabilistic price bands for data products in data-poor markets.
Paper evaluates synthetic retail data for fidelity, utility, and privacy.
problem Ensuring accurate synthetic data in retail.
method Differentiates between continuous and discrete data, measures fidelity and utility, and uses Differential Privacy for privacy.
result Validated framework for reliable and scalable synthetic data evaluation.