Generates synthetic data for benchmarking unsupervised outlier detection.
problem Difficulty in benchmarking unsupervised outlier detection due to rare and varied outliers in real data.
method Proposes a generic process to generate synthetic data with insightful characteristics.
result Demonstrates practicality of the generic process through a benchmark with state-of-the-art detection methods.
In this paper, we propose a novel method for generating a synthetic dataset obeying Gaussian distribution. Compared to the commonly used benchmark datasets with unknown distribution, the synthetic dataset has an explicit distribution, i.e., Gaussian distribution. Meanwhile, it has the same characteristics as the benchm…
DIGEN benchmark provides synthetic datasets for ML algorithm evaluation.
problem Understanding and comparing machine learning algorithms' performance.
method Synthetic datasets generated using 40 mathematical functions to evaluate machine learning algorithms.
result DIGEN resource facilitates understanding why algorithms perform poorly and provides ideas for improvement.
A Python tool generates synthetic data for cluster analysis from high-level descriptions.
problem Creating synthetic data for cluster analysis is laborious and requires detailed geometric parameters.
method Proposes natural language-based synthetic data generation and implements it in a Python package.
result Makes it easy to set up interpretable and reproducible benchmarks for cluster analysis.
SurvHTE-Bench benchmarks HTE estimation in survival analysis with diverse datasets.
problem Challenges in estimating HTEs from right-censored survival data.
method Modular synthetic datasets, semi-synthetic datasets, and real-world datasets.
result First rigorous comparison of survival HTE methods under diverse conditions.
Paper benchmarks machine learning for detecting process curve drifts.
problem Detecting drifts in multivariate manufacturing process data.
method Synthetic data generation and evaluation score introduction.
result Existing algorithms often fail with complex drift scenarios.
TimeGraph creates synthetic datasets for robust time-series causal discovery.
problem Lack of reliable synthetic benchmark datasets for robust time-series causal discovery.
method Developed comprehensive synthetic datasets with temporal properties, including trends, seasonality, and noise.
result Demonstrated significant variations in algorithm performance under realistic temporal conditions.
XAI-Bench releases synthetic datasets for evaluating feature attribution methods.
problem Evaluating and comparing feature attribution methods is challenging.
method Released synthetic datasets and benchmarking library.
result Efficiently evaluates feature attribution methods across various metrics.
Method for creating synthetic multi-fidelity data sets.
problem Lack of representative synthetic datasets for multifidelity optimisation benchmarks.
method Systematic generation of synthetic fidelities from preexisting datasets.
result Allows systematic investigation of lower fidelity proxies' influence.
Scoping review and benchmarking of synthetic EHR data generation methods.
problem Creating realistic synthetic electronic health records for research and training.
method Conducted a scoping review and benchmarked seven methods on open-source EHR datasets.
result GAN-based methods excel in fidelity and utility, while rule-based methods excel in privacy protection.
Paper introduces TAP-Vid, a benchmark for tracking any point in videos.
problem Tackles the problem of tracking arbitrary physical points on surfaces over longer video clips.
method Formalizes the problem as TAP, introduces TAP-Vid benchmark, uses crowdsourced pipeline with optical flow estimates, proposes TAP-Net model.
result TAP-Net outperforms all prior methods on TAP-Vid benchmark when trained on synthetic data.
DSLOB creates synthetic LOB data for benchmarking forecasting algorithms under distributional shifts.
problem Challenges in dealing with out-of-distribution limit order book data.
method Multi-agent market simulator to create labeled synthetic LOB dataset with and without market stress.
result Demonstrates the need for robust forecasting algorithms to handle distributional shifts.
RealCause provides a realistic benchmark for causal inference.
problem Lack of a reliable benchmark for comparing causal effect estimators.
method Flexible generative models to create a benchmark that is both ground-truth and realistic.
result Evaluation of over 1500 causal estimators provides evidence for choosing hyperparameters using predictive metrics.
FinStressTS creates synthetic benchmarks for financial forecasting, revealing model weaknesses.
problem Limited failure attribution in real-world financial benchmarks.
method Synthetic benchmark with 30 diagnostic environments linked to six mechanism families.
result Model performance varies by mechanism type, with autoregressive models often outperforming Transformers.
The report evaluates heuristics for learning timescale graphical event models.
problem Lack of heuristics for determining hyper-parameters in timescale graphical event models.
method Proposed and evaluated different heuristics for hyper-parameter determination and refined an existing distance measure.
result Conclusions about the applicability of different heuristics on synthetic data.
CTBench benchmarks cryptocurrency time series generation for trading applications.
problem Lack of comprehensive benchmarks for cryptocurrency time series generation.
method Developed a comprehensive benchmark extsf{CTBench} with 13 metrics across 5 dimensions.
result Uncovered trade-offs between statistical fidelity and real-world profitability.
KT models improved slightly with synthetic student data.
problem Limited access to real student data and lack of diversity in public datasets.
method Simulated student data using three statistical strategies and tested on KT baselines.
result Synthetic data can lead to similar performance as real data.
TempoPFN models for zero-shot time series forecasting using synthetic data.
problem Efficient long-horizon prediction and reproducibility in zero-shot time series forecasting.
method Linear RNNs pre-trained on synthetic data with GatedDeltaProduct architecture and state-weaving.
result Achieves top-tier competitive performance on various benchmarks.
SMEs provide a transparent testbed for RL evaluation.
problem Lack of precise, white-box diagnostics in RL environments.
method Synthetic Monitoring Environments (SMEs) with fully configurable task characteristics and known optimal policies.
result SMEs allow for precise evaluation of RL algorithms, revealing the impact of specific environmental properties.
Performing controlled experiments on noisy data is essential in understanding deep learning across noise levels. Due to the lack of suitable datasets, previous research has only examined deep learning on controlled synthetic label noise, and real-world label noise has never been studied in a controlled setting. This pa…
MNIST-Nd offers synthetic datasets to benchmark clustering across dimensions.
problem Clustering performance degrades with high-dimensional data.
method Training mixture variational autoencoders on MNIST to create synthetic datasets with varying latent dimensions.
result Leiden clustering algorithm is most robust as dimensionality grows.
New benchmarks improve model performance by accounting for isomorphism classes in multi-relational datasets.
problem Synthetic multi-relational datasets lack isomorphism class awareness, leading to overestimation of model performance.
method Proposed isomorphism-aware synthetic benchmarks and a prioritisation scheme to improve model performance and stability.
result Isomorphism classes can be utilised to improve model performance, stability during training, and reduce training time.
Foundation models leak sensitive data in synthetic tabular data generation, especially LLaMA 3.3 70B.
problem Privacy leakage in synthetic tabular data generation using foundation models.
method Benchmarked three foundation models (GPT-4o-mini, LLaMA 3.3 70B, TabPFN v2) against four baselines on 35 real-world tables.
result Foundation models, especially LLaMA 3.3 70B, have the highest privacy risk in synthetic tabular data generation.
DPDMs use DP-SGD to generate private synthetic data.
problem Generating private synthetic data from sensitive datasets.
method Introduced DP-SGD for DMs, investigated DM parameterization and sampling, proposed noise multiplicity.
result Achieved state-of-the-art performance in image generation benchmarks.
Mamba improves time series forecasting by quantifying uncertainty.
problem Mamba forecasts have high mean errors in benchmarks.
method Dual-network framework for probabilistic forecasting.
result Predictive uncertainty reduced significantly for both synthetic and real-world data.
Generates synthetic manufacturing data for causal discovery benchmarking.
problem Lack of suitable real data for validating causal discovery algorithms.
method Distributional random forests for estimating conditional distributions.
result Semisynthetic manufacturing data adheres to a causal model.
New benchmark for causal reasoning from human video descriptions.
problem Lack of diversity in event types and natural language descriptions, and differences from human judgments.
method Iterative event cloze task and data augmentation techniques.
result Improved data collection efficiency and diverse causal judgments.
This study evaluates subgroup analysis methods for time-to-event outcomes in randomized controlled trials.
problem Identifying subgroups of good responders in non-significant randomized controlled trials.
method Evaluation of several subgroup analysis algorithms for time-to-event outcomes using synthetic and semi-synthetic data.
result Provides a new synthetic and semi-synthetic data generation process and an open-source Python package for benchmarking.
MixBoost generates synthetic instances to balance imbalanced datasets.
problem Training models on imbalanced datasets.
method Iterative data augmentation method that selects and combines instances from majority and minority classes.
result MixBoost outperforms existing approaches on 20 benchmark datasets.
Hybrid LLM generates synthetic data preserving causal parameters.
problem Synthetic data fails to accurately estimate causal effects.
method Combines model-based covariate synthesis with separately learned propensity and outcome models.
result Hybrid framework ensures causal structure in synthetic data.
Itô maps provide a method for any-step SDE integration.
problem Stochastic dynamics
method Itô map formulation
result Empirical results on synthetic and image-generation benchmarks
Paper introduces OARF benchmark suite for federated learning systems.
problem Limited diversity in federated learning benchmarks.
method Characterizes OARF benchmark suite with diverse data and applications.
result Federated learning can effectively increase end-to-end throughput.
Efficient synthetic data generation improves model performance on tabular data.
problem Improving model robustness and performance with scarce or low-quality data.
method Hardness characterization to identify high-value training points, generating synthetic data only from these points.
result Synthetic data generated from hardest points outperforms non-targeted methods on tabular datasets.
HalluWorld benchmarks hallucinations in language models across diverse tasks.
problem Inconsistent benchmarks of hallucination in language models.
method Explicit reference-world formulation, synthetic and semi-synthetic environments, controlled variation of world complexity.
result Models struggle with perceptual hallucination but not multi-step state tracking.
SynthBH uses synthetic data to control FDR in multiple testing.
problem Controlling false discovery rate in multiple hypothesis testing.
method SynthBH, a synthetic-powered multiple testing procedure.
result SynthBH guarantees FDR control with synthetic data.
Generative Adversarial Networks (GANs) have made releasing of synthetic images a viable approach to share data without releasing the original dataset. It has been shown that such synthetic data can be used for a variety of downstream tasks such as training classifiers that would otherwise require the original dataset t…
Improved Compressed Sensing by optimizing sparse solutions with mixed integer programming.
problem Finding sparse solutions to linear measurements with numerical tolerance.
method Introducing an ℓ2 regularized formulation, reformulating as a mixed integer second order cone program, deriving a second order cone relaxation, and developing a custom branch-and-bound algorithm. result Our approach produces solutions that are on average 6.22% more sparse compared to state-of-the-art methods.
CausalTime generates realistic time-series for TSCD evaluation.
problem Lack of realistic synthetic datasets for TSCD performance evaluation.
method Harnessing deep neural networks and normalizing flow for dynamics, extracting causal graphs, and deriving ground truth causal graphs.
result Generated datasets accurately reflect real data and ground truth causal graphs.
Python package for estimating intrinsic dimensionality of datasets.
problem Estimating intrinsic dimensionality for machine learning applications.
method Implementation of various intrinsic dimension estimators in Python.
result Benchmarking of ID estimation methods on real and synthetic data.
Graph learning is often unnecessary for common benchmarks, as node features can suffice.
problem The necessity of graph learning in common graph benchmarks is often assumed.
method We compared graph learning to feature-only models on seven datasets and found that graph structure often adds little to performance.
result Node features can often suffice for common graph benchmarks, challenging the orthodoxy.
A framework combines unsupervised and semi-supervised AD using synthetic anomalies.
problem Improving anomaly detection in both unsupervised and semi-supervised settings.
method Proposes a new framework that uses both known and synthetic anomalies for training.
result Synthetic anomalies improve anomaly modeling in low-density regions and provide optimal convergence guarantees.
We study the role of latent space dimensionality in Wasserstein auto-encoders (WAEs). Through experimentation on synthetic and real datasets, we argue that random encoders should be preferred over deterministic encoders. We highlight the potential of WAEs for representation learning with promising results on a benchmar…
Benchmark for math reasoning models from human proofs.
problem Measuring and accelerating machine learning models in high-level mathematical reasoning.
method Built a non-synthetic dataset from theorem prover proofs, defined a task for model to fill in missing propositions, used hierarchical transformer to improve performance.
result Neural models can capture non-trivial mathematical reasoning, hierarchical transformer outperforms baseline.
The Boolean Satisfiability (SAT) problem is the canonical NP-complete problem and is fundamental to computer science, with a wide array of applications in planning, verification, and theorem proving. Developing and evaluating practical SAT solvers relies on extensive empirical testing on a set of real-world benchmark f…
Study real-world noisy labels from human annotations for better understanding.
problem Understanding and modeling real-world label noise in machine learning.
method Developed two new benchmark datasets (CIFAR-10N, CIFAR-100N) with human-annotated real-world noisy labels.
result Real-world noisy labels exhibit instance-dependent patterns, not class-dependent as previously assumed.
We consider accurately answering smooth queries while preserving differential privacy. A query is said to be K-smooth if it is specified by a function defined on [−1,1]d whose partial derivatives up to order K are all bounded. We develop an ε-differentially private mechanism for the class of K-smooth queries…
This paper uses deep generative models to create synthetic financial data for portfolio and risk modeling.
problem Challenges in empirical research due to privacy, accessibility, and reproducibility issues in financial data.
method Investigates the use of Time-series Generative Adversarial Networks (TimeGAN) and Variational Autoencoders (VAEs) to generate synthetic financial return series.
result Synthetic data from TimeGAN closely mimics real financial data in distributional shapes, volatility, and autocorrelation.
New algorithms for clustering and synthetic data generation of heterogeneous tabular datasets.
problem Clustering and generating synthetic data from heterogeneous tabular datasets with hidden cluster structure.
method Developed MMM and MMMsynth algorithms for clustering and synthetic data generation.
result MMMsynth algorithm outperforms other literature tabular-data generators and approaches real data performance.