The paper develops a faster surrogate model for simulators using hybrid methods.
problem The need for faster validation of automotive technologies using simulators.
method Testing classical methods and building hybrid models combining them.
result A hybrid surrogate model outperforms classical methods in multivariate time series prediction.
Improved GNN simulation of WL test with exponentially lower complexity.
problem Improving the complexity of simulating the Weisfeiler-Lehman test with GNNs.
method Exponentially lower complexity simulation of WL test using GNNs with polylogarithmic parameters and O(log n) bits feature vectors.
result Near-optimal construction with logarithmic lower bounds for feature vector length and neural network size.
Recently, the binary expansion testing framework was introduced to test the independence of two continuous random variables by utilizing symmetry statistics that are complete sufficient statistics for dependence. We develop a new test based on an ensemble approach that uses the sum of squared symmetry statistics and di…
Financial Wind Tunnel generates versatile market data for model testing.
problem Inconsistent market dynamics across different scales and sources.
method Retrieval-augmented diffusion-based simulator integrating macro and micro patterns.
result Enhanced performance and adaptability of downstream models in complex markets.
fintech-kMC simulates financial platforms for AI/ML model validation.
problem Validation of AI/ML models in real-world financial applications.
method Agent-based model with kinetic Monte Carlo engine.
result Generates realistic synthetic data for testing AI/ML models.
Study evaluates ML methods for two-sample testing with right-censored data.
problem Evaluating ML methods for two-sample testing with right-censored data.
method Developed and compared several ML-based methods with classical tests.
result Proposed methods outperform classical tests in terms of statistical power.
A new algorithm improves efficiency and robustness of heuristic optimization in simulation-based problems.
problem Optimizing input parameters for stochastic simulation-based optimization.
method Reactive sample size algorithm based on parametric tests and indifference-zone selection.
result The reactive method improves efficiency and robustness of heuristic optimization techniques.
Agent-based model compares different COVID-19 testing policies and their effectiveness.
problem Understanding how different testing policies reveal the true number of infected cases.
method Developed an agent-based simulation framework in Python to model various testing policies and interventions.
result Contact Tracing consistently captures more positive cases than Random Symptomatic Testing, and LBT performs similarly.
A machine learning environment for detecting autonomous vehicle corner cases.
problem Testing autonomous driving software in the real world is difficult.
method Connecting CARLA simulation software to TensorFlow and custom AI client software.
result The system can identify situations where AI software fails to understand the scenario.
An integrated and extendable approach for stress-testing loan portfolios
problem Stress-testing loan portfolios
method Simulate completed portfolios, generate uncertain cash flow history, compute credit risk metrics
result Enhanced stress-testing practices within any bank
A new test assesses how well observed networks fit a specified ERGM model.
problem Testing the goodness of fit for ERGMs with a single network observation.
method Kernel Stein discrepancy combined with a discrete Stein operator for ERGMs, Monte Carlo simulation.
result The test provides theoretical and practical support for assessing ERGM fit.
Machine learning (especially reinforcement learning) methods for trading are increasingly reliant on simulation for agent training and testing. Furthermore, simulation is important for validation of hand-coded trading strategies and for testing hypotheses about market structure. A challenge, however, concerns the robus…
A martingale framework for concept change detection based on testing data exchangeability was recently proposed (Ho, 2005). In this paper, we describe the proposed change-detection test based on the Doob's Maximal Inequality and show that it is an approximation of the sequential probability ratio test (SPRT). The relat…
While autonomous vehicle (AV) technology has shown substantial progress, we still lack tools for rigorous and scalable testing. Real-world testing, the de-facto evaluation method, is dangerous to the public. Moreover, due to the rare nature of failures, billions of miles of driving are needed to statisticall…
PriorGuide adapts diffusion models to new priors at test time.
problem Limited applicability of prior distributions in diffusion-based inference.
method PriorGuide uses a guidance approximation to adapt diffusion models to new priors at test time.
result Enhances the versatility of pre-trained inference models by allowing flexible adaptation to new priors.
Develops privacy-preserving methods for equivalence testing in healthcare.
problem Protecting patient data in healthcare studies.
method Differential privacy techniques for simulation-based calibration.
result Maintains type-I error control and comparable power to non-private methods.
PRISM infers model structures and parameters from simulations, controlling complexity at test time.
problem Choosing among large model families for scientific discovery.
method Simulation-based encoder-decoder that infers model structures and parameters, with test-time complexity control.
result PRISM scales to large model families and performs model selection in biophysical diffusion MRI.
A method uses neural networks to approximate sampling distributions of test statistics.
problem Accurate modeling of p-value functions or cdfs for correct confidence set coverage.
method Uses neural networks to model the cdf of test statistics, approximating sampling distributions.
result Neural network approximations of sampling distributions are effective and simple.
New method finds failures in high-fidelity simulators with fewer steps.
problem Finding failures in high-fidelity simulators is expensive and impractical.
method Adaptive stress testing with backward algorithm adaptation from low-fidelity to high-fidelity.
result Significantly fewer high-fidelity simulation steps needed to find failures.
Paper develops a new test for high-dimensional matrix-valued data.
problem Hypothesis testing for mean of matrix-valued data in high-dimensional settings.
method Proposes a new test statistic for high-dimensional matrix rank testing.
result Develops a novel approach for sparse singular value decomposition (SVD) estimation.
Paper proposes method to calibrate market simulator for various scenarios.
problem Calibrate market simulator to represent different market conditions.
method Two-step method using GAN with self-attention to train discriminator and optimize simulator parameters.
result Demonstrates effectiveness of method in capturing various market scenarios.
Deep-learning method improves hypothesis testing for independence.
problem Improving hypothesis testing for independence using deep learning.
method Proposes deep-testing, a novel procedure that uses a deep neural network to distinguish between data generated under and outside a given statistical model.
result Deep-testing achieves the highest overall power against nineteen competing methods across various dependence structures.
Efficiently estimates marginal posteriors for complex simulations.
problem Bayesian inference in high-dimensional, intractable likelihood scenarios.
method Simulates and estimates low-dimensional marginal posteriors, using truncated indicators.
result Simulator efficiency and robustness testing of inference results.
We describe a post hoc test for the Sharpe ratio, analogous to Tukey's test for pairwise equality of means. The test can be applied after rejection of the hypothesis that all population Signal-Noise ratios are equal. The test is applicable under a simple correlation structure among asset returns. Simulations indicate t…
A new method for adaptive experiments improves inference.
problem Current inference methods for adaptive experiments are weak and asymptotic.
method Simulation-based inference using optimistic simulations.
result Our method achieves better coverage and narrower intervals.
Novel algorithm solves optimal transport using evolving probability distributions and convolution.
problem Sample-based optimal transport problem.
method Adversarial formulation with convolution of adaptive kernel and evolving measure.
result Algorithm robust to dimensionality and produces complex maps.
We proposed a novel graph convolutional neural network that could construct a coarse, sparse latent point cloud from a dense, raw point cloud. With a novel non-isotropic convolution operation defined on irregular geometries, the model then can reconstruct the original point cloud from this latent cloud with fine detail…
While recent developments in autonomous vehicle (AV) technology highlight substantial progress, we lack tools for rigorous and scalable testing. Real-world testing, the de facto evaluation environment, places the public in danger, and, due to the rare nature of accidents, will require billions of miles in or…
User Simulators are one of the major tools that enable offline training of task-oriented dialogue systems. For this task the Agenda-Based User Simulator (ABUS) is often used. The ABUS is based on hand-crafted rules and its output is in semantic form. Issues arise from both properties such as limited diversity and the i…
This paper provides a new unimodality test with application in hierarchical clustering methods. The proposed method denoted by signature test (Sigtest), transforms the data based on its statistics. The transformed data has much smaller variation compared to the original data and can be evaluated in a simple proposed un…
New deep learning methods improve estimation and GOF assessment for large-scale IFA.
problem Estimating and assessing goodness-of-fit for large-scale confirmatory IFA models.
method Extended deep learning algorithm for parameter estimation and simulation-based tests for GOF assessment.
result Proposed methods provide comparable estimates and detect latent dimensionality misspecification.
PDSim simulates and estimates commodity futures prices using polynomial diffusion models.
problem Simulating and estimating commodity futures prices using polynomial diffusion models.
method Developed an R package with a Shiny app for simulation and estimation of commodity futures prices using polynomial diffusion models.
result PDSim is the only package specifically designed for the simulation and estimation of the polynomial diffusion model.
We present a test platform for visual in-cabin scene analysis and occupant monitoring functions. The test platform is based on a driving simulator developed at the DFKI, consisting of a realistic in-cabin mock-up and a wide-angle projection system for a realistic driving experience. The platform has been equipped with …
The paper proposes a framework to calibrate multi-agent simulation models from output series using Bayesian optimization.
problem Calibrating multi-agent simulation models from observable output series.
method Novel eligibility set concept, two-sample Kolmogorov-Smirnov test with Bonferroni correction, Bayesian optimization (BO), and trust-region BO (TuRBO).
result Demonstrated the efficiency of the proposed framework using numerical experiments.
Generative Adversarial Networks simulate elevator group control without extensive data.
problem Lack of historical real-world data for system testing.
method Used GANs to generate simulation data for a multi-car elevator system.
result GANs can be used as substitutes for expensive simulation runs, but fine-tuning is needed.
Complex phenomena in engineering and the sciences are often modeled with computationally intensive feed-forward simulations for which a tractable analytic likelihood does not exist. In these cases, it is sometimes necessary to estimate an approximate likelihood or fit a fast emulator model for efficient statistical inf…
Study examines if LLMs' trading styles match real market behavior.
problem Lack of behavioral consistency in LLMs' trading strategies.
method Year-long simulations with LLMs, operationalizing behavioral finance drivers, and comparing with financial theory.
result LLMs' strategy switching is only partially consistent with behavioral finance theories.
Approach selects variables and time intervals for comparing high-dimensional time-series data.
problem Comparing high-dimensional time-series data for significant differences.
method Data is split into subintervals, and two-sample tests are performed on each to identify distinguishing variables.
result The approach effectively identifies variables and time intervals where data significantly differs.
Parameter estimation, statistical tests and confidence sets are the cornerstones of classical statistics that allow scientists to make inferences about the underlying process that generated the observed data. A key question is whether one can still construct hypothesis tests and confidence sets with proper coverage and…
Develops a method to simulate rare dangerous events in autonomous systems.
problem Rare dangerous events in safety-critical systems are hard to test in real-world settings.
method Combines exploration, exploitation, and optimization techniques for rare-event simulation.
result Provides rigorous guarantees for the performance of the method.
TSNPE improves SBI efficiency and scalability.
problem Efficient and scalable simulation-based inference for complex models.
method Sequential inference with truncated proposals.
result TSNPE performs on par with previous methods and scales to complex models.
We propose a test of independence of two multivariate random vectors, given a sample from the underlying population. Our approach, which we call MINT, is based on the estimation of mutual information, whose decomposition into joint and marginal entropies facilitates the use of recently-developed efficient entropy estim…
New deep learning method improves financial stress testing accuracy.
problem Traditional stress testing methods are criticized for unrealistic assumptions and estimation errors.
method Proposes a novel Deep Learning approach for Dynamic Balance Sheet Stress Testing.
result Empirical results show significant improvement in accuracy over traditional methods.
Proposes a method to simulate data for testing credit risk scorecard stability.
problem Ensuring credit risk scorecards remain representative of the population over time.
method Specification of bad ratios to generate parameter values for scorecards.
result Simulated data adheres closely to specified bad ratios.
Proposes a new method to improve Bayesian computation accuracy using flexible classification.
problem Bayesian computations accuracy check using rank-based simulation-based calibration has limitations.
method Replaces marginal rank test with a flexible classification approach that learns from data.
result Improves statistical power and provides an interpretable divergence measure of miscalibration.
3DB framework tests and debugs computer vision models using photorealistic simulation.
problem Discovering vulnerabilities and understanding model decision-making in computer vision systems.
method Unified framework using photorealistic simulation to test and debug vision models.
result Insights generated by 3DB transfer to the physical world, enabling robustness analysis.
Kernel-based test detects differences between two conditional distributions efficiently.
problem Detecting differences between two conditional distributions efficiently.
method Kernel-based measure using nearest-neighbor graphs, consistent estimate with Gaussian limit.
result Asymptotic level control and universal consistency for detecting differences.
We develop a pivotal test to assess the statistical significance of the feature variables in a single-layer feedforward neural network regression model. We propose a gradient-based test statistic and study its asymptotics using nonparametric techniques. Under technical conditions, the limiting distribution is given by …