Simulates neuropathic pain to evaluate causal discovery algorithms.
problem Lack of benchmark datasets for evaluating causal discovery algorithms.
method Developed a neuropathic pain diagnosis simulator.
result Simulator produces data similar to real-world data.
Paper uses machine learning to evaluate financial simulations.
problem Quantify the realism of simulated financial time series.
method Machine learning classification of simulated vs real financial data.
result Improved simulation methods through competition results.
Paper uses K-NN resampling to simulate and evaluate LOB markets.
problem Simulating and evaluating limit order book (LOB) markets.
method Applies K-nearest neighbor (K-NN) resampling to LOB simulation and evaluation. result Demonstrates the effectiveness and efficiency of K-NN resampling in LOB simulation and evaluation. Contextual bandit algorithms have become popular for online recommendation systems such as Digg, Yahoo! Buzz, and news recommendation in general. \emph{Offline} evaluation of the effectiveness of new algorithms in these applications is critical for protecting online user experiences but very challenging due to their "p…
User Simulators are one of the major tools that enable offline training of task-oriented dialogue systems. For this task the Agenda-Based User Simulator (ABUS) is often used. The ABUS is based on hand-crafted rules and its output is in semantic form. Issues arise from both properties such as limited diversity and the i…
Study reveals gaps between simulated and real-world treatment effect evaluation metrics.
problem Evaluation of treatment effect estimation models differs between academic and practical settings.
method Comprehensive empirical study comparing semi-simulated benchmarks and real-world datasets.
result Counterfactual metrics do not reliably predict observable metrics, and rankings from simulated benchmarks do not generalize to real-world data.
A multi-agent simulator evaluates trading strategies using Market Replay and Interactive Agent-Based Simulation.
problem Evaluate trading strategies using Market Replay and Interactive Agent-Based Simulation.
method Multi-agent simulator for Market Replay and Interactive Agent-Based Simulation.
result IABS provides a more realistic market environment for evaluating trading strategies.
This work introduces benchmarks for evaluating nanophotonic structures in design simulations.
problem Design and understanding of nanophotonic structures for various applications.
method Development of frameworks and benchmarks for evaluating nanophotonic structures in parametric design problems.
result Strategic use of evaluation fidelity in enhancing structure designs.
The paper shows how shared random seeds can reduce variance in machine learning evaluations.
problem The statistical structure of comparative evaluation under shared random seeds is not well understood.
method An extended learning-based multi-agent economic simulator was used to demonstrate the effects of shared random seeds on variance reduction.
result Pairing seeds can reduce variance in machine learning evaluations, especially when outcomes are positively correlated at the seed level.
This paper compares and evaluates methods for evaluating statistical models using benchmarking data and simulations.
problem Choosing between benchmarking data sets and simulation studies for method comparison studies.
method Borrowing ideas from mixed methods research and Clinical Scenario Evaluation, the paper investigates and develops new approaches to evaluate methods.
result Develops new approaches to evaluate methods by combining the strengths of benchmarking data sets and simulation studies.
New framework tests AVs as a black box, prioritizing rare failure modes.
problem Lack of rigorous and scalable testing methods for AVs.
method Developed a simulation testing framework that learns to identify and rank failure scenarios via adaptive importance-sampling methods.
result First independent evaluation of a full-stack commercial AV system (Comma AI's OpenPilot).
Simulation workflow is a top-level model for the design and control of simulation process. It connects multiple simulation components with time and interaction restrictions to form a complete simulation system. Before the construction and evaluation of the component models, the validation of upper-layer simulation work…
A computational theory reduces agent evaluation errors and speeds up processes.
problem Efficient evaluation of mini agents at reduced cost.
method Developed a computational theory and a meta-learner to handle heterogeneous agents.
result Reduced evaluation errors by 24.1% to 99.0% across various scenarios.
Develops a method to simulate rare dangerous events in autonomous systems.
problem Rare dangerous events in safety-critical systems are hard to test in real-world settings.
method Combines exploration, exploitation, and optimization techniques for rare-event simulation.
result Provides rigorous guarantees for the performance of the method.
This study benchmarks likelihood-free inference methods for models with heavy-tailed or discrete data.
problem Comparing likelihood-free inference methods for models with structural features like heavy-tails or discreteness.
method Four approaches: MLE, NBE, EOT, and AW-NBE are evaluated using simulations.
result The choice of evaluation tools is crucial for models with extremes and discrete data.
Deep-PrAE improves rare-event simulation for black-box systems.
problem Evaluating rare safety-critical events in learning-based systems.
method Combines deep neural networks with IS to create statistically guaranteed estimations.
result Deep-PrAE provides accurate bounds on safety-critical event probabilities.
Parallel Gaussian process surrogate for noisy likelihood evaluations in Bayesian inference.
problem Bayesian inference with limited noisy log-likelihood evaluations from complex models.
method Hierarchical Gaussian process surrogate model for log-likelihood, batch-sequential design strategies.
result Robust, highly parallelizable, and sample-efficient method.
A new neural network model simulates financial markets without assuming underlying dynamics.
problem Modeling financial time series without assuming underlying dynamics.
method Neural network based generative model using a parsimonious Variational Autoencoder framework.
result Works reliably in small data environments, providing a new performance evaluation metric.
Planning problems are among the most important and well-studied problems in artificial intelligence. They are most typically solved by tree search algorithms that simulate ahead into the future, evaluate future states, and back-up those evaluations to the root of a search tree. Among these algorithms, Monte-Carlo tree …
Improved flow-based inference speeds up and boosts accuracy for complex simulations.
problem Challenging inverse problems in astronomy, such as modeling strong gravitational lens systems.
method Refines flow-based generative models with simulator feedback for posterior inference.
result Improves accuracy by 53% and speeds up inference by up to 67x.
New method reduces uncertainty in AI-driven Monte Carlo simulations.
problem Epistemic uncertainty in AI surrogate models affects Monte Carlo sampling outcomes.
method Penalty Ensemble Method (PEM) modifies Metropolis acceptance rule to increase rejection probability in uncertain regions.
result PEM enhances reliability of Monte Carlo simulations by reducing uncertainty propagation.
This paper compares two clustering evaluation metrics, revealing their differences and properties.
problem Understanding the differences between misclassification error distance and adjusted Rand index.
method Population origins, data analysis examples, detailed case studies, and simulation study.
result Reveals previous misconceptions about the two metrics and their distributions.
The study improves GCN performance on friendship-based social networks by simulating and augmenting datasets.
problem GCNs' limitations in friendship-based social networks, especially dependency between node neighbourhood order and layer.
method Developed a Python library to simulate social networks with ground truth labels and features. Introduced sDNA as latent variables for nodes.
result Four new GCN variants significantly outperform the original model on 27 out of 30 simulated datasets.
Simulation speeds AV testing by 2-20 times over real-world methods.
problem Lack of scalable and rigorous testing for autonomous vehicles.
method Adaptive importance-sampling methods for rare-event probability evaluation.
result Accelerates accident probability estimation by 2-20 times over naive Monte Carlo methods.
Improves likelihood-free inference by using a new sampling approach to avoid biased data collection.
problem Efficient Bayesian inference without likelihood evaluation for real-world datasets.
method Introduces Neural Proposal (NP) to sample simulation inputs i.i.d. for unbiased posterior inference.
result Demonstrates improved performance, especially for multi-modal posteriors, through experiments.
The paper addresses GP dynamics by improving simulation and prediction accuracy.
problem GP dynamics often underestimate prediction uncertainty, leading to safety issues.
method The paper introduces sampling-based and linearization-based techniques to account for the correlation between successive function evaluations.
result The proposed methods provide more accurate trajectory distributions and prediction uncertainties.
COCA accelerates N-body simulations by correcting ML errors.
problem Computational expense and limited trustworthiness of ML emulations.
method Hybrid framework combining ML and N-body simulator in an emulated frame of reference. result COCA reduces emulation errors with fewer force evaluations.
Sparse grids reduce xVA exposure evaluations by up to 6000 times.
problem Efficiently computing exposures for xVA in large portfolios with many risk factors.
method Sparse Grid Method combined with Stochastic Collocation and Smolyak's extension.
result Significant reduction in the number of portfolio evaluations, up to 6000 times.
New method combines neural networks with Monte Carlo for complex system reliability.
problem Estimating small failure probabilities in complex systems.
method Subset Simulation with Hamiltonian Neural Networks.
result High acceptance rates and computational efficiency in low-probability regions.
Efficient surrogate model reduces ESM evaluation time.
problem Reducing computational time for large-scale ESM simulations.
method SVD for dimensionality reduction followed by Bayesian optimization for neural network surrogate.
result 20 ESM simulations build a 42660-variable surrogate with 93% consistency and 2% MSE.
This research improves calorimeter simulations by creating a faster model.
problem Efficiently simulating detailed calorimeter data for high-energy physics.
method Developed a conditional normalizing flow model for superresolution.
result The model successfully reproduces reference distributions.
SOCRATES uses LLMs to automate simulation optimization of complex systems.
problem Optimizing complex, expensive-to-sample stochastic systems.
method Two-stage procedure: replica construction and meta-optimization.
result Adaptive hybrid optimization schedule for real systems.
A framework for optimizing prompt selection in generative language models.
problem Efficiently selecting prompts for generative language models.
method Two-stage framework using simulation optimization to maximize a pre-defined score.
result Consistency of the sequential evaluation procedure in the proposed framework.
Proposes a method to evaluate generalizability in causal inference models.
problem Lack of formal procedures to statistically evaluate generalizability in causal inference.
method Frugal parameterization to simulate from causal benchmarks, using mean and distributional regression methods.
result Ensures more realistic evaluations of causal inference models, avoiding over-reliance on conventional metrics.
Accelerates Bayesian optimization of function networks with partial evaluations.
problem Optimizing expensive-to-evaluate function networks with varying node costs.
method Proposes an accelerated algorithm that uses global Monte Carlo simulations to select node-specific candidate inputs.
result Achieves up to a 16x speedup over the original p-KGFN algorithm while maintaining competitive query efficiency.
Study uses three sources to evaluate language models fairly.
problem Bias in offline model evaluation due to confounded model choice.
method Combines observational logs, randomized experiments, and simulators.
result Randomized experiment and simulator together recover causal model values.
Develops methods to create consistent surrogate models for agent-based simulators.
problem High computational costs and misjudgment of interventions in agent-based models.
method Causal abstractions to learn interventionally consistent surrogate models.
result Surrogates trained for interventional consistency closely mimic the agent-based model's behavior under interventions.
New method efficiently simulates fluid flows across various conditions.
problem High computational cost in simulating fluid flows.
method Parameter-conditioned sequential generative modeling of neural networks.
result Trained models simulate fluid flows at orders of magnitude faster than traditional methods.
In this paper we propose a modified version of the simulated annealing algorithm for solving a stochastic global optimization problem. More precisely, we address the problem of finding a global minimizer of a function with noisy evaluations. We provide a rate of convergence and its optimized parametrization to ensure a…
Proposes method for eliciting non-parametric joint priors using normalizing flows.
problem Learning complex non-parametric joint priors for model parameters.
method Expert elicitation combined with normalizing flows for generative modeling.
result Framework supports elicitation of both parametric and non-parametric priors.
Study compares VaR models and finds GARCH-FHS superior.
problem Comparing VaR models for accurate risk assessment.
method Historical Simulation, GARCH-N, GARCH-FHS models evaluated.
result GARCH-FHS provides superior performance in capturing tail risks.
A new method uses active learning to improve chemical simulation efficiency.
problem Efficiently estimating equilibrium-based chemical simulations.
method Sequential data-driven approach using Gaussian process uncertainty.
result Significantly reduced number of function evaluations.
We present GLASSES: Global optimisation with Look-Ahead through Stochastic Simulation and Expected-loss Search. The majority of global optimisation approaches in use are myopic, in only considering the impact of the next function value; the non-myopic approaches that do exist are able to consider only a handful of futu…
USeMOC framework reduces expensive simulations for MO optimization with constraints.
problem Efficiently optimizing multi-objective problems with constraints using expensive function evaluations.
method USeMOC framework uses surrogate models to identify promising candidates and selects the best based on uncertainty.
result USeMOC achieves more than 90% reduction in function evaluations for circuit optimization.
Generative Adversarial Network creates realistic halo merger trees.
problem Comparing galaxy formation theories with observations using halo merger trees.
method Treated halo merger tree construction as a matrix generation problem, using Generative Adversarial Network.
result Generated halo merger trees are of high quality and realistic.
A neural network approach to compute stable metrics for numerical simulation data.
problem Computing stable and generalizing metrics for diverse numerical simulation data.
method A Siamese neural network architecture with a specialized loss function trained on a controlled data generation setup.
result LSiM outperforms existing metrics for vector spaces and image-based metrics.
BOSH optimizes functions with stochastic evaluations more efficiently and precisely.
problem Optimizing functions with noisy evaluations can lead to suboptimal solutions.
method BOSH uses a hierarchical Gaussian process to generate a growing pool of realizations.
result BOSH provides more efficient and higher-precision optimization than standard BO.
New simulation approaches to evaluating path-dependent options without matrix inversion issues nor Euler bias are evaluated. They employ three main contributions: Stochastic approximation replaces regression in the LSM algorithm; Explicit weak solutions to stochastic differential equations are developed and applied to …