DeepEvolution improves testing of deep neural networks by generating diverse test cases.
problem Lack of diversity in generated test cases from random fuzzing or transformations.
method Search-based approach using metaheuristics to ensure diversity in test cases.
result DeepEvolution significantly increases neuronal coverage and detects latent defects.
New methods test discrete distributions faster with local privacy constraints.
problem Testing discrete distributions under local differential privacy constraints.
method Efficient randomized algorithms and test procedures, both non-interactive and interactive.
result Faster separation rates in interactive privacy mechanisms.
Robust hypothesis testing designs a test for worst-case distributions using kernel methods.
problem Design a robust test for hypothesis testing under uncertainty sets.
method Data-driven uncertainty sets constructed using kernel mean embeddings and maximum mean discrepancy (MMD). Bayesian and Neyman-Pearson settings investigated.
result Proposed robust kernel tests are exponentially consistent and asymptotically optimal.
Research proposes a test case generation system for deep learning models using dataset properties.
problem Automated generation of extensive test cases for deep learning models is challenging.
method Measures dataset quality and proposes a test case generation system guided by dataset properties.
result Systematic test case generation for deep learning models is effective.
Proposes provenance and pseudo-provenance for automated test generation.
problem Invalidation of provenance in generated tests.
method Annotation of generated tests with provenance trails and pseudo-provenance.
result Validates the reliability of generated tests and their relation to seeds.
Agent-based model compares different COVID-19 testing policies and their effectiveness.
problem Understanding how different testing policies reveal the true number of infected cases.
method Developed an agent-based simulation framework in Python to model various testing policies and interventions.
result Contact Tracing consistently captures more positive cases than Random Symptomatic Testing, and LBT performs similarly.
Statistical tests for fairness in admissions data reveal hidden patterns.
problem Simpson's paradox in admissions data hides true gender bias.
method Introduces a new statistical test based on Pearl's instrumental-variable inequalities.
result Statistical tests for fairness coincide with causal notions for the Berkeley admissions case.
This paper proposes a method to generate realistic test cases for image classifiers.
problem Ensuring neural networks for image classification are correct with adequate realistic test data.
method Captures patterns in a large input data space using a manifold, then generates fault-revealing test cases.
result Generates thousands of realistic yet fault-revealing test cases efficiently for well-trained models.
Automated vehicles need efficient testing, which this method addresses.
problem Current testing practices are insufficient for automated vehicles.
method Gaussian Process Classification to identify performance boundaries.
result It is feasible to test for challenging scenarios more efficiently.
Automated testing improves deep learning model accuracy by 259.2%.
problem Ensuring robustness of deep learning models through automated testing.
method Jointly optimizing differential behaviors and neuron coverage; generating corner-cases; applying transformations and GANs.
result Deep learning model accuracy increased by 259.2% using automated generated corner cases.
Paper proposes a method to efficiently test rare vehicle failures using kernel methods.
problem Efficiently test rarely occurring but critical vehicle failures.
method Uses kernel methods to approximate and construct sampling distributions for rare event sets.
result Proposed method robustly identifies rare vehicle failures and significantly reduces evaluation time.
A new test compares latent variable models with kernel methods.
problem Comparing latent variable models with intractable marginal distributions.
method Kernel Stein Test, calibrated threshold, low-dimensional latent structure exploitation.
result Significantly outperforms Maximum Mean Discrepancy test in certain latent structure cases.
We propose a class of nonparametric two-sample tests with a cost linear in the sample size. Two tests are given, both based on an ensemble of distances between analytic functions representing each of the distributions. The first test uses smoothed empirical characteristic functions to represent the distributions, the s…
New insights into CI tests reveal key factors for practical performance.
problem Understanding and improving CI tests in practical applications.
method Investigation of the Kernel-based Conditional Independence (KCI) test and analysis of its practical behavior.
result Errors in conditional mean embedding estimates and appropriate conditioning kernel selection are crucial for CI tests.
Develops GLRT for defending against adversarial attacks in hypothesis testing.
problem Adversarial attacks on machine learning models causing misclassification.
method Generalized likelihood ratio test applied to composite hypothesis testing problem.
result GLRT approach yields competitive robustness-accuracy tradeoff under various attacks.
We propose a new algorithmic framework for sequential hypothesis testing with i.i.d. data, which includes A/B testing, nonparametric two-sample testing, and independence testing as special cases. It is novel in several ways: (a) it takes linear time and constant space to compute on the fly, (b) it has the same power gu…
Study on continuous sequence classification with distribution uncertainty.
problem Classifying continuous sequences with varying distribution uncertainty.
method Proposes distribution-free tests for three test designs: fixed-length, sequential, and two-phase tests.
result Error probabilities decay exponentially fast for all test designs.
Private two-sample tests under LDP achieve minimax rates for multinomial and continuous data.
problem Achieving statistical utility while maintaining privacy in two-sample testing.
method Private permutation tests for multinomial data and adaptive tests for continuous data.
result Minimax optimal tests for private two-sample testing under LDP.
optHSIC tests independence between covariates and censored lifetimes using optimal transport.
problem Testing independence between a covariate and right-censored lifetimes.
method optHSIC uses optimal transport to transform censored data into uncensored data, then applies a permutation test with a kernel-based dependence measure.
result optHSIC has power against a wider class of alternatives than Cox regression and maintains type 1 error control even when censoring depends on the covariate.
Discusses MultiFIT for multivariate dependence, comparing it to HSIC tests.
problem Comparing Multiscale Fisher's Independence Test (MultiFIT) to HSIC tests for multivariate dependence.
method Compares MultiFIT to HSIC tests, highlighting exact level control and performance limitations.
result Observes performance limitations of MultiFIT in terms of test power.
Automated test model creation from semi-structured requirements.
problem Lack of automated solution for test model creation from requirements.
method Machine Learning for semi-structured requirement detection and rule-based translation.
result 86% time savings with no loss of quality.
There has been an increasing interest in testing the equality of large Pearson's correlation matrices. However, in many applications it is more important to test the equality of large rank-based correlation matrices since they are more robust to outliers and nonlinearity. Unlike the Pearson's case, testing the equality…
A new method for testing deep learning models using CGF.
problem Traditional testing methods fail to cover corner cases in DNNs.
method Monte Carlo Tree Search for coverage-guided search.
result Generated inputs result in higher coverage than previous methods.
Building machine translation (MT) test sets is a relatively expensive task. As MT becomes increasingly desired for more and more language pairs and more and more domains, it becomes necessary to build test sets for each case. In this paper, we investigate using Amazon's Mechanical Turk (MTurk) to make MT test sets chea…
The study examines property testing and estimation under non-identically distributed samples, finding necessary and sufficient sample complexities.
problem Property testing and estimation under non-identically distributed samples.
method Analysis of distributional property testing and estimation in settings with heterogeneous entities.
result Necessary and sufficient sample complexities for property testing and estimation under non-identically distributed samples.
The paper develops tests for variable selection using LARS in high dimensions.
problem Variable selection and multiple testing in high-dimensional settings.
method Least Angle Regression (LARS) and post-selection joint law of knots.
result Exact non-asymptotic level testing procedures for variable selection.
For smooth test configurations, there always exist C^{1,1} geodesic rays in Kahler metric space parallel to the algebraic ray. The ¥ invariant agrees with Futaki invariant, at least under nice assumptions. Explicit examples in Toric cases are calculated. On simple test configurations, Donaldson's correspondence be…
Optimal decision-making over sensor networks with and without central coordination.
problem Sequential and adaptive hypothesis testing over sensor networks.
method Proposes sequential and adaptive hypothesis tests extending the classic Chernoff test in both centralized and fully distributed network settings.
result Achieves asymptotic optimality in both centralized and distributed network settings, minimizing risk and decision time.
We introduce kernel nonparametric tests for Lancaster three-variable interaction and for total independence, using embeddings of signed measures into a reproducing kernel Hilbert space. The resulting test statistics are straightforward to compute, and are used in powerful interaction tests, which are consistent against…
CALLISTO generates tests and assesses ML data quality using prediction entropy.
problem Validating ML systems for accuracy and data quality.
method Entropy-based test generation and data quality assessment framework.
result CALLISTO detects up to 20x more errors than traditional methods.
New algorithm tests Markov chains without hitting.
problem Testing Markov chains with unknown transition matrix.
method Combining approximation algorithms and spectral analysis.
result Efficient testing of Markov chains without hitting time dependence.
Study on hypothesis testing for densities and multinomials, showing local minimax rates and critical radii.
problem Testing goodness-of-fit for distributions with varying number of categories or unbounded support.
method Developed novel tests for both discrete and continuous cases, considering local minimax rates and critical radii.
result Characterized the dependence of critical radii on the null hypothesis and provided adaptive tests.
Conditional independence testing is an important problem, especially in Bayesian network learning and causal discovery. Due to the curse of dimensionality, testing for conditional independence of continuous variables is particularly challenging. We propose a Kernel-based Conditional Independence test (KCI-test), by con…
We review the main "omnibus procedures" for goodness-of-fit testing for copulas: tests based on the empirical copula process, on probability integral transformations, on Kendall's dependence function, etc, and some corresponding reductions of dimension techniques. The problems of finding asymptotic distribution-free te…
New graph tests improve on existing methods for comparing large graphs.
problem Comparing large graphs from different sources.
method Proposed new tests based on asymptotic distributions.
result New tests are computationally less expensive and more reliable.
The upsilon distribution, the sum of independent chi random variates and a normal, is introduced. As a special case, the upsilon distribution includes Lecoutre's lambda-prime distribution. The upsilon distribution finds application in Frequentist inference on the Sharpe ratio, including hypothesis tests on independent …
Optimal testing for densities under local differential privacy constraints.
problem Testing goodness-of-fit for densities under privacy constraints.
method Estimation of quadratic distance and minimax separation rates.
result First minimax optimal test under local differential privacy constraints.
New algorithms test independence with fewer samples by using predictive information.
problem Testing independence of distributions with limited samples.
method Augmented distribution testing framework that incorporates predictive information.
result Optimal sample complexity achieved, matching lower bounds.
New method tests mixtures of distributions with fewer samples than previously thought.
problem Testing if a distribution is a mixture of known distributions.
method Noise model where the noisy distribution is a mixture of the original and known noise.
result Sample complexity is the same as for non-mixture cases.
Study uses Shapley value for sensor anomaly detection, proving its superiority in certain cases.
problem Anomaly detection in sensor networks using the Shapley value.
method Optimized binary classifiers, mathematical proofs for different scenarios.
result Shapley value test can be superior or inferior to single-term tests, depending on sensor correlation and attack type.
Paper proposes a robust hypothesis testing method using Sinkhorn distance.
problem Hypothesis testing for small samples.
method Data-driven approach using Sinkhorn uncertainty sets.
result The method provides a more flexible detector compared to Wasserstein robust test.
Alternative hypothesis tests for class-conditional noise using local maximum likelihood.
problem Assessing label noise in supervised learning datasets.
method Proposes hypothesis tests based on local maximum likelihood estimation for nonparametric logistic regression.
result Shows improved applicability and flexibility of the proposed tests compared to parametric approaches.
The paper tests hypotheses on two Lévy process-driven streams of observations.
problem Testing hypotheses on two Lévy process-driven streams of observations.
method Infinitesimal generators and super/sub-solutions are used to compute bounds and analyze the model.
result Bounds for infinitesimal generators are computed in terms of super/sub-solutions.
Survey of recent methods for testing high-dimensional multinomial hypotheses.
problem Statistical power issues in high-dimensional multinomial testing.
method Review of recent methods focusing on asymptotic normality and minimax perspectives.
result Refined tests can have high power even when null distributions are non-normal.
New adaptive tests improve statistical dependence detection.
problem Testing statistical dependence between multivariate variables.
method Adaptive nonlinear monotonic transformations of distances.
result Empirical tests outperform existing methods.
In quantitative finance, we often fit a parametric semimartingale model to asset prices. To ensure our model is correct, we must then perform goodness-of-fit tests. In this paper, we give a new goodness-of-fit test for volatility-like processes, which is easily applied to a variety of semimartingale models. In each cas…
The properties of statistical tests for hypotheses concerning the parameters of the multifractal model of asset returns (MMAR) are investigated, using Monte Carlo techniques. We show that, in the presence of multifractality, conventional tests of long memory tend to over-reject the null hypothesis of no long memory. Ou…
A heuristic framework tests the multi-manifold hypothesis in empirical data.
problem Overestimation of parameters in global linear models.
method Heuristic multiscale framework using spline-interpolated manifolds.
result Validates the multi-manifold hypothesis in empirical data.