Unified framework for testing deep learning models with concept activation vectors.
problem Statistical instability and discontinuity in testing with concept activation vectors.
method Introducing α-TCAV, a generalized framework that replaces the indicator function with a parameterized smooth function.
result Unified probabilistic formulation that subsumes TCAV and Multi-TCAV, providing principled guidance on tuning the parameter.
The interpretation of deep learning models is a challenge due to their size, complexity, and often opaque internal state. In addition, many systems, such as image classifiers, operate on low-level features rather than high-level concepts. To address these challenges, we introduce Concept Activation Vectors (CAVs), whic…
Deep learning skin lesion classifier explained using CAVs.
problem Limited acceptance of deep learning CAD systems due to opaque decision-making.
method Mapped human understandable concepts to RECOD model using CAVs.
result Classifier learns and encodes disease-related concepts in its latent representation.
This paper analyzes the variability of Concept Activation Vectors (CAVs).
problem The variability of CAVs in explaining AI models.
method Theoretical analysis and experiments on real-life datasets to quantify CAVs variability.
result The variance of CAVs decreases as 1/N, where N is the number of random examples.
CAVs reveal latent concept distributions, but are vulnerable to adversarial attacks.
problem Understanding latent concept encodings in AI models.
method Probabilistic perspective on CAVs, deriving mean and covariance.
result CAVs can be adversarially manipulated, highlighting a vulnerability.
Future Connected and Automated Vehicles (CAV), and more generally ITS, will form a highly interconnected system. Such a paradigm is referred to as the Internet of Vehicles (herein Internet of CAVs) and is a prerequisite to orchestrate traffic flows in cities. For optimal decision making and supervision, traffic centres…
Connected and autonomous vehicles (CAVs) will form the backbone of future next-generation intelligent transportation systems (ITS) providing travel comfort, road safety, along with a number of value-added services. Such a transformation---which will be fuelled by concomitant advances in technologies for machine learnin…
In this paper we propose a novel observer-based method to improve the safety and security of connected and automated vehicle (CAV) transportation. The proposed method combines model-based signal filtering and anomaly detection methods. Specifically, we use adaptive extended Kalman filter (AEKF) to smooth sensor reading…
Researchers apply concept-based explainability to EEG data.
problem Understanding the internal states of complex EEG transformer models.
method Concept Activation Vectors (CAVs) adapted for EEG data, using externally labeled datasets and anatomically defined concepts.
result Both approaches to concept formation yield valuable insights into EEG model representations.
This paper improves transportation efficiency by teaching automated vehicles to cooperate.
problem Improving efficiency and safety of transportation systems with automated vehicles.
method Multi-agent graph reinforcement learning with attention mechanism.
result Automated vehicles can achieve better performance when learning to cooperate with each other.
Interpreting neural network decisions and the information learned in intermediate layers is still a challenge due to the opaque internal state and shared non-linear interactions. Although (Kim et al, 2017) proposed to interpret intermediate layers by quantifying its ability to distinguish a user-defined concept (from r…
Collision avoidance is a critical task in many applications, such as ADAS (advanced driver-assistance systems), industrial automation and robotics. In an industrial automation setting, certain areas should be off limits to an automated vehicle for protection of people and high-valued assets. These areas can be quaranti…
Many methods for automated software test generation, including some that explicitly use machine learning (and some that use ML more broadly conceived) derive new tests from existing tests (often referred to as seeds). Often, the seed tests from which new tests are derived are manually constructed, or at least simpler t…
A family of maximum mean discrepancy (MMD) kernel two-sample tests is introduced. Members of the test family are called Block-tests or B-tests, since the test statistic is an average over MMDs computed on subsets of the samples. The choice of block size allows control over the tradeoff between test power and computatio…
USP test improves on Pearson's chi-squared and G-test for independence.
problem Deficiencies in Pearson's chi-squared and G-test for independence. method USP test based on U-statistic estimator of population dependence measure. result USP test controls size, handles small cell counts, and detects minimal violations of independence.
E-C2ST uses E-values for high-dimensional data two-sample tests.
problem Statistical testing for high-dimensional data.
method Combines split likelihood ratio tests and predictive independence tests, using E-values for anytime-valid sequential tests.
result E-C2ST achieves enhanced statistical power by partitioning datasets into multiple batches.
This paper provides a comprehensive survey of Machine Learning Testing (ML testing) research. It covers 144 papers on testing properties (e.g., correctness, robustness, and fairness), testing components (e.g., the data, learning program, and framework), testing workflow (e.g., test generation and test evaluation), and …
Cheap permutation tests speed up distribution testing without sacrificing accuracy.
problem Efficiently testing distribution differences and independence.
method Group datapoints into bins and permute only these bins, using stored sufficient statistics.
result Cheap permutation tests maintain the accuracy and optimality of standard tests but are significantly faster.
DRIFT uses RL to automate functional software testing efficiently.
problem Efficient and reliable automated software testing.
method DRIFT employs Q-learning with Graph Neural Networks on symbolic UI representations.
result DRIFT can robustly test software functionalities in a fully automated manner.
Optimizes two-sample tests for non-Euclidean domains using spectral regularization.
problem Optimizing two-sample tests for non-Euclidean domains.
method Spectral regularization of MMD test to achieve minimax optimality.
result Proposes a spectral regularization method that improves test optimality.
Robust test for distributions under Hellinger distance, simpler than optimal tests.
problem Testing and estimating distributions robustly under Hellinger distance.
method Simple robust hypothesis test with optimal sample complexity, robust to Hellinger distance perturbations.
result Empirically demonstrated robustness and power of the test on canonical distributions.
A new method for kernel tests without data splitting increases power.
problem Lack of power in kernel-based tests due to data splitting.
method Selective inference framework to learn hyperparameters and test on full sample.
result Empirically larger test power without data splitting, regardless of split proportion.
Framework for online hypothesis testing across various data types.
problem Testing various nonparametric hypotheses in data streams.
method Unified framework using operators on data distributions, leveraging ML models.
result Efficient, adaptive, and error-controlled sequential tests.
Model-X test detects conditional independence in streaming data.
problem Detecting conditional independence in data streams with arbitrary dependency.
method Sequential testing inspired by model-X and testing by betting.
result Significantly reduces type-I error rate and enhances data efficiency.
Simple methods combine statistical tests for out-of-distribution detection.
problem Detecting data points not following the training distribution.
method Combining classical parametric tests (Rao's score test) and a typicality test.
result Combining Fisher's method of test statistics improves out-of-distribution detection accuracy.
Unified score and distance-based GoF tests for model adequacy.
problem Difficulty in extending score-based GoF tests to nonparametric alternatives.
method Introducing semiparametric kernelized Stein discrepancy (SKSD) test.
result SKSD test is computationally efficient and universally consistent.
AutoML simplifies two-sample tests for detecting distribution shifts.
problem Detecting distribution shifts between datasets.
method Uses mean discrepancy of a witness function with squared loss minimization.
result AutoML simplifies and improves two-sample testing performance.
New methods for CI testing under model misspecification.
problem Challenges in CI testing with misspecified models.
method Proposes new approximations and upper bounds for testing errors of regression-based CI tests.
result Introduces the Rao-Blackwellized Predictor Test (RBPT) robust against misspecified inductive biases.
New KCM tests improve specification testing via RKHS.
problem Improving specification tests for econometric models.
method Kernel conditional moment (KCM) tests based on RKHS.
result KCM tests have better finite-sample performance than existing tests.
Discusses MultiFIT for multivariate dependence, comparing it to HSIC tests.
problem Comparing Multiscale Fisher's Independence Test (MultiFIT) to HSIC tests for multivariate dependence.
method Compares MultiFIT to HSIC tests, highlighting exact level control and performance limitations.
result Observes performance limitations of MultiFIT in terms of test power.
The repeated community-wide reuse of test sets in popular benchmark problems raises doubts about the credibility of reported test-error rates. Verifying whether a learned model is overfitted to a test set is challenging as independent test sets drawn from the same data distribution are usually unavailable, while other …
Develops hypothesis tests for conditional distributions using learning-theoretic bounds.
problem Testing differences in conditional distributions and functionals.
method Transforming learning-theoretic bounds into hypothesis tests for conditional expectations.
result Establishes comprehensive foundation for conditional testing, including theoretical guarantees and practical implementations.
Develops tests for conditional symmetry under group actions.
problem Testing conditional symmetry in distributions under group actions.
method Nonparametric randomization tests with kernel methods and asymptotic consistency.
result Tests achieve finite-sample Type I error control and power.
Paper proposes a new method for more accurate group testing of infected patients.
problem Identifying infected patients efficiently with reduced tests and corrected errors.
method Adaptive design of pools based on Bayesian posterior prediction using belief propagation algorithm.
result The proposed method results in more accurate identification of infected patients.
This paper introduces differentially private permutation tests for hypothesis testing.
problem Privacy concerns in sensitive data analysis.
method Differentially private permutation tests for kernel methods.
result Proposes dpMMD and dpHSIC for two-sample and independence testing, achieving optimal power.
Improved MMD test for two-sample testing with random Fourier features.
problem Quadratic-time complexity of MMD test for large-scale analysis.
method Approximated MMD test using random Fourier features, investigating time-power trade-off.
result Sub-quadratic time complexity with same minimax separation rates as MMD test.
Distance correlation has gained much recent attention in the data science community: the sample statistic is straightforward to compute and asymptotically equals zero if and only if independence, making it an ideal choice to discover any type of dependency structure given sufficient sample size. One major bottleneck is…
Paper proposes a robust test for high-dimensional models with large covariates and instruments.
problem Testing high-dimensional linear instrumental variable models with large covariates and instruments.
method Introduces a test based on the maximum norm of multiple parameters and a power-enhanced test.
result The proposed test is robust to heteroskedastic errors and has higher power than existing tests.
Polynomial-time test for detecting dense subgraphs in heterogeneous networks.
problem Detecting a planted community in heterogeneous networks.
method Proposes a polynomial-time test with a standard normal distribution null limiting distribution.
result The test is efficient and performs well in both simulations and real data.
The study analyzes group testing algorithms for identifying defective items with high confidence.
problem Identifying defective items from a population using group testing with high confidence.
method Formulated as a function learning problem using the PAC framework, analyzed three algorithms: column matching, combinatorial basis pursuit, and definite defectives.
result Derived bounds on the number of tests needed for approximate set identification, comparing with existing bounds and simulating performance.
We propose a novel adaptive test of goodness-of-fit, with computational cost linear in the number of samples. We learn the test features that best indicate the differences between observed samples and a reference model, by minimizing the false negative rate. These features are constructed via Stein's method, meaning th…
A permutation-based SW test achieves minimax-optimal power for two-sample testing.
problem Nonparametric two-sample testing using the sliced Wasserstein distance.
method Proposes a permutation-based SW test and analyzes its performance.
result Achieves minimax separation rate n−1/2 over multinomial and bounded-support alternatives. We conduct an extensive evaluation of price jump tests based on high-frequency financial data. After providing a concise review of multiple alternative tests, we document the size and power of all tests in a range of empirically relevant scenarios. Particular focus is given to the robustness of test performance to the …
This paper explores the information-theoretic limitations of graph property testing in zero-field Ising models. Instead of learning the entire graph structure, sometimes testing a basic graph property such as connectivity, cycle presence or maximum clique size is a more relevant and attainable objective. Since property…
CTT compresses samples to test distributions near-linearly, outperforming existing methods.
problem Efficiently testing distributions with high power and near-linear runtime.
method Sample compression followed by permutation testing.
result CTT achieves near-linear runtime while maintaining high statistical power.
Meta two-sample testing uses auxiliary data to quickly find powerful tests from limited samples.
problem Challenges in identifying powerful kernels for distinguishing complex distributions with limited data.
method Introduces meta two-sample testing (M2ST) to leverage abundant auxiliary data on related tasks.
result Proposed algorithms improve over baselines and identify powerful tests from scarce observations.
We consider the problem of comparing probability densities between two groups. A new probabilistic tensor product smoothing spline framework is developed to model the joint density of two variables. Under such a framework, the probability density comparison is equivalent to testing the presence/absence of interactions.…
Conditional independence tests (CI tests) have received special attention lately in Machine Learning and Computational Intelligence related literature as an important indicator of the relationship among the variables used by their models. In the field of Probabilistic Graphical Models (PGM)--which includes Bayesian Net…