Algorithm detects unmeasured confounding in observational data.
problem Estimating treatment effects in observational studies with untestable conditions.
method Two-stage procedure that detects dependencies between causal mechanisms.
result Algorithm efficiently detects confounding on simulated and semi-synthetic data.
Proposes a falsification framework to test algorithmic discriminant validity.
problem Unintended model behavior in predictive algorithms.
method Falsification framework based on statistical tests comparing prediction losses across outcomes.
result Establishes discriminant validity for some outcomes but not others.
VerifAI toolkit improves neural network-based aircraft taxiing system safety.
problem Improving safety of autonomous aircraft taxiing systems using neural networks.
method Unified approach to formal analysis and retraining of AI systems, including falsification, debugging, and retraining.
result Improved neural network performance and reduced failure cases in aircraft taxiing system.
Proposes a test to ensure predictive algorithms predict intended outcomes better than unintended ones.
problem Unintended model behavior leading to prediction of unintended outcomes.
method Falsification framework using nonparametric hypothesis testing to compare prediction losses across outcomes.
result Establishes discriminant validity with respect to gender but not race in an admissions setting.
Audit financial machine learning workflows to detect spurious predictability.
problem Spurious predictability in financial machine learning models.
method Falsification audit testing predictive workflows against synthetic environments.
result Many apparent financial predictions are artifacts, not genuine.
Study finds no statistically significant trading edge in MNQ futures signals from OHLCV data.
problem Testing intraday momentum signals from OHLCV data in MNQ futures under realistic execution constraints.
method 947 trading days of five-minute data, 14 signal families evaluated, strict institutional criteria applied.
result No signal satisfies all criteria simultaneously, gross edge insufficient to overcome costs.
This work examines fundamental limits in model falsification without assuming specific distributions.
problem Establishing lower bounds on model class risk in distribution-free settings.
method Model-agnostic fundamental hardness result for constructing lower bounds on test error.
result No positive lower bound on model class risk is possible in certain settings.
This paper considers the problem of detection in distributed networks in the presence of data falsification (Byzantine) attacks. Detection approaches considered in the paper are based on fully distributed consensus algorithms, where all of the nodes exchange information only with their neighbors in the absence of a fus…
A new method predicts causal relationships without joint data.
problem Falsifying causal discovery algorithms without ground truth.
method Leave-One-Variable-Out (LOVO) prediction for causal graphs.
result LOVO method correlates prediction error with causal algorithm accuracy.
We information-theoretically reformulate two measures of capacity from statistical learning theory: empirical VC-entropy and empirical Rademacher complexity. We show these capacity measures count the number of hypotheses about a dataset that a learning algorithm falsifies when it finds the classifier in its repertoire …
Tests validity of DML estimators without assumptions.
problem Validating DML estimators without making assumptions.
method Develops tests to falsify assumptions for DML estimators.
result Falsifies assumptions for DML estimators with non-trivial power.
Efficiently generates models resistant to falsification.
problem Creating models that cannot be disproven by tests.
method Exploits connections between high-dimensional multicalibration and expected variational inequality problems to develop an efficient algorithm.
result First to efficiently produce online outcome indistinguishable generative models resistant to infinite classes of tests.
The standard taxonomy of predictive uncertainty is inconsistent with standard measures.
problem Uncertainty taxonomy and measure inconsistency
method Proof of inconsistency
result Uncertainty is not reducible to data collection
New method falsifies causal discovery results without ground truth.
problem Evaluation of causal discovery algorithms without ground truth data.
method Detects incompatibilities between causal graphs learned on different subsets of variables.
result Detection of incompatibilities can falsify wrongly inferred causal relations.
The paper argues that machine learning is a falsificationist process.
problem The role of falsification in machine learning is underexplored.
method The paper presents a falsificationist account of artificial neural networks, emphasizing empirical risk minimization and implicit regularization.
result Artificial neural networks can be seen as a falsificationist process, rejecting inadequate prediction rules.
A new metric assesses causal graphs using node permutations to detect inconsistencies.
problem Quantifying the goodness of causal graphs and distinguishing them from random graphs.
method Constructing a baseline through node permutations and comparing inconsistencies.
result The proposed metric can distinguish between true and wrong causal graphs.
Study cyber-attacks on RL algorithms, focusing on cost signal manipulation.
problem Adversarial manipulation of cost signals in reinforcement learning.
method Quantitative analysis of TD(λ) and Q-learning algorithms under manipulation. result Bound on approximation error for TD(λ) and convergence properties for Q-learning under stealthy attacks. There are (at least) three approaches to quantifying information. The first, algorithmic information or Kolmogorov complexity, takes events as strings and, given a universal Turing machine, quantifies the information content of a string as the length of the shortest program producing it. The second, Shannon information…
Survey of algorithms for testing AI-driven CPS safety.
problem Testing AI-driven CPS for safety in complex environments.
method Survey of applied algorithms for safety validation.
result Survey of existing tools and techniques for safety validation.
New method tightens bounds on causation probabilities using independent datasets.
problem Challenging point identification of causation probabilities without strong assumptions.
method Imposes counterfactual consistency between SCMs constructed from independent datasets and uses conditional mutual information.
result Significantly tighter bounds on causation probabilities are established.
New methods improve robust decision-making under uncertainty in off-policy evaluation.
problem Statistical uncertainty and causal considerations in off-policy evaluation.
method Marginal Ratio (MR) estimator, Conformal Off-Policy Prediction (COPP), causal bounds.
result Improved robustness and uncertainty quantification in off-policy decision-making.
Faster Tsetlin Machines use clause indexing to speed inference and learning.
problem Overfitting and slow inference in Tsetlin Machines.
method Introduced a look-up table that indexes clauses based on feature falsification, enabling faster evaluation of clauses.
result Up to 15 times faster classification and three times faster learning on MNIST and Fashion-MNIST.
GE finds failures in autonomous systems without domain heuristics.
problem Finding failures in autonomous systems without domain-specific heuristics.
method Adaptive stress testing using go-explore (GE) algorithm.
result GE finds failures in scenarios other RL techniques cannot solve.
Causal Bayesian networks interpret actions as interventions to connect models to real-world outcomes.
problem Connecting causal model predictions to real-world outcomes.
method Formal framework to interpret actions as interventions and prove impossibility results.
result No non-circular interpretation exists that satisfies natural desiderata without violating some.