Proposes a new test for validating multivariate dynamic regression models.
problem Inadequate exogeneity conditions for conventional model specification tests in dynamic systems.
method Develops a generalized Durbin estimator for multiple-equation systems with dynamic dependencies, and constructs Wald tests.
result Bootstrap-based Wald tests improve finite-sample size control and validate the null hypothesis in multifactor models.
Study provides selective inference method for latent block models.
problem Challenges in constructing a test on a block structure selected by clustering algorithms.
method Developed a selective inference method for latent block models using squared residue minimization and simulated annealing.
result Proposed tests effectively handle selective bias in block structures compared to naive tests.
We propose a novel framework of the model specification test in regression using unlabeled test data. In many cases, we have conducted statistical inferences based on the assumption that we can correctly specify a model. However, it is difficult to confirm whether a model is correctly specified. To overcome this proble…
This work constructs a hypothesis test for detecting whether an data-generating function h:Rp→R belongs to a specific reproducing kernel Hilbert space H0 , where the structure of H0 is only partially known. Utilizing the theory of reproducing kernels, we reduce this hypothesis …
New KCM tests improve specification testing via RKHS.
problem Improving specification tests for econometric models.
method Kernel conditional moment (KCM) tests based on RKHS.
result KCM tests have better finite-sample performance than existing tests.
Test log-likelihood comparisons can be misleading.
problem Misinterpretation of test log-likelihood in model comparison.
method Simple examples of model comparison and forecast accuracy.
result Test log-likelihood does not always correlate with model accuracy.
New method estimates data influence efficiently by leveraging test samples.
problem Efficiently estimating influence of training data on model predictions.
method Mirrored Influence Hypothesis, forward pass for test samples.
result Significant improvement in efficiency over existing methods.
Markov Chain Monte Carlo (MCMC) algorithms are a workhorse of probabilistic modeling and inference, but are difficult to debug, and are prone to silent failure if implemented naively. We outline several strategies for testing the correctness of MCMC algorithms. Specifically, we advocate writing code in a modular way, w…
Three bootstrap tests compare categorical time series generating processes.
problem Testing equality of generating processes in categorical time series.
method Proposes three tests using dissimilarity measures and bootstrap methods.
result Advantages and disadvantages of each bootstrap method discussed.
New test identifies specific biological parameters for personalized CVD detection.
problem Ineffectual pathology tests fail to consider platelet activation and inter-individual variability.
method Stochastic platelet deposition model and approximate Bayesian computation with discriminative summary statistics.
result Inferred parameters help identify specific biological parameters for personalized CVD detection.
Detects which features have shifted in data distributions.
problem Identifying which specific features have caused a distribution shift.
method Formalizes the problem as multiple conditional distribution hypothesis tests, proposes non-parametric and parametric statistical tests, and uses a test statistic based on the density model score function.
result Demonstrates methods for identifying when and where a shift occurs in multivariate time-series data.
Study improves statistical power for detecting algorithmic bias in educational data.
problem Challenges in measuring algorithmic bias using ABROCA due to skewed distribution.
method Investigates ABROCA's distributional properties and proposes nonparametric randomization tests.
result ABROCA-based bias assessments are underpowered in typical EDM sample sizes.
TTT improves transformer models for in-context learning.
problem Improving transformer models for efficient in-context learning.
method Gradient-based TTT method for linear transformers, with theoretical and empirical analysis.
result TTT significantly reduces the sample size required for in-context learning.
Test-asset construction affects factor model performance.
problem How test assets are constructed impacts factor model performance.
method Forming characteristic-unsorted random portfolios and varying stock selection, initial weighting, holding, and rebalancing.
result Test-asset construction shifts factor model rankings materially.
Simple methods combine statistical tests for out-of-distribution detection.
problem Detecting data points not following the training distribution.
method Combining classical parametric tests (Rao's score test) and a typicality test.
result Combining Fisher's method of test statistics improves out-of-distribution detection accuracy.
Financial econometrics has become an increasingly popular research field. In this paper we review a few parametric and nonparametric models and methods used in this area. After introducing several widely used continuous-time and discrete-time models, we study in detail dependence structures of discrete samples, includi…
Neuron-specific dropout reduces overfitting and data needs for neural networks.
problem Overfitting and insufficient training data for deep neural networks.
method Compares training and validation passes of a layer, drops targeted neurons based on feature analysis.
result Achieves similar or better testing accuracy with less data, reducing overfitting.
New method for group testing robust to errors in group membership specifications.
problem Errors in specifying group memberships during group testing.
method Debiased Robust Lasso Test Method (DRLT) based on Lasso debiasing.
result Extends LASSO bias mitigation to handle group membership specification errors.
Risk monitoring detects when TTA models degrade at test time.
problem Detecting when TTA models degrade at test time.
method Extended risk monitoring tools based on sequential testing with confidence sequences.
result Demonstrated effectiveness of TTA monitoring framework across various datasets and methods.
Information retrieval (IR) systems often leverage query data to suggest relevant items to users. This introduces the possibility of unfairness if the query (i.e., input) and the resulting recommendations unintentionally correlate with latent factors that are protected variables (e.g., race, gender, and age). For instan…
Active testing reduces label costs for efficient model evaluation.
problem Real-world applications require expensive test labels, disconnecting from existing model evaluation methods.
method Derives acquisition strategies to select test points efficiently, addressing label bias and variance.
result Active testing improves model evaluation efficiency without sacrificing accuracy.
Study tests if input gradients highlight discriminative features, finds they often fail.
problem Validity of assumption that input gradients highlight discriminative features in model predictions.
method Developed DiffROAR framework and BlockMNIST dataset to test assumption on four benchmarks.
result Input gradients of standard models often fail to highlight discriminative features, while robust models do.
A keyword spotting (KWS) system determines the existence of, usually predefined, keyword in a continuous speech stream. This paper presents a query-by-example on-device KWS system which is user-specific. The proposed system consists of two main steps: query enrollment and testing. In query enrollment step, phonetic pos…
Proposes a reverse stress testing framework for dynamic models.
problem Finding plausible models under adverse stresses.
method Compound Poisson process, Kullback-Leibler divergence, optimization problem.
result Intensity and severity of process depend on time and state.
In this work, we consider hypothesis testing and anomaly detection on datasets where each observation is a weighted network. Examples of such data include brain connectivity networks from fMRI flow data, or word co-occurrence counts for populations of individuals. Current approaches to hypothesis testing for weighted n…
New method tests DAGs without assuming linear or independent data.
problem Testing DAGs with nonlinear and time-dependent data.
method Structural, supervised and generative adversarial learning.
result Asymptotic guarantees for the test, allowing diverging data dimensions.
The Multinomial Logit (MNL) model and the axiom it satisfies, the Independence of Irrelevant Alternatives (IIA), are together the most widely used tools of discrete choice. The MNL model serves as the workhorse model for a variety of fields, but is also widely criticized, with a large body of experimental literature cl…
Over 150,000 new people in the United States are diagnosed with colorectal cancer each year. Nearly a third die from it (American Cancer Society). The only approved noninvasive diagnosis tools currently involve fecal blood count tests (FOBTs) or stool DNA tests. Fecal blood count tests take only five minutes and are av…
Kernel tests assess equivalence between distributions without assuming specific moments.
problem Traditional goodness-of-fit tests fail to detect meaningful distributional differences.
method Proposes kernel-based tests using kernel Stein discrepancy and Maximum Mean Discrepancy.
result Tests assess the absence of meaningful distributional differences under controlled error rates.
Deep-learning method improves hypothesis testing for independence.
problem Improving hypothesis testing for independence using deep learning.
method Proposes deep-testing, a novel procedure that uses a deep neural network to distinguish between data generated under and outside a given statistical model.
result Deep-testing achieves the highest overall power against nineteen competing methods across various dependence structures.
Develops model selection for bandits balancing adversarial and stochastic guarantees.
problem Model selection in bandit scenarios with simultaneous adversarial and stochastic high-probability regret.
method Nested policy classes, balanced candidate regret bounds, mis-specification tests.
result Best of both world guarantees in linear bandits with simultaneous adversarial and stochastic environments.
The paper discusses fairness in bank stress tests, comparing various methods to address institutional differences.
problem Fair aggregation of bank-specific stress test models into a common model.
method Comparing various notions of regression fairness, including estimating and discarding centered bank fixed effects.
result The method of estimating and discarding centered bank fixed effects is preferable for linear models, improving forecast accuracy and equal treatment.
We develop a simple and computationally efficient significance test for the features of a machine learning model. Our forward-selection approach applies to any model specification, learning task and variable type. The test is non-asymptotic, straightforward to implement, and does not require model refitting. It identif…
Model interpretability is a requirement in many applications in which crucial decisions are made by users relying on a model's outputs. The recent movement for "algorithmic fairness" also stipulates explainability, and therefore interpretability of learning models. And yet the most successful contemporary Machine Learn…
New method combines randomization tests and flexible models for valid inference without splitting data.
problem Valid inference in randomized panel experiments with complex effect heterogeneity.
method Model-assisted randomization tests that estimate unsigned CATE from residualized outcomes.
result CATE-assisted tests control Type I error and achieve higher power than alternatives.
Unified score and distance-based GoF tests for model adequacy.
problem Difficulty in extending score-based GoF tests to nonparametric alternatives.
method Introducing semiparametric kernelized Stein discrepancy (SKSD) test.
result SKSD test is computationally efficient and universally consistent.
Recursive Feature Machines show grokking in modular arithmetic without neural networks.
problem Grokking in modular arithmetic tasks.
method Recursive Feature Machines (RFM) with Average Gradient Outer Product (AGOP).
result RFM and neural networks learn block-circulant features to solve modular arithmetic.
Concolic testing combines program execution and symbolic analysis to explore the execution paths of a software program. This paper presents the first concolic testing approach for Deep Neural Networks (DNNs). More specifically, we formalise coverage criteria for DNNs that have been studied in the literature, and then d…
A new test for conditional independence in discretized data.
problem Testing conditional independence when only discretized observations are available.
method Proposes a conditional independence test designed for discretized observations, using bridge equations to recover latent variables' information.
result Demonstrates the effectiveness of the proposed test through theoretical and empirical validation.
Most previous works usually explained adversarial examples from several specific perspectives, lacking relatively integral comprehension about this problem. In this paper, we present a systematic study on adversarial examples from three aspects: the amount of training data, task-dependent and model-specific factors. Pa…
Two-sample tests using MMD control type I error and achieve optimal power.
problem Developing reliable nonparametric two-sample tests for small sample sizes.
method Maximum Mean Discrepancy (MMD) for constructing novel nonparametric tests, proving non-asymptotic error control and optimality.
result MMDAgg test controls type I error and achieves minimax rate over Sobolev balls, outperforming other tests.
The paper develops a faster surrogate model for simulators using hybrid methods.
problem The need for faster validation of automotive technologies using simulators.
method Testing classical methods and building hybrid models combining them.
result A hybrid surrogate model outperforms classical methods in multivariate time series prediction.
This paper studies the matched network inference problem, where the goal is to determine if two networks, defined on a common set of nodes, exhibit a specific form of stochastic similarity. Two notions of similarity are considered: (i) equality, i.e., testing whether the networks arise from the same random graph model,…
The paper tackles data misappropriation in LLMs by embedding watermarks and testing for their presence.
problem Detecting data misappropriation in LLMs trained on copyrighted data.
method Embedding watermarks, formulating as hypothesis testing, developing statistical framework, constructing test statistics, determining optimal thresholds, controlling errors, establishing asymptotic optimality.
result The proposed statistical testing framework effectively detects data misappropriation in LLMs.
Excessive reuse of test data has become commonplace in today's machine learning workflows. Popular benchmarks, competitions, industrial scale tuning, among other applications, all involve test data reuse beyond guidance by statistical confidence bounds. Nonetheless, recent replication studies give evidence that popular…
Proposes a FoE prior for improving CNN performance in distribution shifts.
problem Improving CNN performance in image analysis tasks with distribution shifts.
method Uses a field-of-experts (FoE) prior to match feature distributions of test and training images.
result Outperforms previous TTA methods in lesion segmentation and most healthy tissue segmentation tasks.
We propose a method to infer domain-specific models such as classifiers for unseen domains, from which no data are given in the training phase, without domain semantic descriptors. When training and test distributions are different, standard supervised learning methods perform poorly. Zero-shot domain adaptation attemp…
New risk factors improve stress testing accuracy.
problem Improving stress testing accuracy with new risk factors.
method Adapted PCA and autoencoders for dimension reduction and interpretation.
result Aggregated risk factors enhance stress testing outcomes.