nFBST tests neural networks using Bayesian methods.
problem Traditional significance testing struggles with complex nonlinear relationships.
method nFBST uses Bayesian neural networks to test neural networks.
result nFBST can test global, local, and instance-wise significance.
Paper presents a machine learning method to improve significance tests for misspecified linear models.
problem Misspecification of linear assumptions in social science models leads to inaccurate significance levels.
method Apply machine learning to fit ground truth function, calculate linear approximation, and adjust the estimator.
result The method significantly outperforms linear regression for non-linear ground truth functions.
Conditional independence tests (CI tests) have received special attention lately in Machine Learning and Computational Intelligence related literature as an important indicator of the relationship among the variables used by their models. In the field of Probabilistic Graphical Models (PGM)--which includes Bayesian Net…
Develops significance tests for neural networks without strong assumptions or excessive computation.
problem Addressing the black-box nature of deep neural networks for feature relevance testing.
method Derives one-split and two-split tests relaxing assumptions and computational complexity.
result Establishes asymptotic null distributions and consistency in Type II error.
Significant pattern mining, the problem of finding itemsets that are significantly enriched in one class of objects, is statistically challenging, as the large space of candidate patterns leads to an enormous multiple testing problem. Recently, the concept of testability was proposed as one approach to correct for mult…
We discuss a common suspicion about reported financial data, in 10 industrial sectors of the 6 so called "main developing countries" over the time interval [2000-2014]. These data are examined through Benford's law first significant digit and through distribution distances tests. It is shown that several visually anoma…
The paper improves asymmetric causality tests by addressing inefficiencies and statistical significance issues.
problem Inefficiencies and statistical significance issues in asymmetric causality tests.
method Improved asymmetric causality tests via partial cumulative sums for positive and negative components, explicitly testing differences between causal parameters.
result Efficiently tested hypotheses on asymmetric causal interaction between financial markets.
We develop a pivotal test to assess the statistical significance of the feature variables in a single-layer feedforward neural network regression model. We propose a gradient-based test statistic and study its asymptotics using nonparametric techniques. Under technical conditions, the limiting distribution is given by …
Proposes a method to calibrate data for more accurate linear correlation testing.
problem Inaccurate Pearson's correlation coefficient due to sample size and data non-normality.
method Predictive data calibration using machine learning to condition data on expected linear relationship.
result Calibrated Pearson's correlation coefficient yields a calibrated p-value and r estimate for posterior probability interpretation.
Develops a new method for neural network significance testing without strict constraints.
problem Testing neural networks without bounded weights or specific architectural constraints.
method Uses Rademacher complexity bounds, weakened Sobolev space membership conditions, and a modified sieve space construction.
result Achieves optimal convergence rates and valid asymptotic distributions for test statistics.
New test improves clustering accuracy for Gaussian mixtures.
problem Improving clustering accuracy for Gaussian mixtures.
method Relative fit test for Gaussian Mixture Models.
result New test provides provable error control and higher power.
We develop a simple and computationally efficient significance test for the features of a machine learning model. Our forward-selection approach applies to any model specification, learning task and variable type. The test is non-asymptotic, straightforward to implement, and does not require model refitting. It identif…
The statistical comparison of multiple algorithms over multiple data sets is fundamental in machine learning. This is typically carried out by the Friedman test. When the Friedman test rejects the null hypothesis, multiple comparisons are carried out to establish which are the significant differences among algorithms. …
A new method tests variable significance without assuming model correctness.
problem Testing variable significance in the presence of complex interactions.
method Flexible nonparametric or machine learning methods to estimate conditional mean independence.
result Achieves minimax optimal rate in nonparametric testing problem.
A new framework detects statistical significance of deep learning in neuroimaging studies.
problem Lack of statistical significance testing in deep learning neuroimaging.
method Non-parametric framework using autoencoders and SVM, with random-effects inference and cross-validation.
result CV and RUB methods offer acceptable false positive rates and statistical power, but low generalization ability.
Investigates the number of experiments needed for statistical significance in medication testing.
problem Determining the number of experiments needed for a statistically significant result.
method Examines binomial and general probability distributions, considering placebo efficacy and varying distributions.
result The number of experiments needed can be significantly higher when placebo efficacy is considered.
AICO tests feature significance in machine learning models.
problem Lack of transparency in machine learning models.
method AICO framework for feature significance testing.
result AICO provides statistical guarantees for feature importance.
si4onnx enables selective inference on deep learning models.
problem Establishing the reliability of AI systems through statistical significance of identified regions.
method Selective inference techniques implemented through a Python package.
result Controlled type I error rates for hypothesis testing on deep learning models.
Test assesses if a linear classifier is random or significant.
problem Determining if a linear classifier captures meaningful differences between classes.
method Proposes a homogeneity test related to linear separability, establishes upper bounds for p-values.
result Upper bounds for p-values are highly accurate for normally distributed samples.
We show that univariate and symmetric multivariate Hawkes processes are only weakly causal: the true log-likelihoods of real and reversed event time vectors are almost equal, thus parameter estimation via maximum likelihood only weakly depends on the direction of the arrow of time. In ideal (synthetic) conditions, test…
This paper predicts significant stock price changes using neural networks.
problem Predicting significant stock price changes.
method Three neural network models (MLP, CNN, LSTM) and two benchmark models (Random Forest, Relative Strength Index) were tested on 10-year daily stock price data of four major US companies.
result Neural network models significantly outperform traditional methods in predicting significant stock price changes.
The problem of multiple hypothesis testing arises when there are more than one hypothesis to be tested simultaneously for statistical significance. This is a very common situation in many data mining applications. For instance, assessing simultaneously the significance of all frequent itemsets of a single dataset entai…
Consistently checking the statistical significance of experimental results is one of the mandatory methodological steps to address the so-called "reproducibility crisis" in deep reinforcement learning. In this tutorial paper, we explain how the number of random seeds relates to the probabilities of statistical errors. …
The paper designs tests for comparing ranked preference data and finds significant differences.
problem Comparing pairwise comparison and ranking data in various applications.
method Developed two-sample tests for pairwise comparison and ranking data, proving upper and lower bounds.
result Upper and lower bounds show tightness of the proposed tests, and significant differences in preferences were found.
The problem of finding itemsets that are statistically significantly enriched in a class of transactions is complicated by the need to correct for multiple hypothesis testing. Pruning untestable hypotheses was recently proposed as a strategy for this task of significant itemset mining. It was shown to lead to greater s…
We investigate the problem of testing whether d random variables, which may or may not be continuous, are jointly (or mutually) independent. Our method builds on ideas of the two variable Hilbert-Schmidt independence criterion (HSIC) but allows for an arbitrary number of variables. We embed the d-dimensional joint …
New deep learning method improves financial stress testing accuracy.
problem Traditional stress testing methods are criticized for unrealistic assumptions and estimation errors.
method Proposes a novel Deep Learning approach for Dynamic Balance Sheet Stress Testing.
result Empirical results show significant improvement in accuracy over traditional methods.
CovRegRF estimates covariance matrix from covariates using random forests.
problem Estimating conditional covariances or correlations among multivariate responses.
method Random forest trees with a custom splitting rule to maximize covariance difference.
result Accurate covariance matrix estimates and controlled Type-1 error.
A new protocol evaluates small machine learning improvements conservatively.
problem Uncertainty in small gains reported in machine learning papers.
method Paired bootstrap protocol with BCa confidence intervals and sign-flip permutation tests.
result Conservative evaluation reduces over-claiming of small improvements.
Bayesian methods detect significant IIA violations in similarity choice data.
problem Detecting IIA violations in similarity choice data complicates classical models.
method Proposed two statistical methods: classical goodness-of-fit test and Bayesian PPC.
result Significant IIA violations confirmed in both datasets, driven by context effects.
We develop a theoretical trading conditioning model subject to price volatility and return information in terms of market psychological behavior, based on analytical transaction volume-price probability wave distributions in which we use transaction volume probability to describe price volatility uncertainty and intens…
Paper tests for time-varying entropy in stock prices, finding periods of inefficiency.
problem Testing for time-varying entropy in stock price dynamics.
method Unbiased approximation of Shannon entropy variance, optimal rolling window selection, hypothesis testing.
result Existence of periods of market inefficiency for meme stocks.
In real-world classification problems, the class balance in the training dataset does not necessarily reflect that of the test dataset, which can cause significant estimation bias. If the class ratio of the test dataset is known, instance re-weighting or resampling allows systematical bias correction. However, learning…
Sharpe ratio (sometimes also referred to as information ratio) is widely used in asset management to compare and benchmark funds and asset managers. It computes the ratio of the (excess) net return over the strategy standard deviation. However, the elements to compute the Sharpe ratio, namely, the expected returns and …
Developing state-of-the-art approaches for specific tasks is a major driving force in our research community. Depending on the prestige of the task, publishing it can come along with a lot of visibility. The question arises how reliable are our evaluation methodologies to compare approaches? One common methodology to i…
New measures and tests for high-order interactions in complex data.
problem Challenges in capturing high-order interactions in multivariate data.
method Hierarchy of d-order interaction measures and kernel-based tests. result Established statistical significance of high-order interactions.
Improved change point detection using matched filters for non-parametric tests.
problem False positives and localization ambiguity in non-parametric two-sample tests.
method Derived and applied matched filters for various two-sample tests.
result Matched filters reduce false positives and improve test precision.
Hypothesis tests are a crucial statistical tool for data mining and are the workhorse of scientific research in many fields. Here we present a differentially private analogue of the classic Wilcoxon signed-rank hypothesis test, which is used when comparing sets of paired (e.g., before-and-after) data values. We present…
Study improves statistical power for detecting algorithmic bias in educational data.
problem Challenges in measuring algorithmic bias using ABROCA due to skewed distribution.
method Investigates ABROCA's distributional properties and proposes nonparametric randomization tests.
result ABROCA-based bias assessments are underpowered in typical EDM sample sizes.
E-C2ST uses E-values for high-dimensional data two-sample tests.
problem Statistical testing for high-dimensional data.
method Combines split likelihood ratio tests and predictive independence tests, using E-values for anytime-valid sequential tests.
result E-C2ST achieves enhanced statistical power by partitioning datasets into multiple batches.
Improves A/B testing by detecting minor treatment effects.
problem Challenges in identifying small average treatment effects.
method Maximum probability-driven two-armed bandit (TAB) process with weighted mean volatility statistic.
result Significant improvement in A/B testing with reduced experimental costs.
A new method for kernel tests without data splitting increases power.
problem Lack of power in kernel-based tests due to data splitting.
method Selective inference framework to learn hyperparameters and test on full sample.
result Empirically larger test power without data splitting, regardless of split proportion.
The statistical analysis of discrete data has been the subject of extensive statistical research dating back to the work of Pearson. In this survey we review some recently developed methods for testing hypotheses about high-dimensional multinomials. Traditional tests like the χ2 test and the likelihood ratio test ca…
Group Shapley evaluates feature groups in business data, improving explainability in AI.
problem Evaluating the importance of feature groups in business and economic data.
method Developed Group Shapley and a significance testing procedure based on chi-square approximation.
result Market-related variables are identified as the most influential feature group.
Study tests UK FTSE-listed companies' financial data for Benford's Law conformity.
problem Ensuring the fairness of public revenue collection and reducing tax avoidance risks.
method Utilised pre-tax income and total assets data from 567 FTSE companies, tested for Benford's Laws conformity using χ2 and MAD tests. result MAD test rejects Benford's Laws conformity, suggesting potential issues with reported financial data.
Study confirms Indian stock market is weak form inefficient.
problem Impact of stock market efficiency on investment returns.
method Runs test, Autocorrelation test, Autoregression test on daily stock indices.
result Indian stock market is weak form inefficient and can be outperformed.
The paper proposes a test to assess rater accuracy while accounting for rater covariates.
problem Assessing the accuracy of raters in medical imaging and forensic studies.
method Covariate-adjusted homogeneity test to determine differences in accuracy among multiple rater groups.
result The proposed test identifies statistically significant differences among five participant groups in a face recognition study.
A sequential classifier minimizes test samples for binary and multi-class classification.
problem Minimizing test samples for sequential classification with unknown distributions.
method Proposes a classifier for binary and multi-class problems, analyzing error probabilities and extending results.
result Significant advantage over non-sequential classifiers, achieving same exponents without rejection option.