Deep learning models are evaluated for sensory information processing.
problem Unclear interpretation of model comparison techniques for DNNs.
method Explicitly define conclusions from existing model comparison techniques.
result Stronger conclusions about sensory processing mechanisms possible with DNNs.
Neuroimaging research has predominantly drawn conclusions based on classical statistics, including null-hypothesis testing, t-tests, and ANOVA. Throughout recent years, statistical learning methods enjoy increasing popularity, including cross-validation, pattern classification, and sparsity-inducing regression. These t…
The authors argue against the classification of forecasting methods as machine learning or statistical.
problem The classification of forecasting methods as machine learning or statistical limits insights into their appropriateness and effectiveness.
method Alternative characteristics of forecasting methods are proposed to draw meaningful conclusions.
result The distinction between machine learning and statistical forecasting methods is not fundamental.
SMILE improves explainability of machine learning models.
problem Difficulty in understanding and trusting the conclusions of black-box machine learning models.
method Statistical Model-agnostic Interpretability with Local Explanations (SMILE).
result SMILE makes machine learning models more interpretable.
The thesis tackles overconfident approximations in simulation-based inference.
problem Overconfident conclusions from machine learning approximations in statistical analyses.
method Introduces balancing and Bayesian neural networks to reduce overconfidence.
result Balancing and Bayesian neural networks lead to less overconfident approximations.
The paper argues for using more degrees of freedom in empirical financial analysis to improve conclusions.
problem Improving trustworthiness of financial analysis conclusions.
method Using more degrees of freedom and forking paths in multiple testing.
result Forking paths raises the bar for significance in multiple testing.
We extend nonparametric models to handle extrapolation, providing bounds for inference.
problem Challenges in nonparametric statistical inference when evaluating outside the conditioning variable's support.
method Introduced a class of extrapolation assumptions and a consistent estimation procedure to handle extrapolation.
result Validated extrapolation-aware conclusions through various applications and real-world data.
The paper evaluates index-based allocation policies using data from randomized control trials.
problem Evaluating index-based allocation policies in resource-scarce scenarios.
method Using data from randomized control trials, the paper introduces an efficient estimator and methods for computing asymptotically correct confidence intervals.
result Valid statistical conclusions can be drawn for index-based allocation policies.
New estimator improves statistical validity of synthetic data integration.
problem Combining synthetic data generated by large language models with real data for valid inference.
method Generalized method of moments estimator with theoretical guarantees.
result Improves estimates of target parameter through interactions between synthetic and real data.
Paper introduces a new gradient statistic to improve deep learning convergence.
problem Fluctuation effect of gradient updates between iterations.
method Introduces an unbiased stratified statistic \(\bar{G}_{mst}\) and a new algorithm MSSG.
result MSSG algorithm outperforms other sgd-like algorithms in training deep models.
Proposes incorporating noise sources in machine learning evaluation for more reliable conclusions.
problem Inadequate handling of nondeterminism in machine learning research leads to unreliable results.
method Uses linear mixed effects models (LMEMs) and generalized likelihood ratio tests (GLRT) to analyze performance evaluation scores and assess performance differences.
result Demonstrates how to incorporate various sources of noise and data properties into statistical significance testing and reliability analysis.
Deep RL evaluation underestimates uncertainty, leading to misleading conclusions.
problem Statistical uncertainty in deep RL performance evaluations is underestimated, leading to misleading conclusions.
method Advocates for reporting interval estimates of aggregate performance and proposes performance profiles to account for variability.
result Substantial discrepancies in prior performance comparisons are revealed, highlighting the need for more rigorous evaluation methods.
The possibility of statistical evaluation of the market completeness and incompleteness is investigated for continuous time diffusion stock market models. It is known that the market completeness is not a robust property: small random deviations of the coefficients convert a complete market model into a incomplete one.…
Deep learning depends on tuning layers near critical points.
problem Understanding how deep learning architectures depend on tuning parameters.
method Random energy approach to analyze statistical dependence in deep belief networks.
result Statistical dependence can propagate only if layers are tuned near critical points.
A research frontier has emerged in scientific computation, wherein numerical error is regarded as a source of epistemic uncertainty that can be modelled. This raises several statistical challenges, including the design of statistical methods that enable the coherent propagation of probabilities through a (possibly dete…
Big Data bring new opportunities to modern society and challenges to data scientists. On one hand, Big Data hold great promises for discovering subtle population patterns and heterogeneities that are not possible with small-scale data. On the other hand, the massive sample size and high dimensionality of Big Data intro…
A new method uses randomized trials to estimate the strength of unobserved confounding.
problem Unobserved confounding compromises causal conclusions from non-randomized studies.
method Designs a statistical test to detect unobserved confounding strength and estimates a lower bound.
result Estimates an asymptotically valid lower bound on unobserved confounding strength.
Adaptive auditing improves AI robustness testing with anytime-valid guarantees.
problem Cost and time of annotation limit rigorous AI failure mode characterization.
method Introduces hypothesis testing framework for adaptive audits using SAVI.
result Proves anytime-valid type-I error control and robustness certification.
Targeted Learning uses robust statistics for reproducible research.
problem Improving reproducibility and rigor in statistical analyses.
method Principled standard for statistical estimation and inference, minimizing assumptions.
result Enhances reliability of statistical conclusions.
Over the past two decades, several consistent procedures have been designed to infer causal conclusions from observational data. We prove that if the true causal network might be an arbitrary, linear Gaussian network or a discrete Bayes network, then every unambiguous causal conclusion produced by a consistent method f…
We apply basic statistical reasoning to signal reconstruction by machine learning -- learning to map corrupted observations to clean signals -- with a simple and powerful conclusion: it is possible to learn to restore images by only looking at corrupted examples, at performance at and sometimes exceeding training using…
New method for valid inference from ML-predicted data.
problem Invalid scientific conclusions from ML-predicted outcomes.
method Assumption-Lean and Data-Adaptive Post-Prediction Inference (PSPA)
result Valid and powerful inference based on ML-predicted data.
The paper extends hypothesis testing to non-diagonalizable matrices, improving network statistics inference.
problem Testing on non-diagonalizable matrices for network statistics.
method Generalizes Wald and t-tests to non-symmetric matrices, controlling convergence rates.
result Improved inference on network statistics from directed networks.
The common assumption of universal behavior in stock market data can sometimes lead to false conclusions. In statistical physics, the Hurst exponents characterizing long-range correlations are often closely related to universal exponents. We show, that in the case of time series of the traded value, these Hurst exponen…
The paper analyzes optimal statistical arbitrage strategies for co-integrated stocks.
problem Finding optimal portfolio weights for co-integrated stocks.
method Solving a Hamilton-Jacobi-Bellman (HJB) partial differential equation for optimal portfolio weights.
result The proposed co-integrated model with eigenportfolios can generate stable growth rates over a long time horizon.
Choosing a reference group in Oaxaca-Blinder decomposition can reverse conclusions.
problem The choice of reference group in Oaxaca-Blinder decomposition can lead to different conclusions.
method The study uses the Oaxaca-Blinder decomposition to investigate how the choice of reference group affects the results.
result The Oaxaca-Blinder decomposition can yield different conclusions based on the choice of reference group.
ML reduces high-dimensional data to reveal its underlying structure.
problem Handling large, high-dimensional data sets.
method Non-linear dimension reduction techniques.
result Reveals the geometric shape of high-dimensional data.
DAGnosis uses DAGs to identify and localize data inconsistencies.
problem Handling data inconsistencies in machine learning models at deployment time.
method Directed acyclic graphs (DAGs) to encode feature probability distribution and independencies.
result Localization of inconsistencies and insights into their causes.
The study improves the assessment of fairness in face recognition using ROC curves and statistical guarantees.
problem Improving the assessment of fairness in face recognition systems.
method Proves asymptotic guarantees for empirical ROC curves and fairness metrics, and introduces a recentering technique to avoid bootstrap pitfalls.
result Demonstrates the practical relevance of the methods for assessing fairness in face recognition systems.
Because of the prominent position of urban rail in reducing urban transport-related problems, such as congestion and air pollution, insights into the costs of possible new urban rail projects is very relevant for those involved with cost estimations, policy makers, cost-benefit analysts, and other target groups. Knowle…
Understanding the causes of crime is a longstanding issue in researcher's agenda. While it is a hard task to extract causality from data, several linear models have been proposed to predict crime through the existing correlations between crime and urban metrics. However, because of non-Gaussian distributions and multic…
Researchers formalize PD and PFI to relate them to data generating process.
problem Lack of theory linking PD and PFI to data generating process.
method Formalize PD and PFI as estimators of ground truth estimands, account for model variance with learner-PD and learner-PFI.
result PD and PFI estimates deviate from ground truth due to statistical biases, model variance, and Monte Carlo approximation errors.
This study analyzes factors affecting China's stock market volatility.
problem Investigating factors influencing China's stock market volatility.
method Used ARDL model and principal component analysis on daily data from 2010-2024.
result Exchange rate and bond yields significantly impact stock market volatility.
Maximum regularized likelihood estimators (MRLEs) are arguably the most established class of estimators in high-dimensional statistics. In this paper, we derive guarantees for MRLEs in Kullback-Leibler divergence, a general measure of prediction accuracy. We assume only that the densities have a convex parametrization …
We present an overview of some representative Agent-Based Models in Economics. We discuss why and how agent-based models represent an important step in order to explain the dynamics and the statistical properties of financial markets beyond the Classical Theory of Economics. We perform a schematic analysis of several m…
Study shows heavy-tailed distributions affect reliability of machine learning calibration statistics.
problem Reliability of calibration statistics for machine learning regression tasks is affected by heavy-tailed uncertainty and error distributions.
method Examined two calibration error estimation methods (CE and ZMS) and found ZMS to be less sensitive to heavy-tailed distributions.
result Heavy-tailed distributions make MSE and MV unreliable, but ZMS remains a reliable approach.
Traditional statistical theory assumes that the analysis to be performed on a given data set is selected independently of the data themselves. This assumption breaks downs when data are re-used across analyses and the analysis to be performed at a given stage depends on the results of earlier stages. Such dependency ca…
Meta-Surrogate model speeds up HPO benchmarking.
problem Limited and expensive real-world HPO benchmarks.
method Meta-surrogate model trained on off-line data.
result More coherent and statistically significant conclusions faster.
seMCD computes depth functions with statistical guarantees using sequential Monte Carlo.
problem Computing depth functions is computationally challenging, especially in high dimensions.
method Sequential Monte Carlo methodology with theoretical and empirical guarantees.
result The seMCD method provides accurate depth approximations with fewer samples than traditional methods.
Quantile TD learning outperforms classical TD learning for value estimation.
problem Temporal-difference learning in reinforcement learning.
method Quantile Temporal-Difference Learning (QTD) for policy evaluation.
result QTD offers superior performance to classical TD learning, even in tabular settings.
It is usual to consider data protection and learnability as conflicting objectives. This is not always the case: we show how to jointly control inference --- seen as the attack --- and learnability by a noise-free process that mixes training examples, the Crossover Process (cp). One key point is that the cp~is typicall…
Machine learning experiments often contain errors, especially in confusion matrices and statistical tests.
problem Errors in machine learning experiments, particularly in confusion matrices and statistical tests.
method Analyzed 49 papers describing 2456 experiments, checking for errors in confusion matrices and statistical significance.
result 22 out of 49 papers contain demonstrable errors, with 7 statistical and 16 related to confusion matrix inconsistency.
STAND-DA improves AD in DA target domains with limited data.
problem Statistical validity of AD after DA with limited data.
method Selective Inference framework for GPU-accelerated p-value computation. result Valid p-values and controlled false positive rate. Paper develops a method to compare generative models using KL divergence.
problem Lack of principled uncertainty quantification for generative models.
method Employ Kullback-Leibler divergence to measure generative model distance.
result Effective coverage rates and higher power compared to kernel-based methods.
The stability of statistical analysis is an important indicator for reproducibility, which is one main principle of scientific method. It entails that similar statistical conclusions can be reached based on independent samples from the same underlying population. In this paper, we introduce a general measure of classif…
This survey explores causal inference in banking, finance, and insurance.
problem Explaining decisions in banking, finance, and insurance using causal inference.
method Categorizes 37 papers on causal inference applications in banking, finance, and insurance.
result Causal inference is still in its infancy in banking and insurance sectors.
Study improves statistical power for detecting algorithmic bias in educational data.
problem Challenges in measuring algorithmic bias using ABROCA due to skewed distribution.
method Investigates ABROCA's distributional properties and proposes nonparametric randomization tests.
result ABROCA-based bias assessments are underpowered in typical EDM sample sizes.
This paper develops embeddings that preserve likelihood-based statistical inference.
problem Modern machine learning embeddings destroy the geometric structure required for likelihood-based inference.
method Developed a rigorous theory of likelihood-preserving embeddings and introduced the Likelihood-Ratio Distortion metric.
result Controlling the distortion Δn is necessary and sufficient for preserving inference.