This paper evaluates metrics for graph generative models, addressing common pitfalls.
problem Evaluating and comparing graph generative models effectively.
method Systematic evaluation of MMD, analysis of synthetic and real graphs, practical recommendations.
result MMD can be problematic; practical solutions are provided.
Machine learning suffers from poor design, data, and evaluation practices.
problem Ad hoc design, poor data hygiene, and lack of statistical rigor in model evaluation.
method Examines the entire machine learning process from design to evaluation, highlighting common pitfalls and providing recommendations.
result Common pitfalls in machine learning research and development are identified and actionable recommendations are provided.
Machine Learning (ML) and Deep Learning (DL) innovations are being introduced at such a rapid pace that model owners and evaluators are hard-pressed analyzing and studying them. This is exacerbated by the complicated procedures for evaluation. The lack of standard systems and efficient techniques for specifying and pro…
Benchmark assesses LLMs' causal inference skills, revealing significant limitations.
problem Lack of rigorous evaluation of LLMs' causal inference capabilities.
method CausalPitfalls benchmark with structured challenges and grading rubrics.
result Significant limitations in current LLMs' statistical causal inference.
This paper evaluates test selection methods for deep neural networks, revealing their limitations.
problem Challenges in testing deep learning systems due to high labeling costs.
method Analysis and empirical testing of 11 test selection methods on five datasets.
result Test selection methods can fail under certain conditions, leading to significant drops in test relative coverage.
This paper highlights the overlooked role of preprocessing hyperparameters in machine learning model performance.
problem The neglect of preprocessing hyperparameters in machine learning model evaluation.
method Empirical review and illustration of different procedures for generating and evaluating prediction models.
result Users may fail to report or account for preprocessing hyperparameter optimization, leading to exaggerated performance claims.
Paper proposes an ensemble of attacks to evaluate adversarial robustness more reliably.
problem Insufficient evaluation of adversarial defenses leads to incorrect robustness assessments.
method Developed two extensions of PGD-attack and combined them with two complementary attacks.
result Identified several broken defenses with lower robust test accuracy than reported.
Model-agnostic interpretation methods can mislead if not used carefully.
problem Misinterpretation of machine learning models due to improper use of techniques.
method General pitfalls of model-agnostic interpretation methods.
result Many pitfalls exist when using global interpretation techniques for machine learning models.
Study pitfalls of deep learning ensembles in uncertainty estimation.
problem Pitfalls in in-domain uncertainty estimation and ensembling in deep learning.
method Exploration of standards for uncertainty quantification and broad study of ensembling techniques.
result Many sophisticated ensembling techniques are equivalent to a simple ensemble of few networks.
Correctly evaluating defenses against adversarial examples has proven to be extremely difficult. Despite the significant amount of recent work attempting to design defenses that withstand adaptive attacks, few have succeeded; most papers that propose defenses are quickly shown to be incorrect. We believe a large contri…
Deep learning applied to SAR data is explored in this paper.
problem Limited use of deep learning in SAR data processing.
method Introduction to relevant deep learning models, analysis of SAR data characteristics, review of state-of-the-art applications, and future research directions.
result Unlocking the potential of deep learning in SAR data processing.
This work studies the problem of batch off-policy evaluation for Reinforcement Learning in partially observable environments. Off-policy evaluation under partial observability is inherently prone to bias, with risk of arbitrarily large errors. We define the problem of off-policy evaluation for Partially Observable Mark…
Statistical analysis reveals pitfalls in climate network construction.
problem Uncertainty in correlation estimates affects network characteristics.
method Simulated isotropic random fields and resampling procedures.
result Spurious link bundles and high-degree clusters are common.
Kernel quadrature improves CRPS estimation for probabilistic time-series forecasting.
problem Intractable integrations in CRPS evaluation metrics lead to improper rankings of forecasting models.
method Introduced kernel quadrature approach for unbiased CRPS estimation and scalable computation.
result Our approach consistently outperforms existing CRPS estimators.
Study identifies pitfalls in assessing hierarchies for multi-class classification.
problem Lack of understanding in selecting hierarchies for multi-class classification.
method Analyzed and compared popular approaches to extracting hierarchies.
result Hierarchy quality becomes irrelevant when using powerful classifiers.
New datasets reveal neural networks can rely on simple features, leading to poor generalization.
problem Neural networks' reliance on simple features can lead to poor generalization and robustness.
method Designing datasets with varying levels of simplicity and incorporating non-robustness.
result Neural networks can exclusively rely on the simplest feature, leading to poor performance on complex data.
Semi-supervised node classification in graphs is a fundamental problem in graph mining, and the recently proposed graph neural networks (GNNs) have achieved unparalleled results on this task. Due to their massive success, GNNs have attracted a lot of attention, and many novel architectures have been put forward. In thi…
Paper explores limitations of generative models in finance, proposing a new method for portfolio generation.
problem Challenges in applying generative models to financial portfolio and risk management.
method Theoretical analysis and empirical testing of generative models, proposing a new pipeline for multivariate return generation.
result A new method for generating multivariate returns that meets portfolio evaluation standards and avoids pitfalls.
New categorization of community detection methods to avoid pitfalls.
problem Misuse of community detection methods in practice.
method Descriptive vs. inferential approaches.
result Inferential methods provide clearer insights into network formation.
Bayesian optimization improves molecule design by addressing three pitfalls.
problem Bayesian optimization pitfalls cause poor performance in molecule design.
method Identified and addressed three pitfalls: incorrect prior width, over-smoothing, and inadequate acquisition function maximization.
result Basic BO setup achieves highest performance on PMO benchmark.
Unified framework for ranking-and-selection with multiple correct answers and non-answerable estimates
problem Fixed-precision ranking-and-selection in structured settings with non-unique answers and non-answerable estimates
method Unified framework based on answer-wise acceptance sets, restricted generalized likelihood ratio stopping, and answer-pitfall decomposition
result Unified recipe performs well across a broad range of pure-exploration problems
This guide explains statistical distances for evaluating generative models.
problem Evaluating the quality of samples from generative models.
method Four statistical distances: SW, C2ST, MMD, FID.
result Different distances can yield varying results on similar data.
Study explores efficient continual learning of diffusion models.
problem Efficiently reusing computation in training diffusion models as data evolves.
method Benchmarked and analyzed CL methods for Denoising Diffusion Probabilistic Models (DDPMs).
result Experience replay with reduced rehearsal coefficient shows strong performance.
Bayesian optimization has emerged as a strong candidate tool for global optimization of functions with expensive evaluation costs. However, due to the dynamic nature of research in Bayesian approaches, and the evolution of computing technology, using Bayesian optimization in a parallel computing environment remains a c…
This paper improves conditional sampling for VAEs by overcoming structural issues.
problem Computational intractability of conditional sampling in VAEs.
method Proposes two methods to address pitfalls in Metropolis-within-Gibbs (MWG) for VAEs.
result Improved performance on sampling tasks.
Recent progress in the field of artificial intelligence, machine learning and also in computer industry resulted in the ongoing boom of using these techniques as applied to solving complex tasks in both science and industry. Same is, of course, true for the financial industry and mathematical finance. In this paper we …
Generative adversarial networks (GANs) are a class of deep generative models which aim to learn a target distribution in an unsupervised fashion. While they were successfully applied to many problems, training a GAN is a notoriously challenging task and requires a significant number of hyperparameter tuning, neural arc…
Neural network pruning lacks standardized benchmarks and metrics.
problem Lack of standardized benchmarks and metrics in neural network pruning.
method Meta-analysis of 81 papers, controlled conditions, ShrinkBench framework.
result Neural network pruning community lacks standardized benchmarks and metrics.
Cross-validation pitfalls in change-point regression are addressed with new approaches.
problem Cross-validation's prediction error-based criterion may lead to under- or over-estimation of change-points.
method Proposes two approaches: absolute error loss and modified holdout sets.
result Consistent estimation of the number of change-points under certain conditions.
New findings show a balance between data fit and complexity in kernel hyperparameters.
problem Overcorrelation due to reparametrization of kernel hyperparameters.
method Reparametrization of kernel hyperparameters and analysis of marginal likelihood.
result Data fit term influences all other kernel hyperparameters, not just the complexity penalty.
A new neural network model simulates financial markets without assuming underlying dynamics.
problem Modeling financial time series without assuming underlying dynamics.
method Neural network based generative model using a parsimonious Variational Autoencoder framework.
result Works reliably in small data environments, providing a new performance evaluation metric.
Paper evaluates and proposes a new method for robust curvature estimation from noisy point cloud data.
problem Challenges in discovering a robust manifold structure in point cloud data.
method Proposes a new robust curvature estimation method based on Weingarten map.
result Demonstrates superior performance of the new method in noisy data conditions.
LLMs improve stock price forecasting from financial news and reports.
problem Predicting stock prices with high accuracy and robustness.
method Analyzing financial news, reports, and transcripts using LLMs.
result LLMs can improve stock price forecasting but face practical challenges.
LLA shows strong performance in Bayesian optimization but has unbounded search space issues.
problem Applying LLA in unbounded search spaces for Bayesian optimization.
method Linearized-Laplace approximation applied to Bayesian optimization problems.
result LLA demonstrates strong performance but also presents unbounded search space challenges.
Unified and extended precision-recall metrics for generative models.
problem Evaluation of generative models using scalar metrics like FID and IS.
method Unified various precision-recall approaches under a common framework.
result Recovered entire precision-recall curves and exposed metric pitfalls.
Bayesian quadrature (BQ) is a sample-efficient probabilistic numerical method to solve integrals of expensive-to-evaluate black-box functions, yet so far,active BQ learning schemes focus merely on the integrand itself as information source, and do not allow for information transfer from cheaper, related functions. Here…
The study analyzes how probabilistic forecasts improve battery trading strategies in electricity markets.
problem Improvements in statistical forecast quality do not directly translate to economic value in battery trading strategies.
method The study frames battery optimization as a stochastic program based on fully probabilistic forecasts and examines decision quality under different uncertainty models.
result The study identifies two critical flaws in quantile-based trading strategies and provides theoretical justification and empirical evidence.
New approach improves convergence of online learning for ARIMA models.
problem Pitfalls in computational complexity and convergence of Online Newton Step optimization method.
method Proposes a new approach combining different Online Gradient Descent learners for iterative training of ARIMA models.
result The proposed approach outperforms existing methods resulting in lower prediction error.
Success in the quest for artificial intelligence has the potential to bring unprecedented benefits to humanity, and it is therefore worthwhile to investigate how to maximize these benefits while avoiding potential pitfalls. This article gives numerous examples (which should by no means be construed as an exhaustive lis…
Machine learning methods have gained a great deal of popularity in recent years among public administration scholars and practitioners. These techniques open the door to the analysis of text, image and other types of data that allow us to test foundational theories of public administration and to develop new theories. …
Proposes a new undersampling method for imbalanced data classification.
problem Challenges of oversampling and undersampling in imbalanced data.
method Bilevel optimization framework for identifying optimal subset of majority training data.
result Improves F1 scores by up to 10% compared to state-of-the-art methods.
This document provides a tutorial description of the use of the MDL principle in complex graph analysis. We give a brief summary of the preliminary subjects, and describe the basic principle, using the example of analysing the size of the largest clique in a graph. We also provide a discussion of how to interpret the r…
TBAL reduces manual annotation but requires validated data.
problem Creating large, high-quality labeled datasets.
method Threshold-based auto-labeling using human validation data.
result Sample complexity bounds on validation data needed.
Improved method for unbiased causal discovery in presence of unobserved confounding.
problem Unbiased data synthesis for causal discovery algorithms in the presence of unobserved confounding.
method Explicit block-hierarchical ancestral sampling to address limitations of implicit parameterization.
result Our approach fully covers the space of causal models, including those generated by implicit parameterization.
Statistical tests for fairness in admissions data reveal hidden patterns.
problem Simpson's paradox in admissions data hides true gender bias.
method Introduces a new statistical test based on Pearl's instrumental-variable inequalities.
result Statistical tests for fairness coincide with causal notions for the Berkeley admissions case.
This document is an invited chapter covering the specificities of ABC model choice, intended for the incoming Handbook of ABC by Sisson, Fan, and Beaumont (2017). Beyond exposing the potential pitfalls of ABC based posterior probabilities, the review emphasizes mostly the solution proposed by Pudlo et al. (2016) on the…
This letter uses the Block Maxima Extreme Value approach to quantify catastrophic risk in international equity markets. Risk measures are generated from a set threshold of the distribution of returns that avoids the pitfall of using absolute returns for markets exhibiting diverging levels of risk. From an application t…
Framework adds human knowledge to AI decisions to improve outcomes.
problem Conflict between AI recommendations and human insights.
method Develops a framework for integrating human knowledge as a guardrail for AI decisions.
result Human knowledge can improve AI decisions, especially in specific pitfalls.