Model-agnostic interpretation methods can mislead if not used carefully.
problem Misinterpretation of machine learning models due to improper use of techniques.
method General pitfalls of model-agnostic interpretation methods.
result Many pitfalls exist when using global interpretation techniques for machine learning models.
Study pitfalls of deep learning ensembles in uncertainty estimation.
problem Pitfalls in in-domain uncertainty estimation and ensembling in deep learning.
method Exploration of standards for uncertainty quantification and broad study of ensembling techniques.
result Many sophisticated ensembling techniques are equivalent to a simple ensemble of few networks.
Machine learning suffers from poor design, data, and evaluation practices.
problem Ad hoc design, poor data hygiene, and lack of statistical rigor in model evaluation.
method Examines the entire machine learning process from design to evaluation, highlighting common pitfalls and providing recommendations.
result Common pitfalls in machine learning research and development are identified and actionable recommendations are provided.
This paper evaluates metrics for graph generative models, addressing common pitfalls.
problem Evaluating and comparing graph generative models effectively.
method Systematic evaluation of MMD, analysis of synthetic and real graphs, practical recommendations.
result MMD can be problematic; practical solutions are provided.
Machine Learning (ML) and Deep Learning (DL) innovations are being introduced at such a rapid pace that model owners and evaluators are hard-pressed analyzing and studying them. This is exacerbated by the complicated procedures for evaluation. The lack of standard systems and efficient techniques for specifying and pro…
Survey of topological 4-manifold theory, highlighting foundational theorems and pitfalls.
problem Understanding the topological properties of 4-dimensional manifolds.
method Compilation and explanation of foundational theorems, with cautionary notes.
result Many intuitive results in differential topology are not true in the topological category.
The paper addresses pitfalls in calibrating option pricing models using deep learning.
problem Calibrating option pricing models to market data using deep learning.
method Identifies and resolves issues in existing approaches, improving model performance and accuracy.
result Proposes solutions that enhance the accuracy and performance of deep learning models in option pricing.
Study identifies pitfalls in assessing hierarchies for multi-class classification.
problem Lack of understanding in selecting hierarchies for multi-class classification.
method Analyzed and compared popular approaches to extracting hierarchies.
result Hierarchy quality becomes irrelevant when using powerful classifiers.
Benchmark assesses LLMs' causal inference skills, revealing significant limitations.
problem Lack of rigorous evaluation of LLMs' causal inference capabilities.
method CausalPitfalls benchmark with structured challenges and grading rubrics.
result Significant limitations in current LLMs' statistical causal inference.
Researchers share pitfalls and improvements in implementing Hinton's capsule network.
problem Implementation pitfalls in Hinton's capsule network hindered progress in the field.
method Identifying and addressing common mistakes in capsule network implementations.
result Improved implementation of Hinton's capsule network outperforms existing open-source implementations.
New categorization of community detection methods to avoid pitfalls.
problem Misuse of community detection methods in practice.
method Descriptive vs. inferential approaches.
result Inferential methods provide clearer insights into network formation.
Bayesian optimization improves molecule design by addressing three pitfalls.
problem Bayesian optimization pitfalls cause poor performance in molecule design.
method Identified and addressed three pitfalls: incorrect prior width, over-smoothing, and inadequate acquisition function maximization.
result Basic BO setup achieves highest performance on PMO benchmark.
Unified framework for ranking-and-selection with multiple correct answers and non-answerable estimates
problem Fixed-precision ranking-and-selection in structured settings with non-unique answers and non-answerable estimates
method Unified framework based on answer-wise acceptance sets, restricted generalized likelihood ratio stopping, and answer-pitfall decomposition
result Unified recipe performs well across a broad range of pure-exploration problems
This paper improves conditional sampling for VAEs by overcoming structural issues.
problem Computational intractability of conditional sampling in VAEs.
method Proposes two methods to address pitfalls in Metropolis-within-Gibbs (MWG) for VAEs.
result Improved performance on sampling tasks.
This paper evaluates test selection methods for deep neural networks, revealing their limitations.
problem Challenges in testing deep learning systems due to high labeling costs.
method Analysis and empirical testing of 11 test selection methods on five datasets.
result Test selection methods can fail under certain conditions, leading to significant drops in test relative coverage.
Deep learning applied to SAR data is explored in this paper.
problem Limited use of deep learning in SAR data processing.
method Introduction to relevant deep learning models, analysis of SAR data characteristics, review of state-of-the-art applications, and future research directions.
result Unlocking the potential of deep learning in SAR data processing.
Humorous article on pitfalls in machine learning research
problem Difficulty in reproducing and validating machine learning results
method Discussion of common issues in machine learning papers
result Highlighting watch out points for researchers
This paper highlights the overlooked role of preprocessing hyperparameters in machine learning model performance.
problem The neglect of preprocessing hyperparameters in machine learning model evaluation.
method Empirical review and illustration of different procedures for generating and evaluating prediction models.
result Users may fail to report or account for preprocessing hyperparameter optimization, leading to exaggerated performance claims.
Cross-validation pitfalls in change-point regression are addressed with new approaches.
problem Cross-validation's prediction error-based criterion may lead to under- or over-estimation of change-points.
method Proposes two approaches: absolute error loss and modified holdout sets.
result Consistent estimation of the number of change-points under certain conditions.
New findings show a balance between data fit and complexity in kernel hyperparameters.
problem Overcorrelation due to reparametrization of kernel hyperparameters.
method Reparametrization of kernel hyperparameters and analysis of marginal likelihood.
result Data fit term influences all other kernel hyperparameters, not just the complexity penalty.
LLA shows strong performance in Bayesian optimization but has unbounded search space issues.
problem Applying LLA in unbounded search spaces for Bayesian optimization.
method Linearized-Laplace approximation applied to Bayesian optimization problems.
result LLA demonstrates strong performance but also presents unbounded search space challenges.
Success in the quest for artificial intelligence has the potential to bring unprecedented benefits to humanity, and it is therefore worthwhile to investigate how to maximize these benefits while avoiding potential pitfalls. This article gives numerous examples (which should by no means be construed as an exhaustive lis…
Go-Explore improves performance on hard-exploration problems in Atari games.
problem Challenges in reinforcement learning, especially with sparse or deceptive rewards.
method Exploits principles of remembering states, returning to promising states, and solving simulated environments.
result Scores significantly higher than previous state-of-the-art on Montezuma's Revenge and Pitfall.
New datasets reveal neural networks can rely on simple features, leading to poor generalization.
problem Neural networks' reliance on simple features can lead to poor generalization and robustness.
method Designing datasets with varying levels of simplicity and incorporating non-robustness.
result Neural networks can exclusively rely on the simplest feature, leading to poor performance on complex data.
Machine learning methods have gained a great deal of popularity in recent years among public administration scholars and practitioners. These techniques open the door to the analysis of text, image and other types of data that allow us to test foundational theories of public administration and to develop new theories. …
Proposes a new undersampling method for imbalanced data classification.
problem Challenges of oversampling and undersampling in imbalanced data.
method Bilevel optimization framework for identifying optimal subset of majority training data.
result Improves F1 scores by up to 10% compared to state-of-the-art methods.
Statistical analysis reveals pitfalls in climate network construction.
problem Uncertainty in correlation estimates affects network characteristics.
method Simulated isotropic random fields and resampling procedures.
result Spurious link bundles and high-degree clusters are common.
TBAL reduces manual annotation but requires validated data.
problem Creating large, high-quality labeled datasets.
method Threshold-based auto-labeling using human validation data.
result Sample complexity bounds on validation data needed.
This document provides a tutorial description of the use of the MDL principle in complex graph analysis. We give a brief summary of the preliminary subjects, and describe the basic principle, using the example of analysing the size of the largest clique in a graph. We also provide a discussion of how to interpret the r…
Statistical tests for fairness in admissions data reveal hidden patterns.
problem Simpson's paradox in admissions data hides true gender bias.
method Introduces a new statistical test based on Pearl's instrumental-variable inequalities.
result Statistical tests for fairness coincide with causal notions for the Berkeley admissions case.
Paper proposes an ensemble of attacks to evaluate adversarial robustness more reliably.
problem Insufficient evaluation of adversarial defenses leads to incorrect robustness assessments.
method Developed two extensions of PGD-attack and combined them with two complementary attacks.
result Identified several broken defenses with lower robust test accuracy than reported.
This document is an invited chapter covering the specificities of ABC model choice, intended for the incoming Handbook of ABC by Sisson, Fan, and Beaumont (2017). Beyond exposing the potential pitfalls of ABC based posterior probabilities, the review emphasizes mostly the solution proposed by Pudlo et al. (2016) on the…
Framework adds human knowledge to AI decisions to improve outcomes.
problem Conflict between AI recommendations and human insights.
method Develops a framework for integrating human knowledge as a guardrail for AI decisions.
result Human knowledge can improve AI decisions, especially in specific pitfalls.
This letter uses the Block Maxima Extreme Value approach to quantify catastrophic risk in international equity markets. Risk measures are generated from a set threshold of the distribution of returns that avoids the pitfall of using absolute returns for markets exhibiting diverging levels of risk. From an application t…
We construct Zero-Coupon Bond markets driven by a cylindrical Brownian motion in which the notion of generalized portfolio has important flaws: There exist bounded smooth random variables with generalized hedging portfolios for which the price of their risky part is +∞ at each time. For these generalized portfol…
Abstract MDPs enable strategic exploration and fast reward transfer in complex environments.
problem Challenging to learn accurate MDPs for high-dimensional states.
method Learn an abstract MDP over low-dimensional coarse states, using an abstraction function.
result Achieves superhuman performance on Pitfall! and higher reward with fewer samples.
In this paper we sketch some reflections on the pitfalls and inconsistencies of the research program - currently dominant among the profession - aimed at providing microfoundations to macroeconomics along a Walrasian perspective. We argue that such a methodological approach constitutes an unsatisfactory answer to a wel…
We highlight a pitfall when applying stochastic variational inference to general Bayesian networks. For global random variables approximated by an exponential family distribution, natural gradient steps, commonly starting from a unit length step size, are averaged to convergence. This useful insight into the scaling of…
When faced with high frequency streams of data, clustering raises theoretical and algorithmic pitfalls. We introduce a new and adaptive online clustering algorithm relying on a quasi-Bayesian approach, with a dynamic (i.e., time-dependent) estimation of the (unknown and changing) number of clusters. We prove that our a…
The purpose of these notes is to provide a systematic quantitative framework - in what is intended to be a "pedagogical" fashion - for discussing mean-reversion and optimization. We start with pair trading and add complexity by following the sequence "mean-reversion via demeaning -> regression -> weighted regression ->…
The success of modern Artificial Intelligence (AI) technologies depends critically on the ability to learn non-linear functional dependencies from large, high dimensional data sets. Despite recent high-profile successes, empirical evidence indicates that the high predictive performance is often paired with low robustne…
Machine learning aids excited-state molecular dynamics studies.
problem Challenges in studying electronically excited states of molecules.
method Employing machine learning techniques for excited-state molecular dynamics.
result Highlight successes and challenges in machine learning for excited-state processes.
When confronted with massive data streams, summarizing data with dimension reduction methods such as PCA raises theoretical and algorithmic pitfalls. Principal curves act as a nonlinear generalization of PCA and the present paper proposes a novel algorithm to automatically and sequentially learn principal curves from d…
There is an increasing interest in estimating expectations outside of the classical inference framework, such as for models expressed as probabilistic programs. Many of these contexts call for some form of nested inference to be applied. In this paper, we analyse the behaviour of nested Monte Carlo (NMC) schemes, for w…
Paper explores limitations of generative models in finance, proposing a new method for portfolio generation.
problem Challenges in applying generative models to financial portfolio and risk management.
method Theoretical analysis and empirical testing of generative models, proposing a new pipeline for multivariate return generation.
result A new method for generating multivariate returns that meets portfolio evaluation standards and avoids pitfalls.
GrowNet uses shallow neural networks for gradient boosting, outperforming existing methods.
problem Improving gradient boosting performance through shallow neural networks.
method Unified gradient boosting framework with shallow neural networks as weak learners, incorporating corrective steps.
result GrowNet outperformed state-of-the-art boosting methods in classification, regression, and learning to rank tasks.
In this paper, we briefly review the basic scheme of the pseudoinverse learning (PIL) algorithm and present some discussions on the PIL, as well as its variants. The PIL algorithm, first presented in 1995, is a non-gradient descent and non-iterative learning algorithm for multi-layer neural networks and has several adv…
In this paper we propose a novel approach for learning from data using rule based fuzzy inference systems where the model parameters are estimated using Bayesian inference and Markov Chain Monte Carlo (MCMC) techniques. We show the applicability of the method for regression and classification tasks using synthetic data…