Research
On-device research index

arXiv research

A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.

168,695 papers · 148 categories

Trend · papers per month

2468 · Jun 202019922001200920172026
48 results for pitfalls

Model-agnostic interpretation methods can mislead if not used carefully.

problem Misinterpretation of machine learning models due to improper use of techniques.
method General pitfalls of model-agnostic interpretation methods.
result Many pitfalls exist when using global interpretation techniques for machine learning models.

Study pitfalls of deep learning ensembles in uncertainty estimation.

problem Pitfalls in in-domain uncertainty estimation and ensembling in deep learning.
method Exploration of standards for uncertainty quantification and broad study of ensembling techniques.
result Many sophisticated ensembling techniques are equivalent to a simple ensemble of few networks.

Machine learning suffers from poor design, data, and evaluation practices.

problem Ad hoc design, poor data hygiene, and lack of statistical rigor in model evaluation.
method Examines the entire machine learning process from design to evaluation, highlighting common pitfalls and providing recommendations.
result Common pitfalls in machine learning research and development are identified and actionable recommendations are provided.

This paper evaluates metrics for graph generative models, addressing common pitfalls.

problem Evaluating and comparing graph generative models effectively.
method Systematic evaluation of MMD, analysis of synthetic and real graphs, practical recommendations.
result MMD can be problematic; practical solutions are provided.

Machine Learning (ML) and Deep Learning (DL) innovations are being introduced at such a rapid pace that model owners and evaluators are hard-pressed analyzing and studying them. This is exacerbated by the complicated procedures for evaluation. The lack of standard systems and efficient techniques for specifying and pro…

2018-11-24abs ↗pdf ↗

Study identifies pitfalls in assessing hierarchies for multi-class classification.

problem Lack of understanding in selecting hierarchies for multi-class classification.
method Analyzed and compared popular approaches to extracting hierarchies.
result Hierarchy quality becomes irrelevant when using powerful classifiers.

Benchmark assesses LLMs' causal inference skills, revealing significant limitations.

problem Lack of rigorous evaluation of LLMs' causal inference capabilities.
method CausalPitfalls benchmark with structured challenges and grading rubrics.
result Significant limitations in current LLMs' statistical causal inference.

Bayesian optimization improves molecule design by addressing three pitfalls.

problem Bayesian optimization pitfalls cause poor performance in molecule design.
method Identified and addressed three pitfalls: incorrect prior width, over-smoothing, and inadequate acquisition function maximization.
result Basic BO setup achieves highest performance on PMO benchmark.

Unified framework for ranking-and-selection with multiple correct answers and non-answerable estimates

problem Fixed-precision ranking-and-selection in structured settings with non-unique answers and non-answerable estimates
method Unified framework based on answer-wise acceptance sets, restricted generalized likelihood ratio stopping, and answer-pitfall decomposition
result Unified recipe performs well across a broad range of pure-exploration problems

This paper evaluates test selection methods for deep neural networks, revealing their limitations.

problem Challenges in testing deep learning systems due to high labeling costs.
method Analysis and empirical testing of 11 test selection methods on five datasets.
result Test selection methods can fail under certain conditions, leading to significant drops in test relative coverage.

This paper highlights the overlooked role of preprocessing hyperparameters in machine learning model performance.

problem The neglect of preprocessing hyperparameters in machine learning model evaluation.
method Empirical review and illustration of different procedures for generating and evaluating prediction models.
result Users may fail to report or account for preprocessing hyperparameter optimization, leading to exaggerated performance claims.

Cross-validation pitfalls in change-point regression are addressed with new approaches.

problem Cross-validation's prediction error-based criterion may lead to under- or over-estimation of change-points.
method Proposes two approaches: absolute error loss and modified holdout sets.
result Consistent estimation of the number of change-points under certain conditions.

New findings show a balance between data fit and complexity in kernel hyperparameters.

problem Overcorrelation due to reparametrization of kernel hyperparameters.
method Reparametrization of kernel hyperparameters and analysis of marginal likelihood.
result Data fit term influences all other kernel hyperparameters, not just the complexity penalty.

LLA shows strong performance in Bayesian optimization but has unbounded search space issues.

problem Applying LLA in unbounded search spaces for Bayesian optimization.
method Linearized-Laplace approximation applied to Bayesian optimization problems.
result LLA demonstrates strong performance but also presents unbounded search space challenges.

Success in the quest for artificial intelligence has the potential to bring unprecedented benefits to humanity, and it is therefore worthwhile to investigate how to maximize these benefits while avoiding potential pitfalls. This article gives numerous examples (which should by no means be construed as an exhaustive lis…

2016-02-10abs ↗pdf ↗

New datasets reveal neural networks can rely on simple features, leading to poor generalization.

problem Neural networks' reliance on simple features can lead to poor generalization and robustness.
method Designing datasets with varying levels of simplicity and incorporating non-robustness.
result Neural networks can exclusively rely on the simplest feature, leading to poor performance on complex data.

This document provides a tutorial description of the use of the MDL principle in complex graph analysis. We give a brief summary of the preliminary subjects, and describe the basic principle, using the example of analysing the size of the largest clique in a graph. We also provide a discussion of how to interpret the r…

2018-10-31abs ↗pdf ↗

Statistical tests for fairness in admissions data reveal hidden patterns.

problem Simpson's paradox in admissions data hides true gender bias.
method Introduces a new statistical test based on Pearl's instrumental-variable inequalities.
result Statistical tests for fairness coincide with causal notions for the Berkeley admissions case.

Paper proposes an ensemble of attacks to evaluate adversarial robustness more reliably.

problem Insufficient evaluation of adversarial defenses leads to incorrect robustness assessments.
method Developed two extensions of PGD-attack and combined them with two complementary attacks.
result Identified several broken defenses with lower robust test accuracy than reported.

This document is an invited chapter covering the specificities of ABC model choice, intended for the incoming Handbook of ABC by Sisson, Fan, and Beaumont (2017). Beyond exposing the potential pitfalls of ABC based posterior probabilities, the review emphasizes mostly the solution proposed by Pudlo et al. (2016) on the…

2015-03-26abs ↗pdf ↗

Abstract MDPs enable strategic exploration and fast reward transfer in complex environments.

problem Challenging to learn accurate MDPs for high-dimensional states.
method Learn an abstract MDP over low-dimensional coarse states, using an abstraction function.
result Achieves superhuman performance on Pitfall! and higher reward with fewer samples.

In this paper we sketch some reflections on the pitfalls and inconsistencies of the research program - currently dominant among the profession - aimed at providing microfoundations to macroeconomics along a Walrasian perspective. We argue that such a methodological approach constitutes an unsatisfactory answer to a wel…

2006-08-14abs ↗pdf ↗

When faced with high frequency streams of data, clustering raises theoretical and algorithmic pitfalls. We introduce a new and adaptive online clustering algorithm relying on a quasi-Bayesian approach, with a dynamic (i.e., time-dependent) estimation of the (unknown and changing) number of clusters. We prove that our a…

2016-02-01abs ↗pdf ↗

The purpose of these notes is to provide a systematic quantitative framework - in what is intended to be a "pedagogical" fashion - for discussing mean-reversion and optimization. We start with pair trading and add complexity by following the sequence "mean-reversion via demeaning -> regression -> weighted regression ->…

2014-08-10abs ↗pdf ↗

Correctly evaluating defenses against adversarial examples has proven to be extremely difficult. Despite the significant amount of recent work attempting to design defenses that withstand adaptive attacks, few have succeeded; most papers that propose defenses are quickly shown to be incorrect. We believe a large contri…

2019-02-18abs ↗pdf ↗

There is an increasing interest in estimating expectations outside of the classical inference framework, such as for models expressed as probabilistic programs. Many of these contexts call for some form of nested inference to be applied. In this paper, we analyse the behaviour of nested Monte Carlo (NMC) schemes, for w…

2016-12-03abs ↗pdf ↗

Paper explores limitations of generative models in finance, proposing a new method for portfolio generation.

problem Challenges in applying generative models to financial portfolio and risk management.
method Theoretical analysis and empirical testing of generative models, proposing a new pipeline for multivariate return generation.
result A new method for generating multivariate returns that meets portfolio evaluation standards and avoids pitfalls.

A grand challenge in reinforcement learning is intelligent exploration, especially when rewards are sparse or deceptive. Two Atari games serve as benchmarks for such hard-exploration domains: Montezuma's Revenge and Pitfall. On both games, current RL algorithms perform poorly, even those with intrinsic motivation, whic…

2019-01-30abs ↗pdf ↗

GrowNet uses shallow neural networks for gradient boosting, outperforming existing methods.

problem Improving gradient boosting performance through shallow neural networks.
method Unified gradient boosting framework with shallow neural networks as weak learners, incorporating corrective steps.
result GrowNet outperformed state-of-the-art boosting methods in classification, regression, and learning to rank tasks.

In this paper, we briefly review the basic scheme of the pseudoinverse learning (PIL) algorithm and present some discussions on the PIL, as well as its variants. The PIL algorithm, first presented in 1995, is a non-gradient descent and non-iterative learning algorithm for multi-layer neural networks and has several adv…

2018-05-20abs ↗pdf ↗

In this paper we propose a novel approach for learning from data using rule based fuzzy inference systems where the model parameters are estimated using Bayesian inference and Markov Chain Monte Carlo (MCMC) techniques. We show the applicability of the method for regression and classification tasks using synthetic data…

2016-10-28abs ↗pdf ↗