Leo Breiman's Rashomon Effect and Occam Dilemma are re-evaluated in the context of modern machine learning.
problem The tradeoff between model complexity and accuracy in machine learning.
method Modern perspective on Breiman's arguments using current computational capabilities.
result Algorithmic models can be accurate without being complex, nullifying the Occam Dilemma.
This paper resolves Breiman's dilemma in neural networks by analyzing phase transitions of margin dynamics.
problem Breiman's dilemma in neural networks: uniform margin improvement does not guarantee reduced generalization errors.
method Revisiting Breiman's dilemma in deep neural networks with spectrally normalized margins, analyzing phase transitions of normalized margin distributions.
result Margin-based generalization bounds can predict test error trends during training phase transitions.
Breiman's data analysis dichotomy is outdated, offering a third approach: mechanistic models.
problem Data analysis dichotomy between data modelers and algorithmic modelers.
method Interpolating between simple interpretable models and flexible function approximations using mechanistic models.
result Flexible, interpretable, and scientifically-informed hybrids can provide accurate and robust predictions.
Breiman's two cultures reconciled through blending statistical thinking.
problem Tension between parametric statistical and machine learning approaches.
method Establishing a link between parametric statistical and machine learning frameworks.
result Integrated statistical thinking can bridge the gap between two cultures.
Breiman's paper sparked debate on the future of statistics and machine learning.
problem The tension between traditional statistical modeling and model-free machine learning approaches.
method Discussion of the implications of machine learning's success and the need for new inferential approaches.
result The importance of understanding 'why' and 'if' questions in machine learning is now recognized.
RFX-Fuse combines Breiman and Cutler's Random Forest with modern ML capabilities.
problem Lack of a unified ML engine with diverse capabilities.
method Unified ML engine with native GPU/CPU support, delivering 5+ functionalities in one model.
result Native explainable similarity and imputation validation.
Breiman discusses two statistical cultures, advocating for more research on 'before' and 'after' the black box.
problem Statistical modeling lacks exploration of processes before and after the 'black box'.
method Analyzes Breiman's visual metaphor of two statistical cultures.
result Promotes the importance of studying the 'before' and 'after' of data transformations.
Heterogeneous SVO leads to diverse policies in sequential social dilemmas.
problem Understanding how diverse social value orientations affect behavior in sequential social dilemmas.
method Extending prior reinforcement learning studies, we instantiated heterogeneous SVO in a sequential social dilemma setting and measured task-specific diversity metrics.
result Heterogeneous SVO leads to meaningfully diverse policies across various incentive structures.
SLIM model tackles graph classification by resolving part-interaction dilemmas.
problem Difficulty in modeling graph parts and their interactions in graph classification.
method SLIM model, which solves resolution dilemmas and leverages explicit interactions.
result SLIM offers improved interpretability, accuracy, and new insights in graph representation learning.
The Shannon theorem is extended to locally compact groups.
problem Identifying the Poisson boundary of locally compact groups.
method Random walks and Shannon-McMillan-Breiman theorem.
result Generalized criteria for identifying Poisson boundaries.
NFTs raise concerns like scams, racism, and sexism; centralization vs decentralization debate.
problem Concerns and value judgments of stakeholders in NFT market.
method Mixed quantitative and qualitative methods: social media analysis and interviews.
result Identified financial scams, counterfeit NFTs, hacking, and unethical NFTs as major issues.
Random forests are a scheme proposed by Leo Breiman in the 2000's for building a predictor ensemble with a set of decision trees that grow in randomly selected subspaces of data. Despite growing interest and practical use, there has been little exploration of the statistical properties of random forests, and little is …
Commentary on Rashomon Effect complicating model selection.
problem Many models equally predict data; hard to draw conclusions.
method Connections to recent ML literature exploring implications.
result Grasping Rashomon Effect can foster collaboration.
Breiman (2001) proposed to statisticians awareness of two cultures: 1. Parametric modeling culture, pioneered by R.A.Fisher and Jerzy Neyman; 2. Algorithmic predictive culture, pioneered by machine learning research. Parzen (2001), as a part of discussing Breiman (2001), proposed that researchers be aware of many cultu…
Adding data can sometimes hurt model performance in multi-source healthcare tasks.
problem Identifying when adding more data helps or hinders model outcomes in multi-source healthcare tasks.
method Identified the Data Addition Dilemma, demonstrated empirically observed trade-offs, introduced distribution shift heuristics.
result Adding data can sometimes reduce model performance due to distribution shift.
Random forest (Leo Breiman 2001a) (RF) is a non-parametric statistical method requiring no distributional assumptions on covariate relation to the response. RF is a robust, nonlinear technique that optimizes predictive accuracy by fitting an ensemble of trees to stabilize model estimates. Random survival forests (RSF) …
New Random Forest variants estimate heterogeneous treatment effects using Wasserstein distances.
problem Estimating heterogeneous treatment effects in complex situations.
method Proposes natural variants of Random Forests using Wasserstein distances.
result Natural variants of Random Forests are well-suited for estimating conditional distributions.
NN-Stacking improves predictive power of regression models by adjusting stacking coefficients with features.
problem Low predictive power of linear stacking methods.
method NN-Stacking uses neural networks to estimate adaptive stacking coefficients.
result NN-Stacking leads to better predictive power, especially in large datasets.
Examines algorithmic modeling across three cultures.
problem Tackles algorithmic modeling in different cultural contexts.
method Uses parametric regressions, interpretable algorithms, and complex algorithms.
result Extension of Leo Breiman's thesis to include cultural differences.
This paper tackles clustering generalization by introducing a new concept based on multidimensional persistent homology.
problem The lack of general-purpose learning guarantees for data clustering.
method Introducing a new concept based on multidimensional persistent homology to analyze clustering generalization.
result The CR dilemma clarifies the contrast between overfitting and underfitting in clustering models.
Random forests are a learning algorithm proposed by Breiman [Mach. Learn. 45 (2001) 5--32] that combines several randomized decision trees and aggregates their predictions by averaging. Despite its wide usage and outstanding practical performance, little is known about the mathematical properties of the procedure. This…
The study examines causal razors and their logical relations, highlighting a dilemma in causal discovery.
problem Selecting a reasonable scoring criterion for causal discovery algorithms.
method Review and logical comparison of numerous causal razors, focusing on parameter minimality in multinomial models.
result Parameter minimality poses a dilemma in selecting a reasonable scoring criterion for causal discovery algorithms.
Random forests have become an important tool for improving accuracy in regression and classification problems since their inception by Leo Breiman in 2001. In this paper, we revisit a historically important random forest model originally proposed by Breiman in 2004 and later studied by Gérard Biau in 2012, where a feat…
New method improves OOD detection without sacrificing generalization.
problem Improving OOD detection models that also generalize well.
method Decouples uncertainty learning from Bayesian perspective.
result Achieves state-of-the-art OOD detection performance.
Bayesian active learning tackles nuisance parameters, leading to bias and dilemmas.
problem Bayesian active learning with nuisance parameters leads to bias and dilemmas.
method Characterizes and mitigates negative interference by accurately estimating nuisance parameters.
result The extent of negative interference can be extremely large, and accurate estimation of nuisance parameters is critical.
The explore{exploit dilemma is one of the central challenges in Reinforcement Learning (RL). Bayesian RL solves the dilemma by providing the agent with information in the form of a prior distribution over environments; however, full Bayesian planning is intractable. Planning with the mean MDP is a common myopic approxi…
Kernel methods are popular in clustering due to their generality and discriminating power. However, we show that many kernel clustering criteria have density biases theoretically explaining some practically significant artifacts empirically observed in the past. For example, we provide conditions and formally prove the…
This paper is a comment on the survey paper by Biau and Scornet (2016) about random forests. We focus on the problem of quantifying the impact of each ingredient of random forests on their performance. We show that such a quantification is possible for a simple pure forest , leading to conclusions that could apply more…
Given a collection of data points, non-negative matrix factorization (NMF) suggests to express them as convex combinations of a small set of `archetypes' with non-negative entries. This decomposition is unique only if the true archetypes are non-negative and sufficiently sparse (or the weights are sufficiently sparse),…
Random Forests [Breiman:2001] (RF) are a fully non-parametric statistical method requiring no distributional assumptions on covariate relation to the response. RF are a robust, nonlinear technique that optimizes predictive accuracy by fitting an ensemble of trees to stabilize model estimates. The randomForestSRC packag…
We present PredRNN++, an improved recurrent network for video predictive learning. In pursuit of a greater spatiotemporal modeling capability, our approach increases the transition depth between adjacent states by leveraging a novel recurrent unit, which is named Causal LSTM for re-organizing the spatial and temporal m…
The aim of this work is to extend the capital growth theory developed by Kelly, Breiman, Cover and others to asset market models with transaction costs. We define a natural generalization of the notion of a numeraire portfolio proposed by Long and show how such portfolios can be used for constructing growth-optimal inv…
Bayesian model averaging, model selection and its approximations such as BIC are generally statistically consistent, but sometimes achieve slower rates og convergence than other methods such as AIC and leave-one-out cross-validation. On the other hand, these other methods can br inconsistent. We identify the "catch-up …
Research tackles alliance formation in many-player zero-sum games, showing reinforcement learning fails but a contract mechanism can help.
problem Tackles the challenge of alliance formation in many-player zero-sum games.
method Demonstrates the social dilemma aspect of alliance formation, introduces a contract mechanism to augment reinforcement learning.
result Naïve reinforcement learning fails to form alliances, but a contract mechanism can help.
Paper finds efficient OPE estimator for multiple logging policies with minimum variance.
problem Finding optimal importance sampling weights for multiple logging policies with varying variances.
method Established efficiency bound under stratified sampling and proposed an estimator achieving this bound.
result Proposed estimator achieves minimum variance for any instance.
Improved bounds for neural network approximations of functions.
problem Bounding the width of neural networks for function approximation.
method Extending Radon-based norms to bounded open sets and deriving new approximation bounds.
result Improved sparse approximation bounds for neural networks.
Generative AI predicts Arctic sea ice dynamics over decades.
problem Reproducing realistic sea ice dynamics from days to decades is computationally challenging.
method Introduced GenSIM, a generative AI model trained on 20 years of sea-ice-ocean simulation data.
result Generative AI predicts realistic sea ice evolution for 30 years, capturing long-term trends and physical consistency.
Paper benchmarks MBRL algorithms with new environments.
problem Lack of standardized benchmarking for MBRL algorithms.
method Proposes 18 new benchmarking environments and standardizes problem settings.
result Characterizes three key research challenges for MBRL.
SAE improves VAE's latent space precision in high dimensions.
problem High-dimensional latent codes vs probabilistic inference in VAEs.
method Spherical Auto-Encoder (SAE) with spherical normalization on latent space.
result SAE improves latent code inference precision in high dimensions.
AANets balance stability and plasticity in CIL.
problem Stability-plasticity dilemma in class-incremental learning.
method Adaptive Aggregation Networks (AANets) with stable and plastic residual blocks.
result AANets improve performance on CIL benchmarks.
RFX accelerates and compresses Random Forests for large datasets.
problem Memory bottleneck in proximity matrices limits Random Forest analysis.
method QLORA compression, CPU TriBlock storage, GPU batch sizing, 3D MDS visualization.
result Proximity-based Random Forest analysis on larger datasets is feasible.
New method calculates tail probabilities of random vectors under linear transformations.
problem Computing tail probabilities of random vectors under linear transformations.
method Characterization of regular variation on cones in [0,∞)d under random linear transformations. result Allows computation of probabilities of tail events that were previously negligible.
Given an ensemble of randomized regression trees, it is possible to restructure them as a collection of multilayered neural networks with particular connection weights. Following this principle, we reformulate the random forest method of Breiman (2001) into a neural network setting, and in turn propose two new hybrid p…
We consider a novel stochastic multi-armed bandit problem called {\em good arm identification} (GAI), where a good arm is defined as an arm with expected reward greater than or equal to a given threshold. GAI is a pure-exploration problem that a single agent repeats a process of outputting an arm as soon as it is ident…
Proposes SWA for adversarial training to improve model robustness.
problem Overfitting in adversarial training due to insufficient sample complexity.
method Adversarial training with Stochastic Weight Average (SWA).
result Improves model robustness on CIFAR-10, CIFAR-100, and SVHN datasets.
Investigates model selection challenges in heterogeneous treatment effect estimation.
problem Lack of validation metrics for choosing the best model in treatment effect estimation.
method Empirical investigation of different model selection criteria.
result Complex interplay between selection strategies, estimators, and data.
We derive an optimal strategy for minimizing the expected loss in the two-period economy when a pivotal decision needs to be made during the first time period and cannot be subsequently reversed. Our interest in the problem has been motivated by the classical shopper's dilemma during the Black Friday promotion period, …
New algorithms for batch decision-making with high-dimensional user data.
problem Maximizing treatment efficacy in groups of users.
method Teamwork LASSO Bandit algorithm, switching between teamwork and selfish stages.
result Upper bound on expected cumulative regret for the proposed algorithm.