Recent studies have shown that tuning prediction models increases prediction accuracy and that Random Forest can be used to construct prediction intervals. However, to our best knowledge, no study has investigated the need to, and the manner in which one can, tune Random Forest for optimizing prediction intervals { thi…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,742 papers · 148 categories
Trend · papers per month
48 results for “validation techniques”
Analyzing different prototype selection techniques for dynamic classifier and ensemble selectioncs.LG
In dynamic selection (DS) techniques, only the most competent classifiers, for the classification of a specific test sample are selected to predict the sample's class labels. The more important step in DES techniques is estimating the competence of the base classifiers for the classification of each specific test sampl…
Deep learning techniques have been hugely successful for traditional supervised and unsupervised machine learning problems. In large part, these techniques solve continuous optimization problems. Recently however, discrete generative deep learning models have been successfully used to efficiently search high-dimensiona…
New validation method prevents privacy breaches and biases in federated learning.
problem Privacy breaches and data leakage in federated learning.
method Stratified cross-validation for unbiased and privacy-preserving federated learning.
result Stratified cross-validation prevents data leakage without demanding deduplication algorithms.
The correct use of model evaluation, model selection, and algorithm selection techniques is vital in academic machine learning research as well as in many industrial settings. This article reviews different techniques that can be used for each of these three subtasks and discusses the main advantages and disadvantages …
Survey of algorithms for testing AI-driven CPS safety.
problem Testing AI-driven CPS for safety in complex environments.
method Survey of applied algorithms for safety validation.
result Survey of existing tools and techniques for safety validation.
ECV method optimizes ensemble parameters for randomized ensembles.
problem Efficient tuning of ensemble parameters in randomized ensembles.
method ECV (Extrapolated Cross-Validation) method for tuning ensemble and subsample sizes.
result ECV yields δ-optimal ensembles for squared prediction risk.
Machine learning detects survey validity from user behavior.
problem Detecting valid responses in web surveys.
method Uses mouse activity and machine learning models (LSTM, HMM).
result Predicts survey validity without analyzing specific answers.
Proposes approximating computationally expensive explainability techniques using conformal regression.
problem Computational expense of score-based explainability techniques limits their applicability in time-critical contexts.
method Uses conformal prediction framework to approximate SHAP and TreeSHAP explanations.
result Significantly improves execution time and produces tight validity guarantees.
Interpretable ML helps discover insights from big data.
problem Validating data-driven discoveries from complex datasets.
method Statistical and machine learning techniques for interpretable models.
result Challenges in validating data-driven discoveries remain.
Neural Architecture Search remains a very challenging meta-learning problem. Several recent techniques based on parameter-sharing idea have focused on reducing the NAS running time by leveraging proxy models, leading to architectures with competitive performance compared to those with hand-crafted designs. In this pape…
The paper identifies overfitting as the main bottleneck in efficient deep reinforcement learning.
problem Improving sample efficiency in deep reinforcement learning.
method Empirical analysis on DMC tasks to identify overfitting as the main issue and developing a hill-climbing method targeting validation TD error.
result Overfitting is the primary bottleneck in sample-efficient deep RL, and regularization techniques can control this.
We provide a rigorous numerical computation method to validate periodic, homoclinic and heteroclinic orbits as the continuation of singular limit orbits for the fast-slow system with one-dimensional slow variable . Our validation procedure is based on topological tools called isolatin…
Validation of internal rating systems and PD estimatesphysics.soc-ph
This paper elaborates on the validation requirements for rating systems and probabilities of default (PDs) which were introduced with the New Capital Standards (Basel II). We start in Section 2 with some introductory remarks on the topics and approaches that will be discussed later on. Then we have a view on the develo…
Study compares resampling methods for rare event prediction in longitudinal studies.
problem Predicting rare events in longitudinal follow-up studies.
method Comparison of resampling methods to improve standard regression models.
result Effect of sampling rate on model predictive performance.
New method narrows prediction intervals for individual treatment effects.
problem Insufficiently conservative prediction intervals for individual treatment effects.
method Conformal inference using conditional density estimates.
result Narrower prediction intervals compared to existing methods.
Method selects valid IVs from a large set using clustering and test of overidentifying restrictions.
problem Selecting valid instrumental variables from a large set of candidates.
method Agglomerative hierarchical clustering combined with a test of overidentifying restrictions.
result Achieves oracle properties when the largest group of IVs is valid.
New IF method improves accuracy in deep neural networks with noisy data.
problem Inaccurate influence estimates in deep neural networks, especially with noisy data.
method Established a connection between influence estimation error, validation set risk, and sharpness, introducing a novel estimation form for flat validation minima.
result Our novel Influence Function approach provides more accurate influence estimates, validated across various tasks.
Proves and tests methods for learning time-series with breaks.
problem Learning time-series with structural breaks.
method Complete proofs and experimental validation of a regularized loss function.
result Experimental results support the validity of the techniques.
Scalable Autonomous Vehicle Safety Validation through Dynamic Programming and Scene Decompositioncs.RO
New approach uses dynamic programming to efficiently discover failures in autonomous vehicle simulations.
problem Efficiently discovering rare failure events in autonomous vehicle simulations.
method Approximate dynamic programming and scene decomposition to estimate failure distribution.
result Increased number of failures discovered compared to baseline approaches.
Linear principal component analysis (PCA) can be extended to a nonlinear PCA by using artificial neural networks. But the benefit of curved components requires a careful control of the model complexity. Moreover, standard techniques for model selection, including cross-validation and more generally the use of an indepe…
The paper develops a cross-validation method for improving signal denoising techniques.
problem Improving signal denoising methods for nonparametric regression.
method Develops a general cross-validation framework for signal denoising and applies it to Trend Filtering and Dyadic CART.
result Cross validated versions of Trend Filtering and Dyadic CART achieve nearly optimal convergence rates.
Paper improves PAC-Bayes bounds for various loss types.
problem Improving PAC-Bayes bounds for different types of losses.
method Introducing new high-probability PAC-Bayes bounds for bounded and general tail behaviors losses, and extending to anytime-valid bounds.
result New fast-rate and mixed-rate bounds for losses with bounded ranges, and parameter-free bounds for losses with general tail behaviors.
New approach uses interpolation models and error bounds for verifiable scientific machine learning.
problem Challenges in verifying and validating modern scientific machine learning workflows.
method Combines multiple standard interpolation techniques with error bounds for efficient computation and comparative performance analysis.
result Error bounds for interpolation techniques can be computed or estimated efficiently, aiding in validation goals.
We introduce a new normalization technique that exhibits the fast convergence properties of batch normalization using a transformation of layer weights instead of layer outputs. The proposed technique keeps the contribution of positive and negative weights to the layer output balanced. We validate our method on a set o…
A novel validation method improves feature importance analysis in subject-specific ML models.
problem Limited effectiveness of general ML models trained on large datasets for individual patient outcomes.
method A novel validation approach using a general ML model with repeated trials and random seed variation.
result Consistent identification of key features at the subject level and improved group-level feature importance analysis.
We aim at developing and improving the imbalanced business risk modeling via jointly using proper evaluation criteria, resampling, cross-validation, classifier regularization, and ensembling techniques. Area Under the Receiver Operating Characteristic Curve (AUC of ROC) is used for model comparison based on 10-fold cro…
A fast bootstrap method estimates cross-validation standard error.
problem Uncertainty quantification in cross-validation estimates.
method Random-effects model to estimate variance component.
result Valid confidence intervals for model performance.
Causal ML methods failed to validate their personalized treatment effects in two large trials.
problem Validating causal machine learning methods for personalized treatment effects in precision medicine.
method Assessed 17 mainstream causal heterogeneity ML methods using two large randomized controlled trials.
result None of the ML methods reliably validated their performance, internal or external, showing significant discrepancies between training and test data.
Method improves treatment effect estimation in randomized experiments.
problem Estimating distributional treatment effects in randomized experiments.
method Distributional regression framework with machine learning for variance reduction.
result The proposed method reduces variance of distributional treatment effect estimators.
Popular machine learning estimators involve regularization parameters that can be challenging to tune, and standard strategies rely on grid search for this task. In this paper, we revisit the techniques of approximating the regularization path up to predefined tolerance in a unified framework and show that its comp…
We extend the validity of the Penrose singularity theorem to spacetime metrics of regularity . The proof is based on regularisation techniques, combined with recent results in low regularity causality theory.
Early stopping is a widely used technique to prevent poor generalization performance when training an over-expressive model by means of gradient-based optimization. To find a good point to halt the optimizer, a common practice is to split the dataset into a training and a smaller validation set to obtain an ongoing est…
Cross-validation estimates model performance on unseen data, not training data.
problem Understanding how cross-validation estimates prediction error and its limitations.
method Analyzing linear models and popular prediction error estimates, introducing nested cross-validation.
result Cross-validation estimates the average prediction error of models fit on other unseen training sets, not the model at hand.
Confounding bias, missing data, and selection bias are three common obstacles to valid causal inference in the data sciences. Covariate adjustment is the most pervasive technique for recovering casual effects from confounding bias. In this paper, we introduce a covariate adjustment formulation for controlling confoundi…
We consider the problem of decomposing a multivariate polynomial as the difference of two convex polynomials. We introduce algebraic techniques which reduce this task to linear, second order cone, and semidefinite programming. This allows us to optimize over subsets of valid difference of convex decompositions (dcds) a…
Paper presents a workflow for reliable unsupervised learning in science.
problem Lack of standardization in unsupervised learning workflows for reproducible scientific discoveries.
method Structured workflow including data preparation, modeling, validation, and communication.
result Illustrates the importance of validation in unsupervised learning.
New insights into ML models' accuracy and generalization for scientific problems.
problem Quantifying accuracy and generalization of ML models in scientific applications.
method Rigorous numerical analysis and theoretical bounds for linear differential equations.
result Different ML models can have opposing generalization behaviors, contrary to intuition.
A new method controls risk for set predictors using cross-validation.
problem Inefficient set predictors when data limited.
method Cross-validation conformal risk control (CV-CRC).
result CV-CRC offers theoretical guarantees and reduces set size.
Model inference, such as model comparison, model checking, and model selection, is an important part of model development. Leave-one-out cross-validation (LOO) is a general approach for assessing the generalizability of a model, but unfortunately, LOO does not scale well to large datasets. We propose a combination of u…
Study finds unsupervised imputation before cross-validation can reduce computational costs without significantly degrading model performance.
problem High computational costs in pipeline modeling algorithms with imputation steps.
method Empirical assessment of unsupervised imputation before vs during cross-validation.
result Reduced variance of imputation before cross-validation leads to lower overall root mean squared error.
A novel ML verification technique using manifold learning.
problem Ensuring trust in machine learning systems.
method Variational autoencoder for extracting a low-dimensional manifold from high-dimensional training data.
result The manifold provides diverse test data, fault-revealing test cases, and independent runtime trust assessment.
The defining equations for Killing vector fields and conformal Killing vector fields are overdetermined systems of PDE. This makes it difficult to solve the systems numerically. We propose an approach which reduces the computation to the solution of a symmetric eigenvalue problem. The eigenvalue problem is then solved …
Unified techniques improve stability and replicability in changing data.
problem Concept drift in data generating distribution.
method Removing hidden confounding and causal regularization.
result Improves stability, replicability, and robustness in heterogeneous data.
Machine learning systems increasingly depend on pipelines of multiple algorithms to provide high quality and well structured predictions. This paper argues interaction effects between clustering and prediction (e.g. classification, regression) algorithms can cause subtle adverse behaviors during cross-validation that m…
Recently, new defense techniques have been developed to tolerate Byzantine failures for distributed machine learning. The Byzantine model captures workers that behave arbitrarily, including malicious and compromised workers. In this paper, we break two prevailing Byzantine-tolerant techniques. Specifically we show robu…
The paper validates statistical models for groundwater data.
problem Validating statistical models for groundwater data.
method Traditional time-series models and modern neural networks.
result Validation techniques ensure lower computational cost and robust predictions.
Techniques for understanding the functioning of complex machine learning models are becoming increasingly popular, not only to improve the validation process, but also to extract new insights about the data via exploratory analysis. Though a large class of such tools currently exists, most assume that predictions are p…