The paper analyzes LOCV for high-dimensional risk estimation, proving error bounds.
problem Estimating out-of-sample prediction error in high-dimensional settings.
method Theoretical analysis of leave-one-out cross validation (LOCV) in penalized regression.
result Finite sample upper bounds on LOCV error, showing it converges to zero as n,p → ∞.
New method for estimating out-of-sample R² from gene expression data.
problem Lack of a well-defined and unbiased estimator for out-of-sample R².
method Explicitly defined out-of-sample R², provided an unbiased estimator, and calculated standard error.
result Demonstrated improved model comparison for gene expression phenotypes.
Estimates error for robust M-estimators with convex penalties.
problem Estimating out-of-sample error for robust M-estimators in high-dimensional linear regression.
method Proposes a generic out-of-sample error estimate for robust M-estimators with convex penalties, using observed data and derivatives. result The out-of-sample error estimate has a relative error of order n−1/2 under certain conditions. Paper introduces DOO models to outperform SAA out-of-sample.
problem Outperforming SAA in out-of-sample performance.
method Introduces DOO models that consider both worst-case and best-case scenarios.
result DOO models can always outperform SAA out-of-sample.
Improves test set performance and reduces out-of-sample disappointment for unstable models.
problem Ensuring strong test set performance via cross-validation for unstable models.
method Nested k-fold cross-validation with hyperparameter selection based on a weighted sum of cross-validation metric and model stability measure.
result Improves out-of-sample MSE for sparse ridge regression and CART by 4% and 2% respectively, compared to k-fold cross-validation.
We identify and validate a model for PCR in high dimensions, improving prediction guarantees.
problem Model identification and out-of-sample prediction in high-dimensional error-in-variables settings.
method Analysis of principal component regression (PCR) in fixed design settings, introducing a linear algebraic condition.
result Consistent model identification and improved out-of-sample prediction guarantees.
Optimal number of voters for a voting ensemble can be estimated from the distribution of classifier errors.
problem Finding the optimal number of voters for a voting ensemble to minimize error rate.
method Estimate the distribution of classifier errors and infer error rates for different numbers of voters.
result Lower-variance estimates of error rates can be obtained by inferring them for different numbers of voters.
Paper presents a new way to analyze machine learning generalization without probabilistic assumptions.
problem Traditional generalization analysis assumes i.i.d. data, which is often unverifiable.
method Uses sensitivity analysis of optimization problems to derive deterministic generalization bounds.
result Obtains generalization bounds that relate in-sample and out-of-sample evaluations through an error term quantifying data similarity.
New bounds for RFM-KRR with weak assumptions and easy verification.
problem Establishing accurate out-of-sample bounds for RFM-KRR.
method Elementary linear algebra and weak assumptions.
result Novel out-of-sample error upper and lower bounds with weak assumptions.
Many popular dimensionality reduction procedures have out-of-sample extensions, which allow a practitioner to apply a learned embedding to observations not seen in the initial training sample. In this work, we consider the problem of obtaining an out-of-sample extension for the adjacency spectral embedding, a procedure…
Study improves prediction accuracy and uncertainty for mobile sensor data using randomized neural networks.
problem Improving prediction accuracy and uncertainty for mobile sensor data.
method Cross-validation and uncertainty determination for randomized neural networks.
result Improved out-of-sample performance and confidence intervals for prediction error.
Dimensionality reduction methods are very common in the field of high dimensional data analysis. Typically, algorithms for dimensionality reduction are computationally expensive. Therefore, their applications for the analysis of massive amounts of data are impractical. For example, repeated computations due to accumula…
When the in-sample Sharpe ratio is obtained by optimizing over a k-dimensional parameter space, it is a biased estimator for what can be expected on unseen data (out-of-sample). We derive (1) an unbiased estimator adjusting for both sources of bias: noise fit and estimation error. We then show (2) how to use the adjust…
Paper studies M-estimators with derivatives and residual distribution for robust adaptive tuning.
problem Tackles robustness and adaptive tuning of M-estimators with heavy-tailed noise.
method Provides formulae for derivatives, characterizes residual distribution, proposes adaptive criterion.
result Characterizes distribution of residuals and proposes adaptive criterion as out-of-sample error proxy.
Fast, reliable, and error-bounded option pricing with neural networks
problem Fast, reliable, and error-bounded option pricing
method Mixture Density Network
result Out-of-sample CDF error of 1.4imes10−4 Nonlinear kernels can be approximated using finite-dimensional feature maps for efficient risk minimization. Due to the inherent trade-off between the dimension of the (mapped) feature space and the approximation accuracy, the key problem is to identify promising (explicit) features leading to a satisfactory out-of-sam…
New bounds for DeepONets reduce out-of-sample error without width dependence.
problem Measuring out-of-sample error in DeepONets without width dependence.
method Proved Rademacher complexity bound and Huber loss choice for DeepONets.
result Generalization error bounds have no explicit dependence on net size.
Proposes a new framework to optimize portfolios with reduced estimation errors.
problem Estimation errors in multiperiod mean-variance portfolio optimization.
method Reference-regulated multiperiod mean-variance (RRMV) framework.
result Improves portfolio stability and out-of-sample Sharpe ratios.
Diffusion maps are a nonlinear manifold learning technique based on harmonic analysis of a diffusion process over the data. Out-of-sample extensions with computational complexity O(N), where N is the number of points comprising the manifold, frustrate applications to online learning applications requiring…
Sparse modeling improves portfolio optimization by reducing errors in complex market systems.
problem Errors in multivariate modeling of markets and economy.
method L0-norm sparse elliptical modeling to reduce oversimplification, and study likelihood in- and out-of-sample for different parameter lengths.
result Sparse models lead to better portfolio performance, higher out-of-sample likelihood, and lower volatility.
Under the framework of spectral clustering, the key of subspace clustering is building a similarity graph which describes the neighborhood relations among data points. Some recent works build the graph using sparse, low-rank, and ℓ2-norm-based representation, and have achieved state-of-the-art performance. Howeve…
A new deep learning model improves asset pricing predictions.
problem Improving asset pricing models for better predictions.
method Pseudo-Siamese Network (SNAP) for conditional asset pricing.
result The SNAP model outperforms benchmarks in out-of-sample prediction and Sharpe ratio.
The paper tackles stock prediction models by improving their generalizability to out-of-sample domains using causal representation learning.
problem Low signal-to-noise ratio and nonstationary nature of financial markets lead to poor performance of stock prediction models.
method The paper investigates Domain Generalization techniques, focusing on causal representation learning to improve model generalizability. It introduces a novel error bound and a causal discovery technique to mitigate spurious correlations.
result The proposed approach enhances the generalizability of stock prediction models, as demonstrated by numerical results.
Paper proposes a method to improve prediction intervals for neural networks.
problem Improving prediction intervals for neural network models.
method Adapting extremely randomized trees to neural networks to create ensembles.
result The method yields gains in out-of-sample accuracy and is superior to existing methods.
The paper improves ALO for ℓ1-regularized models.
problem Estimating out-of-sample error for ℓ1-regularized models. method Developed a novel theory for ℓ1-regularized problems, bounding ALO error. result For ℓ1-regularized problems, ALO error goes to zero as p goes to infinity. A new framework for time series forecasting that adapts to varying patterns.
problem Forecasting multivariate time series with predictive heterogeneity.
method Validation-driven clustering framework that applies specialization based on out-of-sample predictive performance.
result Improves robustness to heavy-tailed errors and local anomalies.
Let X=X∪Z be a data set in RD, where X is the training set and Z is the test one. Many unsupervised learning algorithms based on kernel methods have been developed to provide dimensionality reduction (DR) embedding for a given training set $Φ: \mathbf{X} \to \mat…
This work gives a simultaneous analysis of both the ordinary least squares estimator and the ridge regression estimator in the random design setting under mild assumptions on the covariate/response distributions. In particular, the analysis provides sharp results on the ``out-of-sample'' prediction error, as opposed to…
Variant of mSSA improves time series prediction error.
problem Improve prediction error in multivariate time series.
method Introduce spatio-temporal factor model, establish prediction error scaling.
result Prediction error scales as 1 / √(min(N, T)T).
A new hyperparameter optimization method reduces overfitting.
problem Overfitting in hyperparameter optimization.
method PAC-Bayes bound minimization using gradient-based algorithm.
result Significant reduction in out-of-sample error.
Clustering stocks reduces estimation error in global minimum variance portfolio.
problem High estimation error in covariance matrix estimation.
method Bounded clustering to limit maximum cluster size.
result Reduction in out-of-sample volatility and gap between in-sample and out-of-sample volatility.
We study the out-of-sample properties of robust empirical optimization problems with smooth φ-divergence penalties and smooth concave objective functions, and develop a theory for data-driven calibration of the non-negative "robustness parameter" δ that controls the size of the deviations from the nominal model. Bu…
Robustifies Markowitz portfolios to reduce transaction costs and improve performance.
problem Markowitz portfolios are unreliable due to estimation errors and extreme weights.
method Projected gradient descent and robust statistics for stable weights and costs.
result Robustified Markowitz portfolios have lower turnover and maintain or improve performance.
The paper optimizes asset selection for index trackers and enhanced trackers with varying cardinality constraints.
problem Optimizing asset selection for index trackers and enhanced trackers with cardinality constraints.
method Divided into two steps: asset pre-selection and asset weight estimation. Used eight pre-selection procedures with different combinations of selection methods and regression types.
result Out-of-sample tracking errors are roughly proportional to 1/sqrt(cardinality). OLS is more effective than LAD, BE marginally more effective than FS, and (n) marginally more effective than (c).
This paper presents an alternative approach to p-values in regression settings. This approach, whose origins can be traced to machine learning, is based on the leave-one-out bootstrap for prediction error. In machine learning this is called the out-of-bag (OOB) error. To obtain the OOB error for a model, one draws a bo…
Manifold learning has been successfully applied to a variety of medical imaging problems. Its use in real-time applications requires fast projection onto the low-dimensional space. To this end, out-of-sample extensions are applied by constructing an interpolation function that maps from the input space to the low-dimen…
This paper presents an out-of-sample prediction comparison between major machine learning models and the structural econometric model. Over the past decade, machine learning has established itself as a powerful tool in many prediction applications, but this approach is still not widely adopted in empirical economic stu…
We study model evaluation and model selection from the perspective of generalization ability (GA): the ability of a model to predict outcomes in new samples from the same population. We believe that GA is one way formally to address concerns about the external validity of a model. The GA of a model estimated on a sampl…
The paper analyzes prediction error in nonstationary settings using weighted risk minimization.
problem Prediction under distribution drift and nonstationary conditions.
method General decomposition of excess risk into learning and drift terms, proving oracle inequalities under mixing conditions.
result Oracle inequalities for the learning error, providing bounds that hold uniformly over arbitrary weight classes.
Alpha-based performance evaluation may fail to capture correlated residuals due to model errors. This paper proposes using the Generalized Information Ratio (GIR) to measure performance under misspecified benchmarks. Motivated by the theoretical link between abnormal returns and residual covariance matrix, GIR is deriv…
The paper proves limit theorems for graph embeddings out-of-sample.
problem Proving limit theorems for graph embeddings out-of-sample.
method Least-squares and maximum-likelihood objectives for adjacency and Laplacian spectral embeddings.
result Out-of-sample extensions based on these objectives obey central limit theorems and concentration inequalities.
In the past decade many researchers have proposed new optimal portfolio selection strategies to show that sophisticated diversification can outperform the naïve 1/N strategy in out-of-sample benchmarks. Providing an updated review of these models since DeMiguel et al. (2009b), I test sixteen strategies across six empir…
Paper analyzes high-dimensional portfolio risks and finds empirical out-of-sample relative loss is more reliable.
problem Analyzing risks in high-dimensional portfolios using empirical variance.
method Derives asymptotic behavior of out-of-sample variance and relative loss in high-dimensional settings.
result Empirical out-of-sample relative loss is more reliable than variance in high-dimensional portfolios.
This paper proposes an out-of-sample extension framework for a global manifold learning algorithm (Isomap) that uses temporal information in out-of-sample points in order to make the embedding more robust to noise and artifacts. Given a set of noise-free training data and its embedding, the proposed framework extends t…
We consider the problem of vertex classification for graphs constructed from the latent position model. It was shown previously that the approach of embedding the graphs into some Euclidean space followed by classification in that space can yields a universally consistent vertex classifier. However, a major technical d…
Study improves understanding of non-differentiable penalties in high-dimensional settings.
problem Theoretical understanding of non-differentiable penalties like generalized LASSO and nuclear norm in high-dimensional settings.
method Proportional high-dimensional regime analysis with finite sample upper bounds on expected squared error.
result LO provides accurate estimation of out-of-sample risk in high-dimensional settings.
The paper proposes a new model for predicting and analyzing economic variables.
problem Predicting and analyzing economic variables in developed regions.
method Time-varying parameter global vector autoregressive (TVP-GVAR) framework combined with machine learning models.
result The proposed model provides high precision out-of-sample predictions and novel insights into economic variable connectedness.
CASTLE learns causal DAG to improve model generalization.
problem Improving model generalization to out-of-sample data.
method CASTLE learns causal relationships via adjacency matrix embedded in neural network input layers, reconstructing only causal features.
result CASTLE leads to better out-of-sample predictions compared to other regularizers.