A new principle for extrapolating regression outside training data.
problem Regression extrapolation when predictions are outside training data range.
method Data-adaptive marginal transformation and simple relationship assumption.
result Progression method offers guarantees on approximation error beyond training data range.
Proposes engression for extrapolation in distributional regression.
problem Challenging extrapolation problem in nonlinear regression.
method Neural network-based distributional regression.
result Engression successfully performs extrapolation under certain assumptions.
Shape-constrained symbolic regression improves model extrapolation with prior knowledge.
problem Improving model extrapolation with prior knowledge in symbolic regression.
method Shape-constrained symbolic regression using evolutionary algorithms with interval arithmetic.
result Models with shape constraints have improved extrapolation but lower accuracy on test sets.
Extreme value theory enhances statistical learning extrapolation for rare events.
problem Challenges in traditional machine learning methods for extreme data.
method Asymptotic theory and statistical tools for tail behavior.
result Effective extrapolation methods for extreme quantiles and anomalies.
NLR models often perform worse than LR for outlying input data in environmental sciences.
problem NLR models often give poor predictions for input data outside the training domain.
method Screened input data for outliers, using linear extrapolation for outliers based on NLR within the non-outlier domain.
result NLROR approach reduces poor extrapolation and tends to outperform NLR and LR for outliers. Neural network learns equations from data, improving interpretability and extrapolation.
problem Combining neural networks and symbolic regression for better model interpretability and extrapolation.
method Integrating a neural network-based Equation Learner (EQL) network with other deep learning architectures.
result The EQL-based architecture can extrapolate well outside of the training data set.
Accelerates coordinate descent methods for machine learning problems.
problem Slowness of coordinate descent methods in machine learning.
method Extrapolation-based accelerated coordinate descent.
result Significant speed-up in practice compared to existing methods.
Matrix completion and extrapolation (MCEX) are dealt with here over reproducing kernel Hilbert spaces (RKHSs) in order to account for prior information present in the available data. Aiming at a faster and low-complexity solver, the task is formulated as a kernel ridge regression. The resultant MCEX algorithm can also …
Paper proposes a new method to improve BART model predictions outside training data range.
problem Improving prediction and prediction intervals for BART models at extrapolation points.
method Gaussian processes are added to BART leaf nodes for extrapolation.
result The new method outperforms standard BART and frequentist resampling methods in simulations.
TSL learns separable models to avoid signal cancellation and off-support extrapolation.
problem Signal cancellation and off-support extrapolation in additive models.
method Tensor Separation Learning (TSL) via stagewise greedy procedure with orthogonal refitting.
result TSL avoids information loss caused by marginalizing higher-order interactions.
Paper tackles intervention extrapolation using identifiable representations.
problem Predicting effects of unseen interventions on outcomes.
method Combines identifiable representation learning with autoencoders to enforce linear invariance.
result Identifiable representations enable non-linear extrapolation of interventions.
Novel SVM approach for extreme quantile regression with heavy tailed inputs.
problem Learning from extreme values in quantile regression.
method Support Vector Machine framework for handling high-dimensional and nonlinear settings.
result Established finite-sample learning guarantees under mild regularity assumptions.
Gaussian Processes offer a flexible method for modeling and predicting outcomes with uncertainty estimates.
problem Capturing uncertainty in predictions at new data points, especially with poor overlap and extrapolation.
method Gaussian Processes model a posterior distribution over outcomes, reflecting the range of plausible models.
result GPs provide a principled approach to handling extrapolation and uncertainty in predictions.
The paper tackles extrapolation in extreme regions of regression problems.
problem Extrapolation on the tails of covariates in continuous regression problems.
method Statistical regression on a subsample of furthest observations, focusing on their angular components, using multivariate regular variation theory.
result Quantifies predictive performance on tail regions in terms of excess risk, presenting it as a finite sample risk bound with a bias-variance decomposition.
Proposes LRR and LRLR for improving stock prediction accuracy.
problem Improving stock prediction accuracy through nonparametric classification.
method Local radial regression and logistic regression variant.
result LRLR outperforms LPoR and MS-k-NN in real-world stock datasets. This paper examines how adversarial perturbations affect model performance and equilibrium learning.
problem Adversarial perturbations and covariate shifts impact model performance and equilibrium learning.
method Characterizes the extrapolation region in regression and classification, analyzes dynamics of adversarial learning games.
result Establishes two directional convergence results: a blessing in regression and a curse in classification.
Sparse regression models CMs from oscillatory shear data efficiently.
problem Discovering parsimonious constitutive models from oscillatory shear experiments.
method Sparse regression with tensor basis functions, l1 regularization, and greedy two-stage algorithm.
result Inferred CMs extrapolate well beyond training data and flow conditions.
Point forecasting of univariate time series is a challenging problem with extensive work having been conducted. However, nonparametric probabilistic forecasting of time series, such as in the form of quantiles or prediction intervals is an even more challenging problem. In an effort to expand the possible forecasting p…
The paper introduces a knowledge score for GPR predictions to assess their reliability.
problem Uncertainty in probabilistic predictions from Gaussian process regression models.
method A knowledge score quantifying the reduction of uncertainty in GPR predictions.
result The knowledge score improves prediction accuracy in various tasks.
New method uses minimal assumptions for machine learning, improving performance and speed.
problem Current machine learning methods require specific model assumptions that are not derived from prior knowledge.
method Assumes scale invariance principles and differentiability of the true function to derive a novel stochastic process.
result The method achieves equal performance to Gaussian process regression but is less arbitrary, faster, and has better extrapolation.
Outcome regressed on class labels identified by unsupervised clustering is custom in many applications. However, it is common to ignore the misclassification of class labels caused by the learning algorithm, which potentially leads to serious bias of the estimated effect parameters. Due to its generality we suggest to …
RTE enables extrapolation to new tasks by learning task transformations.
problem Learning systems struggle to generalize to unseen tasks.
method Relational Task Extrapolator (RTE) learns task transformations to enable extrapolation.
result RTE substantially outperforms existing approaches on extrapolation tasks.
We extend nonparametric models to handle extrapolation, providing bounds for inference.
problem Challenges in nonparametric statistical inference when evaluating outside the conditioning variable's support.
method Introduced a class of extrapolation assumptions and a consistent estimation procedure to handle extrapolation.
result Validated extrapolation-aware conclusions through various applications and real-world data.
We present an approach to identify concise equations from data using a shallow neural network approach. In contrast to ordinary black-box regression, this approach allows understanding functional relations and generalizing them from observed data to unseen parts of the parameter space. We show how to extend the class o…
Despite its importance, choosing the structural form of the kernel in nonparametric regression remains a black art. We define a space of kernel structures which are built compositionally by adding and multiplying a small number of base kernels. We present a method for searching over this space of structures which mirro…
In this paper, we consider a class of possibly nonconvex, nonsmooth and non-Lipschitz optimization problems arising in many contemporary applications such as machine learning, variable selection and image processing. To solve this class of problems, we propose a proximal gradient method with extrapolation and line sear…
Binary encoding enables neural networks to extrapolate periodic functions.
problem Extrapolating periodic functions without prior knowledge of their form.
method Normalized Base-2 Encoding (NB2E) for continuous numerical values.
result MLPs using NB2E can successfully extrapolate diverse periodic signals.
Method controls extrapolation in prediction profiles for statistical and machine learning models.
problem Avoiding invalid predictions due to extrapolation in prediction profiles.
method Genetic algorithm optimization over constrained factor regions.
result Optimal factor settings without constraint are often invalid and extrapolated.
Neural networks struggle with extrapolation, but a new framework allows them to learn counterfactual invariances.
problem Neural networks' inability to extrapolate beyond training data distribution.
method Introduces a learning framework that allows neural networks to extrapolate over group transformations based on counterfactual invariances.
result Neural networks can learn counterfactual invariances from a single environment, overcoming their limitations in extrapolation.
Study shows LLMs can extrapolate rules from out-of-distribution prompts.
problem Understanding LLMs' ability to generalize from unexpected inputs.
method Formal languages and rule-based scenarios to evaluate LLMs' OOD behavior.
result LLMs can extrapolate rules from out-of-distribution prompts, even in complex scenarios.
Random forest (RF) methodology is one of the most popular machine learning techniques for prediction problems. In this article, we discuss some cases where random forests may suffer and propose a novel generalized RF method, namely regression-enhanced random forests (RERFs), that can improve on RFs by borrowing the str…
New methods reduce extrapolation errors in feature importance.
problem Flawed feature importance methods using unrestricted permutations lead to extrapolation errors.
method Three new approaches: conditional model reliance, Knockoffs with Gaussian transformation, and restricted ALE plot designs.
result Theoretical and numerical results show our strategies reduce/eliminate extrapolation.
The paper extends a spectral evolution model for link prediction in evolving networks.
problem Link prediction in evolving networks.
method Approximated eigenvalue trajectories using Rayleigh quotient and extrapolation.
result Learning algorithms based on approximated trajectories outperform traditional methods.
New models extrapolate false alarms in ASV without new data.
problem Reliable extrapolation of false alarm rates in ASV without new speaker data.
method Generative models in ASV score space for arbitrary systems.
result Models accurately extrapolate false alarm rates for large speaker populations.
Neural networks extrapolate poorly in simple tasks but succeed in complex ones.
problem Understanding neural networks' extrapolation capabilities and conditions for success.
method Analyzing ReLU MLPs and GNNs, connecting to neural tangent kernel.
result ReLU MLPs learn linear functions but not most nonlinear ones, while GNNs succeed in complex tasks due to task-specific non-linearities.
We extend return extrapolation to incorporate asymmetry and saturation, finding that asymmetric nonlinear extrapolation leads to lower welfare loss.
problem Optimal portfolio choice under stochastic volatility
method Smooth, nonlinear extrapolation function with sentiment and variance hedging
result Lower welfare loss with asymmetric nonlinear extrapolation
Logical neural networks solve mazes by filling dead ends, but not all methods generalize well.
problem Understanding how logical neural networks extrapolate solutions to mazes.
method Examined recurrent and implicit neural networks trained on maze-solving tasks.
result Models fail to generalize well to diverse maze sizes, suggesting limitations in learning scalable algorithms.
Study uses regression and ML for COVID-19 mortality forecasting.
problem Forecasting COVID-19 mortality during the first wave in Spain.
method Cyclical curve log-regression, multivariate time series spatial residual correlation analysis, Bayesian approach, machine learning.
result Empirical analysis shows ML regression models perform better than traditional methods.
Accelerates Riemannian gradient methods with extrapolation.
problem Optimizing functions on manifolds efficiently.
method Extrapolating iterates in Riemannian gradient descent.
result Achieves optimal convergence rate and computational advantage.
The paper explores how to make neural networks extrapolate longer sequences.
problem Neural networks struggle to extrapolate beyond seen data, especially for longer sequences.
method The authors propose a model with separate content- and location-based attention mechanisms.
result Models with the proposed attention mechanisms are better at extrapolating longer sequences.
Improves reinforcement learning extrapolation in Gridworlds.
problem Generalizing to unseen states in reinforcement learning.
method Avoiding deterministic action choice, using ego-centric representation, incorporating symmetry, and adding an entropy term.
result Significant improvement in extrapolation performance in Gridworld environments.
New methods improve neural network extrapolation to long sequences.
problem Neural networks struggle with extrapolation to very long or adversarial sequences.
method Activation binning and localized differentiable memory architecture.
result No extrapolation errors detected within memory constraints.
In this paper, we treat the problem of evaluating the asymptotic error in a numerical integration scheme as one with inherent uncertainty. Adding to the growing field of probabilistic numerics, we show that Gaussian process regression (GPR) can be embedded into a numerical integration scheme to allow for (i) robust sel…
FDN improves probabilistic regressors' adaptability to distribution shifts.
problem Overconfidence in modern probabilistic regressors under distribution shift.
method FDN uses input-conditioned distributions over network weights, trained with a Monte Carlo beta-ELBO objective.
result FDN produces predictive mixtures whose dispersion adapts to the input, providing shift-aware uncertainty.
Concept modulation models unify identifiability and extrapolation in conditional latent variable models.
problem Reliable generalization in conditional latent variable models
method Concept modulation models (CMMs) with structure AoΛoCoX result Lifts identifiability to conditional settings and controls extrapolation through attribute potentials.
New framework shows ERM is optimal for both interpolation and extrapolation in domain generalization.
problem Formalizing and solving the challenges of domain generalization.
method Reformulated domain generalization as an online game between a risk-minimizing player and an adversary.
result ERM is minimax-optimal for both interpolation and extrapolation in domain generalization.
NSR enables neural networks to reason with continuous numbers and extrapolate.
problem Quantitative reasoning and extrapolation in neural networks.
method Proposes Neural Status Register (NSR) for continuous number reasoning.
result NSR achieves extrapolation to numbers many orders of magnitude larger than training data.
Convex sparsity-inducing regularizations are ubiquitous in high-dimensional machine learning, but solving the resulting optimization problems can be slow. To accelerate solvers, state-of-the-art approaches consist in reducing the size of the optimization problem at hand. In the context of regression, this can be achiev…