RTE enables extrapolation to new tasks by learning task transformations.
problem Learning systems struggle to generalize to unseen tasks.
method Relational Task Extrapolator (RTE) learns task transformations to enable extrapolation.
result RTE substantially outperforms existing approaches on extrapolation tasks.
New methods improve neural network extrapolation to long sequences.
problem Neural networks struggle with extrapolation to very long or adversarial sequences.
method Activation binning and localized differentiable memory architecture.
result No extrapolation errors detected within memory constraints.
Method controls extrapolation in prediction profiles for statistical and machine learning models.
problem Avoiding invalid predictions due to extrapolation in prediction profiles.
method Genetic algorithm optimization over constrained factor regions.
result Optimal factor settings without constraint are often invalid and extrapolated.
Study shows LLMs can extrapolate rules from out-of-distribution prompts.
problem Understanding LLMs' ability to generalize from unexpected inputs.
method Formal languages and rule-based scenarios to evaluate LLMs' OOD behavior.
result LLMs can extrapolate rules from out-of-distribution prompts, even in complex scenarios.
Logical neural networks solve mazes by filling dead ends, but not all methods generalize well.
problem Understanding how logical neural networks extrapolate solutions to mazes.
method Examined recurrent and implicit neural networks trained on maze-solving tasks.
result Models fail to generalize well to diverse maze sizes, suggesting limitations in learning scalable algorithms.
In this paper, we propose a general framework to accelerate significantly the algorithms for nonnegative matrix factorization (NMF). This framework is inspired from the extrapolation scheme used to accelerate gradient methods in convex optimization and from the method of parallel tangents. However, the use of extrapola…
Paper analyzes LSA algorithm bias and error bounds with RR extrapolation.
problem Analyzing bias and high-order error bounds of LSA with Markovian noise.
method Polyak-Ruppert averaging, linearization, Richardson-Romberg extrapolation.
result RR extrapolation effectively cancels the leading bias term.
Iteratively reweighted ℓ1 algorithm is a popular algorithm for solving a large class of optimization problems whose objective is the sum of a Lipschitz differentiable loss function and a possibly nonconvex sparsity inducing regularizer. In this paper, motivated by the success of extrapolation techniques in accele…
Neural networks extrapolate poorly in simple tasks but succeed in complex ones.
problem Understanding neural networks' extrapolation capabilities and conditions for success.
method Analyzing ReLU MLPs and GNNs, connecting to neural tangent kernel.
result ReLU MLPs learn linear functions but not most nonlinear ones, while GNNs succeed in complex tasks due to task-specific non-linearities.
New framework shows ERM is optimal for both interpolation and extrapolation in domain generalization.
problem Formalizing and solving the challenges of domain generalization.
method Reformulated domain generalization as an online game between a risk-minimizing player and an adversary.
result ERM is minimax-optimal for both interpolation and extrapolation in domain generalization.
We propose probabilistic models that can extrapolate learning curves of iterative machine learning algorithms, such as stochastic gradient descent for training deep networks, based on training data with variable-length learning curves. We study instantiations of this framework based on random forests and Bayesian recur…
The paper explores how to extrapolate from limited data points using causal mechanisms.
problem Handling distribution shifts with limited target samples.
method Formulates the extrapolation problem with a latent-variable model embodying the minimal change principle in causal mechanisms, and identifies conditions for identification.
result Theoretical understanding and practical methods for extrapolation without requiring an on-support target distribution.
Shape-constrained symbolic regression improves model extrapolation with prior knowledge.
problem Improving model extrapolation with prior knowledge in symbolic regression.
method Shape-constrained symbolic regression using evolutionary algorithms with interval arithmetic.
result Models with shape constraints have improved extrapolation but lower accuracy on test sets.
This paper is concerned with improving the empirical convergence speed of block-coordinate descent algorithms for approximate nonnegative tensor factorization (NTF). We propose an extrapolation strategy in-between block updates, referred to as heuristic extrapolation with restarts (HER). HER significantly accelerates t…
In this paper, we study the proximal gradient algorithm with extrapolation for minimizing the sum of a Lipschitz differentiable function and a proper closed convex function. Under the error bound condition used in [19] for analyzing the convergence of the proximal gradient algorithm, we show that there exists a thresho…
Prior information can be incorporated in matrix completion to improve estimation accuracy and extrapolate the missing entries. Reproducing kernel Hilbert spaces provide tools to leverage the said prior information, and derive more reliable algorithms. This paper analyzes the generalization error of such approaches, and…
Study Q-learning with constant stepsize, proving convergence and bias, and applying extrapolation.
problem Understanding and optimizing Q-learning with constant stepsize.
method Connecting Q-learning to a Markov chain, proving distributional convergence and bias, applying Richardson-Romberg extrapolation.
result Explicit expression for the linear coefficient of the asymptotic bias and improvement of RR extrapolation method.
We extend nonparametric models to handle extrapolation, providing bounds for inference.
problem Challenges in nonparametric statistical inference when evaluating outside the conditioning variable's support.
method Introduced a class of extrapolation assumptions and a consistent estimation procedure to handle extrapolation.
result Validated extrapolation-aware conclusions through various applications and real-world data.
Nesterov's extrapolation improves convergence in nonsmooth optimization.
problem Improving convergence rate in nonsmooth convex optimization.
method Nesterov's extrapolation applied to projected subgradient methods.
result Nesterov's extrapolation optimizes individual convergence for nonsmooth problems.
Kan extensions help in data science extrapolation and learning.
problem Generalizing functions over larger sets in data science.
method Kan extensions in category theory applied to data science problems.
result Kan extensions can be used to derive classification and clustering algorithms.
Binary encoding enables neural networks to extrapolate periodic functions.
problem Extrapolating periodic functions without prior knowledge of their form.
method Normalized Base-2 Encoding (NB2E) for continuous numerical values.
result MLPs using NB2E can successfully extrapolate diverse periodic signals.
Neural networks struggle with extrapolation, but a new framework allows them to learn counterfactual invariances.
problem Neural networks' inability to extrapolate beyond training data distribution.
method Introduces a learning framework that allows neural networks to extrapolate over group transformations based on counterfactual invariances.
result Neural networks can learn counterfactual invariances from a single environment, overcoming their limitations in extrapolation.
A critical flaw of existing inverse reinforcement learning (IRL) methods is their inability to significantly outperform the demonstrator. This is because IRL typically seeks a reward function that makes the demonstrator appear near-optimal, rather than inferring the underlying intentions of the demonstrator that may ha…
Paper examines constant stepsize in LSA for Markovian data inference.
problem Improving statistical inference with constant stepsize in LSA for Markovian data.
method Established CLT, used averaged LSA iterates, applied Richardson-Romberg extrapolation.
result Constant stepsize leads to better CI coverage, especially with limited data.
Study on bias and extrapolation in LSA with Markovian data, showing bias reduction with Richardson-Romberg extrapolation.
problem Bias in LSA with constant stepsizes and Markovian data.
method Viewing LSA as a Markov chain, proving convergence and bias expansion, and applying Richardson-Romberg extrapolation.
result Bias is proportional to the stepsize up to higher order terms, and Richardson-Romberg extrapolation reduces the bias.
Imitation learning, followed by reinforcement learning algorithms, is a promising paradigm to solve complex control tasks sample-efficiently. However, learning from demonstrations often suffers from the covariate shift problem, which results in cascading errors of the learned policy. We introduce a notion of conservati…
New methods reduce extrapolation errors in feature importance.
problem Flawed feature importance methods using unrestricted permutations lead to extrapolation errors.
method Three new approaches: conditional model reliance, Knockoffs with Gaussian transformation, and restricted ALE plot designs.
result Theoretical and numerical results show our strategies reduce/eliminate extrapolation.
New models extrapolate false alarms in ASV without new data.
problem Reliable extrapolation of false alarm rates in ASV without new speaker data.
method Generative models in ASV score space for arbitrary systems.
result Models accurately extrapolate false alarm rates for large speaker populations.
A new principle for extrapolating regression outside training data.
problem Regression extrapolation when predictions are outside training data range.
method Data-adaptive marginal transformation and simple relationship assumption.
result Progression method offers guarantees on approximation error beyond training data range.
We extend return extrapolation to incorporate asymmetry and saturation, finding that asymmetric nonlinear extrapolation leads to lower welfare loss.
problem Optimal portfolio choice under stochastic volatility
method Smooth, nonlinear extrapolation function with sentiment and variance hedging
result Lower welfare loss with asymmetric nonlinear extrapolation
Accelerates Riemannian gradient methods with extrapolation.
problem Optimizing functions on manifolds efficiently.
method Extrapolating iterates in Riemannian gradient descent.
result Achieves optimal convergence rate and computational advantage.
FedExProx's performance is no better than GD for quadratic optimization.
problem Improving convergence of parallel proximal algorithms.
method Developed a novel analysis framework to establish tighter convergence rates.
result FedExProx can outperform GD in non-strongly convex quadratic problems.
Paper analyzes \FedAvg's convergence and introduces a new algorithm to reduce bias.
problem Analyzing convergence and bias in Federated Averaging.
method Markov property, first-order bias expansion, Richardson-Romberg extrapolation.
result Bias in \FedAvg can be decomposed into noise and client heterogeneity components.
Improves reinforcement learning extrapolation in Gridworlds.
problem Generalizing to unseen states in reinforcement learning.
method Avoiding deterministic action choice, using ego-centric representation, incorporating symmetry, and adding an entropy term.
result Significant improvement in extrapolation performance in Gridworld environments.
Concept modulation models unify identifiability and extrapolation in conditional latent variable models.
problem Reliable generalization in conditional latent variable models
method Concept modulation models (CMMs) with structure AoΛoCoX result Lifts identifiability to conditional settings and controls extrapolation through attribute potentials.
NSR enables neural networks to reason with continuous numbers and extrapolate.
problem Quantitative reasoning and extrapolation in neural networks.
method Proposes Neural Status Register (NSR) for continuous number reasoning.
result NSR achieves extrapolation to numbers many orders of magnitude larger than training data.
Extends return extrapolation to nonlinear, asymmetric functions under stochastic volatility.
problem Behavioral anomalies in portfolio choice under stochastic volatility.
method Smooth, nonlinear, asymmetric extrapolation function; CRRA investor; Heston stochastic volatility; Hamilton-Jacobi-Bellman equation; Numerical solutions (finite-difference ADI, deep learning-driven iterative).
result Saturation acts as an endogenous correction mechanism, reducing welfare loss.
We suggest ways to enforce given constraints in the output of a Generative Adversarial Network (GAN) generator both for interpolation and extrapolation (prediction). For the case of dynamical systems, given a time series, we wish to train GAN generators that can be used to predict trajectories starting from a given ini…
This paper explores conditions for neural networks to extrapolate to new domains.
problem Understanding when neural networks can extrapolate to unseen domains.
method Analyzes conditions for nonlinear models to extrapolate under specific distribution shifts.
result Neural networks of the form f(x)=∑fi(xi) can extrapolate if feature covariance is well-conditioned. Diverging Flows detects extrapolations in flow models, ensuring reliable predictions.
problem Flow models extrapolate into invalid data, leading to silent failures.
method Structurally enforce inefficient transport for off-manifold inputs.
result Effective detection of extrapolations without compromising predictive fidelity or inference latency.
Proposes a new method to avoid model extrapolation in Shapley values.
problem Model extrapolation in marginal Shapley values leads to unreliable explanations.
method Proposes a new approach that avoids model extrapolation using marginal averaging and causal information.
result Demonstrates the impacts of model extrapolation on Shapley values and proposes a new method to avoid it.
The paper analyzes SGD with Richardson-Romberg extrapolation for convex optimization problems.
problem Solving strongly convex and smooth minimization problems efficiently.
method Combining SGD with Polyak-Ruppert averaging and Richardson-Romberg extrapolation.
result An expansion of the mean-squared error of the estimator with respect to the number of iterations.
The paper extends a spectral evolution model for link prediction in evolving networks.
problem Link prediction in evolving networks.
method Approximated eigenvalue trajectories using Rayleigh quotient and extrapolation.
result Learning algorithms based on approximated trajectories outperform traditional methods.
Paper tackles intervention extrapolation using identifiable representations.
problem Predicting effects of unseen interventions on outcomes.
method Combines identifiable representation learning with autoencoders to enforce linear invariance.
result Identifiable representations enable non-linear extrapolation of interventions.
Paper improves PINNs' extrapolation by TL and adaptive AFs.
problem PINNs' poor extrapolation performance and sensitivity to AFs.
method Transfer learning within an extended domain and adaptive activation functions.
result Average 40% reduction in relative L2 error and 50% in mean absolute error in extrapolation domain.
We present a framework on how to hedge the interest rate sensitivity of liabilities discounted by an extrapolated yield curve. The framework is based on functional analysis in that we consider the extrapolated yield curve as a functional of an observed yield curve and use its Gâteaux variation to understand the sensiti…
The paper tackles extrapolation in generative models by enforcing independence of mechanisms.
problem How to make generative models extrapolate to new, unseen environments?
method Developed a theoretical framework for independence of mechanisms, demonstrated on toy examples and real-world data.
result Extrapolation capabilities of generative models can be improved by enforcing independence of mechanisms explicitly during training.
Methodology extrapolates wind fields from sparse data with uncertainty quantification.
problem Extrapolating wind fields from limited measurements with uncertainty.
method Nonparametric Bayesian dictionary learning for sparse/incomplete data.
result Enhanced extrapolation accuracy, even in high-dimensional data.