We propose probabilistic models that can extrapolate learning curves of iterative machine learning algorithms, such as stochastic gradient descent for training deep networks, based on training data with variable-length learning curves. We study instantiations of this framework based on random forests and Bayesian recur…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “learning curve extrapolation”
New model predicts neural network performance from early training epochs, incorporating architecture impact.
problem Predicting neural network performance from early training epochs, neglecting architecture impact.
method Architecture-aware graph ordinary differential equation model.
result Model outperforms state-of-the-art methods for MLP and CNN learning curves.
LC-PFN predicts learning curve performance more accurately and faster than MCMC.
problem Bayesian extrapolation of learning curves is computationally expensive and overly restrictive.
method Prior-Data Fitted Neural Networks (PFNs) for approximate Bayesian inference.
result LC-PFN outperforms MCMC in accuracy and is significantly faster.
We present a framework on how to hedge the interest rate sensitivity of liabilities discounted by an extrapolated yield curve. The framework is based on functional analysis in that we consider the extrapolated yield curve as a functional of an observed yield curve and use its Gâteaux variation to understand the sensiti…
We use learning curves to analyze deep networks and evaluate model design.
problem Evaluate design choices in deep networks.
method Propose a method to robustly estimate learning curves, abstract their parameters, and evaluate different parameterizations.
result Interesting observations on the effectiveness of different parameterizations.
HAMLET optimizes algorithm selection for machine learning tasks.
problem Limited time budgets and computational resources make traditional bandit approaches ineffective for automated algorithm selection.
method HAMLET incorporates learning curve extrapolation and time-awareness to select machine learning algorithms.
result HAMLET variants outperform other bandit-based strategies in experiments with recorded hyperparameter tuning traces.
New benchmarks provide full training data for NAS research.
problem Limited training data on popular benchmarks restricts multi-fidelity techniques.
method SVD and noise modeling to create surrogate benchmarks with full training info.
result Learning curve extrapolation framework improves single-fidelity algorithms.
RTE enables extrapolation to new tasks by learning task transformations.
problem Learning systems struggle to generalize to unseen tasks.
method Relational Task Extrapolator (RTE) learns task transformations to enable extrapolation.
result RTE substantially outperforms existing approaches on extrapolation tasks.
New method extrapolates spectral densities from smaller models to larger ones.
problem Limited practical computations for large machine learning models.
method Algebraic spectral curve theory for free decompression.
result Framework enables extrapolation of spectral densities with multiple or multi-modal bulks.
Neural networks struggle with extrapolation, but a new framework allows them to learn counterfactual invariances.
problem Neural networks' inability to extrapolate beyond training data distribution.
method Introduces a learning framework that allows neural networks to extrapolate over group transformations based on counterfactual invariances.
result Neural networks can learn counterfactual invariances from a single environment, overcoming their limitations in extrapolation.
Method controls extrapolation in prediction profiles for statistical and machine learning models.
problem Avoiding invalid predictions due to extrapolation in prediction profiles.
method Genetic algorithm optimization over constrained factor regions.
result Optimal factor settings without constraint are often invalid and extrapolated.
Framework for transferring discount curve estimates across fixed-income product classes.
problem Challenges in estimating discount curves from sparse or noisy data.
method Proposes a vector-valued kernel ridge regression (KR) framework with economic regularization.
result Transfer learning tightens confidence intervals and improves extrapolation performance.
Improves reinforcement learning extrapolation in Gridworlds.
problem Generalizing to unseen states in reinforcement learning.
method Avoiding deterministic action choice, using ego-centric representation, incorporating symmetry, and adding an entropy term.
result Significant improvement in extrapolation performance in Gridworld environments.
Binary encoding enables neural networks to extrapolate periodic functions.
problem Extrapolating periodic functions without prior knowledge of their form.
method Normalized Base-2 Encoding (NB2E) for continuous numerical values.
result MLPs using NB2E can successfully extrapolate diverse periodic signals.
Logical neural networks solve mazes by filling dead ends, but not all methods generalize well.
problem Understanding how logical neural networks extrapolate solutions to mazes.
method Examined recurrent and implicit neural networks trained on maze-solving tasks.
result Models fail to generalize well to diverse maze sizes, suggesting limitations in learning scalable algorithms.
Paper improves PINNs' extrapolation by TL and adaptive AFs.
problem PINNs' poor extrapolation performance and sensitivity to AFs.
method Transfer learning within an extended domain and adaptive activation functions.
result Average 40% reduction in relative L2 error and 50% in mean absolute error in extrapolation domain.
Neural networks extrapolate poorly in simple tasks but succeed in complex ones.
problem Understanding neural networks' extrapolation capabilities and conditions for success.
method Analyzing ReLU MLPs and GNNs, connecting to neural tangent kernel.
result ReLU MLPs learn linear functions but not most nonlinear ones, while GNNs succeed in complex tasks due to task-specific non-linearities.
NSR enables neural networks to reason with continuous numbers and extrapolate.
problem Quantitative reasoning and extrapolation in neural networks.
method Proposes Neural Status Register (NSR) for continuous number reasoning.
result NSR achieves extrapolation to numbers many orders of magnitude larger than training data.
New framework shows ERM is optimal for both interpolation and extrapolation in domain generalization.
problem Formalizing and solving the challenges of domain generalization.
method Reformulated domain generalization as an online game between a risk-minimizing player and an adversary.
result ERM is minimax-optimal for both interpolation and extrapolation in domain generalization.
Study shows LLMs can extrapolate rules from out-of-distribution prompts.
problem Understanding LLMs' ability to generalize from unexpected inputs.
method Formal languages and rule-based scenarios to evaluate LLMs' OOD behavior.
result LLMs can extrapolate rules from out-of-distribution prompts, even in complex scenarios.
Kernel method estimates long-term effects from short-term data.
problem Estimating long-term effects from short-term data in continuous actions.
method Kernel ridge regression to embed and extrapolate long-term effects.
result Uniform consistency and nonasymptotic error bounds for the estimator.
Extreme value theory enhances statistical learning extrapolation for rare events.
problem Challenges in traditional machine learning methods for extreme data.
method Asymptotic theory and statistical tools for tail behavior.
result Effective extrapolation methods for extreme quantiles and anomalies.
Paper tackles intervention extrapolation using identifiable representations.
problem Predicting effects of unseen interventions on outcomes.
method Combines identifiable representation learning with autoencoders to enforce linear invariance.
result Identifiable representations enable non-linear extrapolation of interventions.
A new principle for extrapolating regression outside training data.
problem Regression extrapolation when predictions are outside training data range.
method Data-adaptive marginal transformation and simple relationship assumption.
result Progression method offers guarantees on approximation error beyond training data range.
Proposes a new method to avoid model extrapolation in Shapley values.
problem Model extrapolation in marginal Shapley values leads to unreliable explanations.
method Proposes a new approach that avoids model extrapolation using marginal averaging and causal information.
result Demonstrates the impacts of model extrapolation on Shapley values and proposes a new method to avoid it.
Methodology extrapolates wind fields from sparse data with uncertainty quantification.
problem Extrapolating wind fields from limited measurements with uncertainty.
method Nonparametric Bayesian dictionary learning for sparse/incomplete data.
result Enhanced extrapolation accuracy, even in high-dimensional data.
Asymmetric Nonlinear Return Extrapolation and Optimal Portfolio Choice under Stochastic Volatilityq-fin.PM
We extend return extrapolation to incorporate asymmetry and saturation, finding that asymmetric nonlinear extrapolation leads to lower welfare loss.
problem Optimal portfolio choice under stochastic volatility
method Smooth, nonlinear extrapolation function with sentiment and variance hedging
result Lower welfare loss with asymmetric nonlinear extrapolation
The performance of deep neural networks (DNN) is very sensitive to the particular choice of hyper-parameters. To make it worse, the shape of the learning curve can be significantly affected when a technique like batchnorm is used. As a result, hyperparameter optimization of deep networks can be much more challenging th…
Accelerates coordinate descent methods for machine learning problems.
problem Slowness of coordinate descent methods in machine learning.
method Extrapolation-based accelerated coordinate descent.
result Significant speed-up in practice compared to existing methods.
New methods improve neural network extrapolation to long sequences.
problem Neural networks struggle with extrapolation to very long or adversarial sequences.
method Activation binning and localized differentiable memory architecture.
result No extrapolation errors detected within memory constraints.
Study Q-learning with constant stepsize, proving convergence and bias, and applying extrapolation.
problem Understanding and optimizing Q-learning with constant stepsize.
method Connecting Q-learning to a Markov chain, proving distributional convergence and bias, applying Richardson-Romberg extrapolation.
result Explicit expression for the linear coefficient of the asymptotic bias and improvement of RR extrapolation method.
Concept modulation models unify identifiability and extrapolation in conditional latent variable models.
problem Reliable generalization in conditional latent variable models
method Concept modulation models (CMMs) with structure
result Lifts identifiability to conditional settings and controls extrapolation through attribute potentials.
Asymmetric Nonlinear Return Extrapolation and Optimal Portfolio Choice under Stochastic Volatilityq-fin.PM
Extends return extrapolation to nonlinear, asymmetric functions under stochastic volatility.
problem Behavioral anomalies in portfolio choice under stochastic volatility.
method Smooth, nonlinear, asymmetric extrapolation function; CRRA investor; Heston stochastic volatility; Hamilton-Jacobi-Bellman equation; Numerical solutions (finite-difference ADI, deep learning-driven iterative).
result Saturation acts as an endogenous correction mechanism, reducing welfare loss.
We extend nonparametric models to handle extrapolation, providing bounds for inference.
problem Challenges in nonparametric statistical inference when evaluating outside the conditioning variable's support.
method Introduced a class of extrapolation assumptions and a consistent estimation procedure to handle extrapolation.
result Validated extrapolation-aware conclusions through various applications and real-world data.
COVID-19 mortality analysis from soft-data multivariate curve regression and machine learningstat.ML
Study uses regression and ML for COVID-19 mortality forecasting.
problem Forecasting COVID-19 mortality during the first wave in Spain.
method Cyclical curve log-regression, multivariate time series spatial residual correlation analysis, Bayesian approach, machine learning.
result Empirical analysis shows ML regression models perform better than traditional methods.
STMT predicts compounds in unknown areas with trend reflection.
problem Predicting properties of unknown compounds in the extrapolation area.
method Stochastic Threshold Model Trees (STMT) that reflects data trends.
result STMT improves prediction accuracy for one compound in the extrapolation area.
Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscs.LG
A critical flaw of existing inverse reinforcement learning (IRL) methods is their inability to significantly outperform the demonstrator. This is because IRL typically seeks a reward function that makes the demonstrator appear near-optimal, rather than inferring the underlying intentions of the demonstrator that may ha…
BiPE blends intra-segment and inter-segment encodings for better length extrapolation.
problem Improving length extrapolation in language models.
method Bilevel Positional Encoding (BiPE) that separates intra-segment and inter-segment encodings.
result BiPE enhances length extrapolation across various text modalities.
This paper explores conditions for neural networks to extrapolate to new domains.
problem Understanding when neural networks can extrapolate to unseen domains.
method Analyzes conditions for nonlinear models to extrapolate under specific distribution shifts.
result Neural networks of the form can extrapolate if feature covariance is well-conditioned.
The Strength of Nesterov's Extrapolation in the Individual Convergence of Nonsmooth Optimizationcs.LG
Nesterov's extrapolation improves convergence in nonsmooth optimization.
problem Improving convergence rate in nonsmooth convex optimization.
method Nesterov's extrapolation applied to projected subgradient methods.
result Nesterov's extrapolation optimizes individual convergence for nonsmooth problems.
Implicit Bias of Policy Gradient in Linear Quadratic Control: Extrapolation to Unseen Initial Statescs.LG
This paper studies how gradient descent in control systems can perform well on unseen data.
problem The extent of a learned controller's ability to extrapolate to unseen initial states.
method Theoretical study of policy gradient in Linear Quadratic Regulator (LQR) problems, focusing on the role of exploration.
result The performance of a learned controller on unseen initial states depends on the degree of exploration induced by the system.
Paper tackles multi-object reinforcement learning, improving skill extrapolation.
problem Learning robust manipulation tasks in multi-object settings.
method Introduces a linear relation network module to enhance skill generalization.
result Agents can extrapolate and generalize to any new object number, scaling linearly.
The Human Kernelcs.LG
Bayesian nonparametric models, such as Gaussian processes, provide a compelling framework for automatic statistical modelling: these models have a high degree of flexibility, and automatically calibrated complexity. However, automating human expertise remains elusive; for example, Gaussian processes with standard kerne…
Interest-Rate Modelling in Collateralized Markets: Multiple curves, credit-liquidity effects, CCPsq-fin.PR
The market practice of extrapolating different term structures from different instruments lacks a rigorous justification in terms of cash flows structure and market observables. In this paper, we integrate our previous consistent theory for pricing under credit, collateral and funding risks into term structure modellin…
State-space models improve dynamical system predictions efficiently and accurately.
problem Challenges in predicting dynamical systems, including long-time integration and long-range dependencies.
method State-space models implemented in Mamba, addressing limitations of existing architectures.
result Mamba outperforms other models in interpolation and challenging extrapolation tasks.
The recent advent of automated neural network architecture search led to several methods that outperform state-of-the-art human-designed architectures. However, these approaches are computationally expensive, in extreme cases consuming GPU years. We propose two novel methods which aim to expedite this optimization prob…
Repo convexityq-fin.PR
There is an observed basis between repo discounting, implied from market repo rates, and bond discounting, stripped from the market prices of the underlying bonds. Here, this basis is explained as a convexity effect arising from the decorrelation between the discount rates for derivatives and bonds. Using a Hull-White …
Learnable token perturbations boost extrapolation in LLMs.
problem Limited flexibility of current discrete perturbations in large language models.
method Learnable continuous latent vector transformations in embedding space, unbiased estimating equations, stochastic gradient descent optimization.
result Significant gains in out-of-domain settings over state-of-the-art methods.