Automatic differentiation---the mechanical transformation of numeric computer programs to calculate derivatives efficiently and accurately---dates to the origin of the computer age. Reverse mode automatic differentiation both antedates and generalizes the method of backwards propagation of errors used in machine learni…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Automatic Differentiation”
Develops a mathematical model for automatic differentiation in machine learning.
problem Current automatic differentiation lacks a simple mathematical model for machine learning.
method Articulates relationships between program differentiation and nonsmooth functions, provides a class of functions and nonsmooth calculus.
result Shows how nonsmooth calculus applies to stochastic approximation methods and evidence of artificial critical points.
We present a system for the automatic differentiation of a higher-order functional array-processing language. The core functional language underlying this system simultaneously supports both source-to-source automatic differentiation and global optimizations such as loop transformations. Thanks to this feature, we demo…
Develops algorithm to differentiate Metropolis-Hastings for optimization.
problem Optimizing intractable densities with discrete components.
method Fuses stochastic automatic differentiation with Markov chain coupling schemes.
result Unbiased and low-variance gradient estimator for intractable densities.
A method makes particle filters differentiable without altering their forward pass.
problem Compatibility issues between particle filters and automatic differentiation.
method Introduces a correction to particle weights using the stop-gradient operator.
result Automatic differentiation produces good estimators for gradients and second-order derivatives.
This paper deals with the computation of second or higher order greeks of financial securities. It combines two methods, Vibrato and automatic differentiation and compares with other methods. We show that this combined technique is faster than standard finite difference, more stable than automatic differentiation of se…
Automatic Backward Differentiation for American Monte-Carlo Algorithms (Conditional Expectation)q-fin.CP
In this note we derive the backward (automatic) differentiation (adjoint [automatic] differentiation) for an algorithm containing a conditional expectation operator. As an example we consider the backward algorithm as it is used in Bermudan product valuation, but the method is applicable in full generality. The method …
Geometric AD framework simplifies derivative computation in JAX.
problem Efficient and accurate automatic differentiation.
method Jet functors and Weil algebras for geometric analysis.
result Unified view of derivative propagation with algebraic exactness.
Simplifies efficient estimation via automatic differentiation and probabilistic programming.
problem Constructing efficient estimators for complex statistical models.
method Automatic differentiation applied to statistical functionals, avoiding the need to derive efficient influence functions.
result Users can generate efficient estimators with minimal code, simplifying the process for non-experts.
We use AD to compute gradients for complex functionals in stochastic model calibration.
problem Computing gradients for functions involving expectations in stochastic models.
method Automatic Adjoint Differentiation and parallelization.
result Faster and easier to implement approaches for gradient computation.
Proposes efficient, modular method for implicit differentiation.
problem Implicit differentiation of optimization problems.
method Automatic implicit differentiation using autodiff and implicit function theorem.
result Automatic differentiation of optimization problems is made easier and more modular.
In this note, we report the back propagation formula for complex valued singular value decompositions (SVD). This formula is an important ingredient for a complete automatic differentiation(AD) infrastructure in terms of complex numbers, and it is also the key to understand and utilize AD in tensor networks.
Functional AD for Weil algebra computations.
problem Efficient computation of -structures on Weil algebras.
method Multivariate Tower Automatic Differentiation (AD) implementation.
result Functional AD for Weil algebra computation.
Automatic differentiation (AD) is an essential primitive for machine learning programming systems. Tangent is a new library that performs AD using source code transformation (SCT) in Python. It takes numeric functions written in a syntactic subset of Python and NumPy as input, and generates new Python functions which c…
Automatically differentiable estimation for BLP model reduces bias in demand estimation.
problem Estimating the BLP model with reduced bias and improved performance.
method Phrasing BLP as an automatically differentiable moment function, using CUE for estimation, and incorporating MCMC credible intervals.
result CUE estimation shows lower bias but higher MAE compared to 2S-GMM, with MCMC providing closest empirical coverage.
Proposes a method to train neural networks that solve differential equations faster.
problem Training neural networks that solve differential equations becomes computationally expensive.
method Introduces a differentiable surrogate for numerical solver time cost using higher-order derivatives.
result Trains models that are faster to solve while maintaining nearly the same accuracy.
Integral Transforms in a Physics-Informed (Quantum) Neural Network setting: Applications & Use-Casesquant-ph
This work integrates differentiation and integration in Physics-Informed Neural Networks.
problem Solving integro-differential equations and computing integral transforms.
method Augmenting Physics-Informed Neural Networks with automatic integration.
result Solving complex integral transforms and integro-differential equations.
Two of the most important areas in computational finance: Greeks and, respectively, calibration, are based on efficient and accurate computation of a large number of sensitivities. This paper gives an overview of adjoint and automatic differentiation (AD), also known as algorithmic differentiation, techniques to calcul…
Automatic differentiation is involved for long in applied mathematics as an alternative to finite difference to improve the accuracy of numerical computation of derivatives. Each time a numerical minimization is involved, automatic differentiation can be used. In between formal derivation and standard numerical schemes…
Derivatives, mostly in the form of gradients and Hessians, are ubiquitous in machine learning. Automatic differentiation (AD), also called algorithmic differentiation or simply "autodiff", is a family of techniques similar to but more general than backpropagation for efficiently and accurately evaluating derivatives of…
Study shows AD for neural nets with machine-representable numbers can be incorrect.
problem Correctness of AD for neural nets with machine-representable numbers.
method Analyzed two sets of parameters: incorrect and non-differentiable. Proved bounds and conditions for AD correctness.
result AD can be incorrect for machine-representable numbers, but provides a Clarke subderivative on non-differentiable set.
AD-HOC simplifies high-order derivative calculations in C++.
problem Efficiently computing high-order derivatives in C++.
method A C++ package that calculates derivatives of arbitrary order without code generation.
result Derivatives of arbitrary order computed in a single pass.
Paper learns dictionaries for sparse signal recovery using automatic differentiation.
problem Learning dictionaries for sparse signal recovery from noisy data.
method Approximates reconstructions using FB algorithm and learns dictionaries with projected gradient descent.
result Successfully learns 1D TV dictionary from piecewise constant signals.
In this work, we discuss the Automatic Adjoint Differentiation (AAD) for functions of the form , which often appear in the calibration of stochastic models. { We demonstrate that it allows a perfect SIMD\footnote{Single Input Multiple Data} parallelization and provide its relative co…
The paper proposes a method to compute higher infinitesimals in numerical and symbolic analysis.
problem Computing higher-order derivatives with higher infinitesimals.
method Automatic differentiation in terms of C-infinity rings and Weil algebras.
result A unifying theoretical framework for multivariate higher-order derivatives.
This paper introduces a spline-based method for nonparametric ADVI that handles complex posterior distributions.
problem Learning complex posterior distributions with skewness, multimodality, and bounded support.
method Develops a spline-based nonparametric approximation approach for ADVI.
result Establishes the asymptotic consistency of the derived lower bound for importance weighted autoencoder.
RAD estimates gradients with less memory, faster than small batch sizes.
problem Training deep models with stochastic gradient descent requires exact gradients, but they are not needed.
method Developed a framework for randomized automatic differentiation (RAD) to compute unbiased gradient estimates with reduced memory.
result RAD converges in fewer iterations than using a small batch size for feedforward networks and similar number for recurrent networks.
Storchastic improves stochastic AD for complex models in RL and VI.
problem Handling intractable expectations in RL and VI.
method Introduces Storchastic, a framework for AD of stochastic computation graphs with various gradient estimation methods.
result Provable unbiasedness and variance reduction for higher-order gradients.
CoLA automates efficient numerical linear algebra for complex matrix structures.
problem Efficiently solving large-scale linear algebra problems with complex matrix structures.
method Combining linear operator abstraction with compositional dispatch rules.
result Automatic and efficient numerical algorithms for various linear algebra operations.
Teaches matrix calculus for machine learning and optimization.
problem Computing derivatives of functions involving matrices.
method Extends differential calculus to vector spaces, focusing on practical applications in machine learning.
result Introduction of adjoint and reverse-mode differentiation for efficient computation.
This paper extends AD techniques to Monte Carlo processes for efficient derivative calculation.
problem Obtaining derivatives of expectation values in Monte Carlo processes.
method Two approaches: reweighting and Hamiltonian extension of HMC.
result Hamiltonian approach as a change of variables simplifies variance reduction.
Improves numerical solution of ill-conditioned linear systems for machine learning.
problem Wastefulness and instability in solving ill-conditioned linear systems.
method autonugget combines Richardson extrapolation to determine the solution of the ill-conditioned system, improving accuracy over a single nugget.
result Improves accuracy of numerical solution of ill-conditioned linear systems.
ADVI speeds up Bayesian inference for bridge regression models.
problem Slow MCMC for large datasets in bridge regression.
method Automatic Differentiation Variational Inference (ADVI) for Bayesian inference.
result ADVI implementation speeds up inference for large datasets.
A JAX toolbox solves optimal transport problems for point clouds and histograms.
problem Optimal transport problems between point clouds and histograms.
method Automatic and custom reverse mode differentiation, vectorization, just-in-time compilation, and accelerators support.
result Solves a wide range of optimal transport problems including regularized OT, barycenters, Gromov-Wasserstein, and low-rank solvers.
Probabilistic modeling is iterative. A scientist posits a simple model, fits it to her data, refines it according to her analysis, and repeats. However, fitting complex models to large data is a bottleneck in this process. Deriving algorithms for new models can be both mathematically and computationally challenging, wh…
Accelerated Inference for Partially Observed Markov Processes using Automatic Differentiationstat.ME
New AD methods improve likelihood estimation for partially observed systems.
problem Estimating likelihood functions for partially observed nonlinear systems.
method Embedding AD particle filter methods in a theoretical framework, developing new algorithms for likelihood maximization.
result Mean squared error significantly lower than existing algorithms.
New method speeds up Bayesian optimization in high dimensions.
problem High-dimensional expensive function optimization struggles.
method Structured automatic differentiation for kernel matrices.
result First-order Bayesian optimization scalable to high dimensions.
We develop nested automatic differentiation (AD) algorithms for exact inference and learning in integer latent variable models. Recently, Winner, Sujono, and Sheldon showed how to reduce marginalization in a class of integer latent variable models to evaluating a probability generating function which contains many leve…
The need to efficiently calculate first- and higher-order derivatives of increasingly complex models expressed in Python has stressed or exceeded the capabilities of available tools. In this work, we explore techniques from the field of automatic differentiation (AD) that can give researchers expressive power, performa…
In min-min optimization or max-min optimization, one has to compute the gradient of a function defined as a minimum. In most cases, the minimum has no closed-form, and an approximation is obtained via an iterative algorithm. There are two usual ways of estimating the gradient of the function: using either an analytic f…
pySigLib speeds up signature-based computations on CPUs and GPUs.
problem Efficient signature-based computations on large datasets and long sequences.
method Optimised Python library for CPU and GPU, novel differentiation scheme.
result Accurate gradients at a fraction of the runtime of existing libraries.
A new method computes gradients without backpropagation.
problem Optimization of machine learning models.
method Forward mode automatic differentiation to compute gradients.
result Forward gradient is an unbiased estimate of the gradient, eliminating the need for backpropagation.
Banded matrices can be used as precision matrices in several models including linear state-space models, some Gaussian processes, and Gaussian Markov random fields. The aim of the paper is to make modern inference methods (such as variational inference or gradient-based sampling) available for Gaussian models with band…
New methods reduce private federated learning communication automatically.
problem Reducing communication in private federated learning.
method Automatic compression rate adjustment based on training error, using secure aggregation and differential privacy.
result Provable instance-optimal for mean estimation, achieving favorable compression rates.
Efficient implicit differentiation for Lasso hyperparameter optimization.
problem Difficult hyperparameter optimization for Lasso-type models.
method Implicit differentiation algorithm tailored for Lasso-type problems, avoiding matrix inversion and solving linear systems.
result Outperforms standard methods in optimizing error on held-out data or Stein Unbiased Risk Estimator (SURE).
Develops STC for sequential data with missing labels.
problem Learning from partially labeled and unsegmented sequential data.
method Introduces Star Temporal Classification (STC) using a star token and GTN framework.
result Recover most of supervised baseline performance with up to 70% missing labels.
Differential Machine Learningq-fin.CP
Differential ML combines AAD with ML for fast, accurate financial derivatives pricing and risk management.
problem Computational bottlenecks in financial derivatives risk management.
method Novel algorithms using automatic adjoint differentiation (AAD) for training fast, accurate approximations in real-time.
result Convergence guarantees for fast, accurate pricing and risk approximations for arbitrary derivatives instruments.
Variational inference is a scalable technique for approximate Bayesian inference. Deriving variational inference algorithms requires tedious model-specific calculations; this makes it difficult to automate. We propose an automatic variational inference algorithm, automatic differentiation variational inference (ADVI). …