Deep learning has seen tremendous success over the past decade in computer vision, machine translation, and gameplay. This success rests in crucial ways on gradient-descent optimization and the ability to learn parameters of a neural network by backpropagating observed errors. However, neural network architectures are …
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “Reverse-mode Differentiation”
Automatic differentiation---the mechanical transformation of numeric computer programs to calculate derivatives efficiently and accurately---dates to the origin of the computer age. Reverse mode automatic differentiation both antedates and generalizes the method of backwards propagation of errors used in machine learni…
Teaches matrix calculus for machine learning and optimization.
problem Computing derivatives of functions involving matrices.
method Extends differential calculus to vector spaces, focusing on practical applications in machine learning.
result Introduction of adjoint and reverse-mode differentiation for efficient computation.
The need to efficiently calculate first- and higher-order derivatives of increasingly complex models expressed in Python has stressed or exceeded the capabilities of available tools. In this work, we explore techniques from the field of automatic differentiation (AD) that can give researchers expressive power, performa…
A new method for optimizing language-based agentic systems using semantic backpropagation.
problem Lack of proper feedback assignment in optimizing agentic systems.
method Formalization of semantic backpropagation with semantic gradients and semantic gradient descent.
result Our method outperforms existing state-of-the-art methods for solving GASO problems.
ACA method improves gradient estimation for neural ODEs, reducing error and training time.
problem Inaccurate gradient estimation methods hinder the performance of neural ODEs on benchmark tasks.
method Adaptive Checkpoint Adjoint (ACA) method that applies trajectory checkpointing, deletes redundant components, and supports adaptive solvers.
result ACA reduces error rate by half and training time by half compared to adjoint and naive methods on image classification tasks.
A JAX toolbox solves optimal transport problems for point clouds and histograms.
problem Optimal transport problems between point clouds and histograms.
method Automatic and custom reverse mode differentiation, vectorization, just-in-time compilation, and accelerators support.
result Solves a wide range of optimal transport problems including regularized OT, barycenters, Gromov-Wasserstein, and low-rank solvers.
Banded matrices can be used as precision matrices in several models including linear state-space models, some Gaussian processes, and Gaussian Markov random fields. The aim of the paper is to make modern inference methods (such as variational inference or gradient-based sampling) available for Gaussian models with band…
Geometric AD framework simplifies derivative computation in JAX.
problem Efficient and accurate automatic differentiation.
method Jet functors and Weil algebras for geometric analysis.
result Unified view of derivative propagation with algebraic exactness.
A new method computes gradients without backpropagation.
problem Optimization of machine learning models.
method Forward mode automatic differentiation to compute gradients.
result Forward gradient is an unbiased estimate of the gradient, eliminating the need for backpropagation.
We study two procedures (reverse-mode and forward-mode) for computing the gradient of the validation error with respect to the hyperparameters of any iterative learning algorithm such as stochastic gradient descent. These procedures mirror two methods of computing gradients for recurrent neural networks and have differ…
AutoBayes simplifies variational inference by composing models and optimizing them.
problem Generalized variational inference complexities and optimization challenges.
method Compositional framework exploiting chain rules for automatic differentiation.
result Optimized models and parameterized statistical games can be locally optimized.
Study shows AD for neural nets with machine-representable numbers can be incorrect.
problem Correctness of AD for neural nets with machine-representable numbers.
method Analyzed two sets of parameters: incorrect and non-differentiable. Proved bounds and conditions for AD correctness.
result AD can be incorrect for machine-representable numbers, but provides a Clarke subderivative on non-differentiable set.
Study compares methods for computing hypergradients in machine learning problems.
problem Computing exact hypergradients in machine learning is difficult.
method Investigates reverse mode iterative differentiation and approximate implicit differentiation methods.
result Unified analysis provides iteration complexity bounds and hierarchy of methods.
We introduce a general method for improving the convergence rate of gradient-based optimizers that is easy to implement and works well in practice. We demonstrate the effectiveness of the method in a range of optimization problems by applying it to stochastic gradient descent, stochastic gradient descent with Nesterov …
The paper generalizes optimization algorithms using category theory.
problem Optimizing functions in a category-theoretic setting.
method Using the Cartesian reverse derivative to generalize gradient descent and Newton's method.
result Properties of optimization algorithms are preserved in the generalized setting, including invariances and convergence.
RAD estimates gradients with less memory, faster than small batch sizes.
problem Training deep models with stochastic gradient descent requires exact gradients, but they are not needed.
method Developed a framework for randomized automatic differentiation (RAD) to compute unbiased gradient estimates with reduced memory.
result RAD converges in fewer iterations than using a small batch size for feedforward networks and similar number for recurrent networks.
Score matching is a popular method for estimating unnormalized statistical models. However, it has been so far limited to simple, shallow models or low-dimensional data, due to the difficulty of computing the Hessian of log-density functions. We show this difficulty can be mitigated by projecting the scores onto random…
Learning Latent Graph Geometry via Fixed-Point Schrödinger-Type Activation: A Theoretical Studycs.LG
Study neural architectures on learned latent graphs using Schrödinger dynamics.
problem Understanding neural architectures on learned latent graphs.
method Optimizes over stratified moduli space of weighted graphs with Kähler-Hessian metric.
result Multilayer stationary networks are equivalent to global stationary problems on supra-graphs.
The deep learning community has devised a diverse set of methods to make gradient optimization, using large datasets, of large and highly complex models with deeply cascaded nonlinearities, practical. Taken as a whole, these methods constitute a breakthrough, allowing computational structures which are quite wide, very…
NDDV estimates data point value from a single stochastic trajectory.
problem Estimating marginal contributions of data points over stochastic training paths.
method Introduces Neural Dynamic Data Valuation (NDDV) using stochastic state and adjoint equations.
result NDDV provides a one-run, trajectory-conditioned estimator of data point value.
The closure conditions of the inexact exterior differential form and dual form (an equality to zero of differentials of these forms) can be treated as a definition of some differential-geometrical structure. Such a connection discloses the properties and specific features of the differential-geometrical structures. The…
Differential completions and compactifications of differential spaces are introduced and investigated. The existence of the maximal differential completion and the maximal differential compactification is proved. A sufficient condition for the existence of a complete uniform differential structure on a given differenti…
In this paper we give explicit formulas of differential characteristic classes of principal -bundles with connections and prove their expected properties. In particular, we obtain explicit formulas for differential Chern classes, differential Pontryagin classes and differential Euler class. Furthermore, we show that…
We define the Simons-Sullivan differential analytic index by translating the Freed-Lott differential analytic index via explicit ring isomorphisms between Freed-Lott differential K-theory and Simons-Sullivan differential K-theory. We prove the differential Grothendieck-Riemann-Roch theorem in Simons-Sullivan differenti…
We generalize stochastic smoothing for gradient estimation of non-differentiable functions.
problem Gradient estimation for non-differentiable functions.
method Developed a general framework for relaxation and gradient estimation of non-differentiable black-box functions using stochastic smoothing with reduced assumptions.
result Empirically validated the effectiveness of variance reduction strategies for various non-differentiable tasks.
The paper proves Gorenstein contractions for multiscale differentials on nodal curves.
problem Proving Gorenstein contractions for multiscale differentials on nodal curves.
method Addressing the conjecture by Ranganathan and Wise, showing contractions level by level.
result Multiscale differentials can be contracted to Gorenstein singularities, level by level, from the top down.
Classifies components of strata of k-differentials on Riemann surfaces.
problem Classifying connected components of strata of k-differentials.
method Developed new techniques to study connected components of strata of k-differentials for general k.
result Complete classification of connected components of the strata of quadratic differentials with arbitrary poles.
Monotonic differentiable sorting networks improve upon previous methods.
problem Non-monotonicity in differentiable sorting networks.
method Relaxation of conditional swap operations using sigmoid functions to ensure monotonicity.
result Monotonic differentiable sorting networks improve upon previous methods.
Lecture notes introduce differential geometry using sheaves and differential operators.
problem Exploring differential geometry concepts.
method Using sheaves, differential operators, and horizontal subbundles.
result Presented an approach to fundamental differential geometry structures.
Given a unital associatve graded algebra we construct the graded q-differential algebra by means of a graded q-commutator, where q is a primitive N-th root of unity. The N-th power (N>1) of the differential of this graded q-differential algebra is equal to zero. We use our approach to construct the graded q-differentia…
Paper defines quasi-Strebel structures for meromorphic k-differentials and proves their existence.
problem Existence of quasi-Strebel structures for meromorphic k-differentials.
method Introduced quasi-Strebel structures and proved their existence for meromorphic k-differentials.
result Every differential of even order k > 2 satisfying certain conditions admits a quasi-Strebel structure.
Finite intersection numbers between horizontal foliations of quadratic differentials.
problem Intersection properties of horizontal foliations in quadratic differentials.
method Joint continuity of intersection number in -norm.
result Intersection number is finite and jointly continuous.
We give chain homotopy maps of Khovanov-type link homology of a universal differential. The universal differential, discussed by Mikhail Khovanov, Marco Mackaay, Paul Turner and Pedro Vaz, contains the original Khovanov's differential and Lee's differential. We also consider the conditions of any differential ensuring …
We consider differentiable maps in the setting of Abstract Differential Geometry and we study the conditions that ensure the uniqueness of differentials in this setting. In particular, we prove that smooth maps between smooth manifolds admit a unique differential, coinciding with the usual one. Thus smooth manifolds fo…
Paper solves a class of differential equations with specific solutions.
problem Identifying solutions to a class of nonlinear ODEs.
method Solves using a proposed side condition involving a third-order linear ODE.
result New closed and integral-form solutions for the Tzitzeica curve equation.
We develop differential algebraic K-theory for rings of integers in number fields and we construct a cycle map from geometrized bundles of modules over such a ring to the differential algebraic K-theory. We also treat some of the foundational aspects of differential cohomology, including differential function spectra a…
By adapting the Cheeger-Simons approach to differential cohomology, we establish a notion of differential cohomology with compact support. We show that it is functorial with respect to open embeddings and that it fits into a natural diagram of exact sequences which compare it to compactly supported singular cohomology …
We study two notions of relative differential cohomology, using the model of differential characters. The two notions arise from the two options to construct relative homology, either by cycles of a quotient complex or of a mapping cone complex. We discuss the relation of the two notions of relative differential cohomo…
In this paper we discuss the question of integrating differential graded Lie algebras (DGLA) to differential graded Lie groups (DGLG). We first recall the classical problem of integration in the context, and present the construction for (non-graded) differential Lie algebras. Then, we define the category of differentia…
New privacy framework tailored to specific data distributions.
problem Protecting individual data points in decision-making processes.
method Introducing tangent differential privacy, a new form of differential privacy.
result Entropic regularization guarantees tangent differential privacy under general conditions.
Studies projective geometry and partial differential equations prolongation.
problem Understanding the prolongation of overdetermined geometric partial differential equations.
method Introduction to differential geometry and tractor calculus, study of prolongation of equations.
result Recovery of projective tractor and cotractor connections via partial differential equations prolongation.
Develops differential K-theory for noncommutative algebras.
problem Creating a differential extension of algebraic K-theory for noncommutative algebras.
method Introduces secondary transgression forms and a differential refinement of the smooth Serre--Swan correspondence.
result Subsumes differential K-theory for smooth manifolds and fits into a noncommutative differential cohomology hexagon diagram.
Extends differential geometry concepts to manifolds with super tangent bundles.
problem No specific problem stated; extending differential geometry to super tangent bundles.
method Introduces super tangent bundle and extends differential geometry concepts.
result Basic notions of differential geometry extended to manifolds with super tangent bundles.
Generalized differential cohomology theories, in particular differential K-theory (often called "smooth K-theory"), are becoming an important tool in differential geometry and in mathematical physics. In this survey, we describe the developments of the recent decades in this area. In particular, we discuss axiomatic ch…
Differentiable pipeline replaces non-differentiable CAE components for shape optimization.
problem Gradient-based optimization is limited by non-differentiable components in CAE workflows.
method Surrogate models replace non-differentiable pipeline components, enabling gradient-based optimization.
result Gradient-based shape optimization possible without differentiable solvers.
Differentiable programming aids in solving differential equations and their sensitivities.
problem Computing gradients of numerical solutions of differential equations.
method Review of existing techniques and mathematical foundations.
result Established a coherent framework for combining differential equations with data-driven approaches.
The purpose of this paper is to connect two subjects: the theory of quantum integrable systems (complete commutative rings of differential operators), and differential Galois theory. We define quantum completely integrable systems (QCIS), algebraically integrable QCIS, the differential Galois group of a QCIS. We show t…