New geometric proof of convex function differentiability and approximation.
problem Second-order differentiability of convex functions and their approximations.
method Elementary geometric approach to prove classical and recent results.
result New proofs of Lusin approximation of convex functions and bodies by C1,1 functions. We develop a second-order model for limit order books in a single scaling regime.
problem Modeling price and volume dynamics in a limit order book with market and limit orders at a common time scale.
method Established a first- and second-order approximation for an infinite dimensional limit order book model.
result Proved the existence and uniqueness of a solution for the second-order approximation.
Paper introduces STSL, a second-order Tweedie sampler for efficient posterior sampling in inverse problems.
problem Computational challenges in sampling from posterior distributions using latent diffusion models.
method Introduces STSL, a novel second-order Tweedie sampler with tractable reverse process.
result STSL achieves 4X and 8X reduction in neural function evaluations compared to state-of-the-art solvers.
In this paper we derive a second order approximation for an infinite dimensional limit order book model, in which the dynamics of the incoming order flow is allowed to depend on the current market price as well as on a volume indicator (e.g.~the volume standing at the top of the book). We study the fluctuations of the …
The paper debiases mini-batch approximations in deep learning for more accurate optimization and uncertainty quantification.
problem Bias in mini-batch approximations distorts the shape of quadratic approximations used in deep learning.
method Developed and evaluated debiasing strategies for mini-batch approximations.
result Debiasing strategies improve the accuracy of second-order optimization and uncertainty quantification in deep learning.
New algorithm finds approximate stationary points in non-convex optimization.
problem Finding approximate stationary points in non-convex stochastic optimization.
method Design of an algorithm using O(ε−3) stochastic gradient and Hessian-vector products. result Optimal rate of O(ε−3) for finding ε-approximate stationary points, matching lower bounds. A new method for optimizing deep neural networks using TKFAC.
problem Optimizing deep neural networks with second-order methods.
method Proposes Trace-restricted Kronecker-factored Approximate Curvature (TKFAC) for Fisher information matrix approximation.
result TKFAC improves performance on deep network architectures compared to state-of-the-art algorithms.
Paper examines risk measure expansions under FGM dependence, improving accuracy at extreme levels.
problem Capturing higher-order tail behavior and dependence effects in risk measures.
method Second-order asymptotic expansions using extreme value theory and regular variation theory.
result Second-order approximations reduce approximation errors, especially at extreme confidence levels.
Second-order economic theory considers new variables to improve price volatility predictions.
problem Current economic models focus on first-order variables, missing second-order variables that affect price volatility.
method Introduces second-order economic theory with new variables composed of sums of squares of agents' transactions.
result Second-order economic theory complements first-order variables and introduces new macroeconomic variables.
A new method improves ICA performance by approximating MDI.
problem Improving F astICA's performance with nonlinear functions.
method Second-order approximation of MDI for joint maximization.
result Efficiency validated through experiments compared to other ICA algorithms.
Natural gradient descent, which preconditions a gradient descent update with the Fisher information matrix of the underlying statistical model, is a way to capture partial second-order information. Several highly visible works have advocated an approximation known as the empirical Fisher, drawing connections between ap…
Advanced optimization algorithms such as Newton method and AdaGrad benefit from second order derivative or second order statistics to achieve better descent directions and faster convergence rates. At their heart, such algorithms need to compute the inverse or inverse square root of a matrix whose size is quadratic of …
Enhances SMC² with Hessian info for more efficient posterior approximation.
problem Improving accuracy and efficiency in Bayesian inference.
method Integrates second-order information (Hessian) into SMC²'s proposal distribution.
result Second-order proposals lead to more accurate posterior approximations and better step-size selection.
SOAR improves deep networks' robustness against adversarial examples.
problem Improving deep neural networks' robustness against adversarial examples.
method Formulated adversarial robustness problem under robust optimization framework, approximated loss function using second-order Taylor series expansion.
result SOAR significantly improves robustness of networks against adversarial perturbations.
The quantification of diversification benefits due to risk aggregation plays a prominent role in the (regulatory) capital management of large firms within the financial industry. However, the complexity of today's risk landscape makes a quantifiable reduction of risk concentration a challenging task. In the present pap…
Improved robustness in optimization methods using second-order information.
problem Scalability and sensitivity to mini-batch size in optimization methods.
method Mini-Batch Stochastic Variance-Reduced Newton (extttMb−SVRN) algorithm incorporating partial second-order information. result Achieves a fast linear convergence rate independent of mini-batch size for large data sizes.
Efficient method classifies locally stationary time series based on second-order characteristics.
problem Classifying locally stationary time series for various applications.
method Autoregressive approximation, ensemble aggregation, distance-based threshold.
result Zero misclassification error rate asymptotically for mildly differing second-order characteristics.
We study a cutting-plane method for semidefinite optimization problems (SDOs), and supply a proof of the method's convergence, under a boundedness assumption. By relating the method's rate of convergence to an initial outer approximation's diameter, we argue that the method performs well when initialized with a second-…
Paper proposes a new method for efficient second-order neural network training.
problem Infeasibility of Hessian calculation and noisy second-order information in deep learning.
method Adopting complex-step directional derivative (CSFD) for accurate Hessian computation and designing an effective Newton Krylov procedure.
result Our method outperforms existing methods and often converges one-order faster.
Paper introduces arctan pinball loss for XGBoost quantile regression.
problem Efficiently predicting multiple quantiles with XGBoost.
method Smooth approximation of pinball loss for XGBoost, using arctan pinball loss.
result Arctan pinball loss reduces quantile crossings and improves efficiency.
Optimizes hyperparameter tuning for models using approximate leave-one-out cross-validation.
problem Finding optimal hyperparameters for regularized models using approximate leave-one-out cross-validation.
method Derive efficient formulas for gradient and hessian of approximate leave-one-out cross-validation, apply second-order optimization.
result Demonstrates the effectiveness of the approach on real-world data sets.
Multiscale stochastic volatility models have been developed as an efficient way to capture the principle effects on derivative pricing and portfolio optimization of randomly varying volatility. The recent book Fouque, Papanicolaou, Sircar and Sølna (2011, CUP) analyzes models in which the volatility of the underlying i…
Two new algorithms improve federated optimization under second-order similarity.
problem Federated learning under communication constraints and second-order similarity.
method SVRP and Catalyzed SVRP algorithms combining proximal point evaluations, client sampling, and variance reduction.
result Achieves superior performance and uniformly improves upon existing algorithms for federated optimization under second-order similarity and strong convexity.
Paper proposes a method to find approximate SOSP for nonconvex conic optimization problems.
problem Finding approximate second-order stationary points in nonconvex conic optimization.
method Newton-CG based barrier method with complexity guarantees.
result Achieves iteration complexity of O(ε^(-3/2)) for finding (ε,√ε)-SOSP.
Proposes a new method for optimizing large-scale models using Nyström approximation of the Hessian.
problem Optimizing non-convex functions like deep learning models using second-order methods.
method Nyström-approximated curvature for stochastic optimization of large-scale empirical risk minimization.
result The proposed method achieves performance competitive with state-of-the-art first-order and stochastic quasi-Newton methods.
Simplifies convolutions using tensor networks and einsum for efficient second-order methods.
problem Complexity in analyzing and applying convolutions in deep learning.
method Viewing convolutions as tensor networks, drawing diagrams, and using einsum for efficient computation.
result Accelerates a KFAC variant up to 4.5x with reduced memory overhead.
PDHAMS improves sampling for discrete distributions with quadratic potential functions.
problem Sampling discrete distributions efficiently and accurately.
method Integrates a second-order approximation of the potential function and uses Gaussian integral trick.
result PDHAMS yields superior performance compared to other methods.
Paper proposes a second-order method for faster SVI convergence.
problem Poor convergence rate of first-order SVI algorithms.
method Derives Hessian matrix and implements two numerical schemes for efficient second-order SVI.
result Proposed approach achieves faster convergence compared to first-order SVI.
Stein variational gradient descent (SVGD) was recently proposed as a general purpose nonparametric variational inference algorithm [Liu & Wang, NIPS 2016]: it minimizes the Kullback-Leibler divergence between the target distribution and its approximation by implementing a form of functional gradient descent on a reprod…
New pruning method captures global correlations for efficient neural network inference.
problem Efficiently pruning neural networks for faster inference and reduced memory usage.
method Second-order structured pruning (SOSP-H) with innovative saliency-based approaches.
result SOSP-H scales to large-scale vision tasks and improves accuracy without compromising efficiency.
With the rapid adoption of machine learning systems in sensitive applications, there is an increasing need to make black-box models explainable. Often we want to identify an influential group of training samples in a particular test prediction for a given machine learning model. Existing influence functions tackle this…
ISAAC Newton uses input-based curvature for efficient training.
problem Efficient training in small-batch stochastic regimes.
method ISAAC Newton conditions gradients using selected second-order information based on input.
result Effective training even in small-batch stochastic regimes, competitive to first-order and second-order methods.
We study the sensitivity of the expected utility maximization problem in a continuous semi-martingale market with respect to small changes in the market price of risk. Assuming that the preferences of a rational economic agent are modeled with a general utility function, we obtain a second-order expansion of the value …
Second-order optimization speeds up deep hedging for complex options.
problem Hedging exotic options with market frictions in realistic markets.
method Second-order optimization scheme leveraging pathwise differentiability and Kronecker-factoring.
result Our method optimizes the policy in 1/4 the steps of standard optimization.
Develops first and second-order pseudo-mirror descent methods for nonnegative function estimation.
problem Nonnegative function estimation in settings like MLE and trajectory optimization.
method First and second-order pseudo-mirror descent with pseudo-gradients and projections.
result Establishes tradeoffs and non-asymptotic bounds on model complexity.
D2SRM solves complex PDEs using deep learning.
problem High-dimensional, Hessian-dependent fully nonlinear parabolic PDEs.
method Single scalar space-time network generating derivative-consistent approximations trained through residuals and penalties.
result Well-posedness and convergence theory established for globally Lipschitz equations.
High-dimensional partial differential equations (PDE) appear in a number of models from the financial industry, such as in derivative pricing models, credit valuation adjustment (CVA) models, or portfolio optimization models. The PDEs in such applications are high-dimensional as the dimension corresponds to the number …
We study local and global approximations of smooth nets of curvature lines and smooth conjugate nets by respective discrete nets (circular nets and planar quadrilateral nets) with infinitesimal quads. It is shown that choosing the points of discrete nets on the smooth surface one can obtain second-order approximation g…
New method solves stochastic optimization problems with random models.
problem Optimizing stochastic objectives with deterministic constraints.
method Trust-Region Sequential Quadratic Programming with random model.
result Global convergence guarantees for first- and second-order stationary points.
Simple DP algorithms find approximate solutions for nonconvex ERM.
problem Finding approximate solutions to nonconvex ERM problems with privacy.
method Differential privacy, descent directions, line search, mini-batching, two-phase strategy.
result Effective algorithms for nonconvex ERM with privacy guarantees.
Novel IMEX scheme solves financial PDEs with mixed derivatives.
problem Numerical approximations for financial PDEs with mixed derivatives.
method Second order finite volume IMEX Runge-Kutta scheme.
result Achieves true second order convergence with non-regular initial conditions.
Zeroth-order optimization is an important research topic in machine learning. In recent years, it has become a key tool in black-box adversarial attack to neural network based image classifiers. However, existing zeroth-order optimization algorithms rarely extract second-order information of the model function. In this…
COMRADE is a communication-efficient, Byzantine-resilient second-order optimization algorithm.
problem Byzantine failures in distributed optimization.
method COMRADE is a communication-efficient, second-order optimization algorithm that uses a simple norm-based thresholding rule to filter out Byzantine workers.
result COMRADE achieves linear-quadratic convergence and is robust against Byzantine workers.
We set the context for capital approximation within the framework of the Basel II / III regulatory capital accords. This is particularly topical as the Basel III accord is shortly due to take effect. In this regard, we provide a summary of the role of capital adequacy in the new accord, highlighting along the way the s…
New elastic energy for irregular curves defined through polygonal approximations.
problem Defining elastic energy for irregular curves in any space dimension.
method Relaxation process with p-rotation of inscribed polygonals, focusing on geometric curvature distribution. result Energy finite if and only if curve's arc-length parameterization has second order summability.
In the present work, a novel second-order approximation for ATM option prices is derived for a large class of exponential Lévy models with or without Brownian component. The results hereafter shed new light on the connection between both the volatility of the continuous component and the jump parameters and the behavio…
We propose a new, unified approach to solving jump-diffusion partial integro-differential equations (PIDEs) that often appear in mathematical finance. Our method consists of the following steps. First, a second-order operator splitting on financial processes (diffusion and jumps) is applied to these PIDEs. To solve the…
We consider the case of derivative-free algorithms for non-convex optimization, also known as zero order algorithms, that use only function evaluations rather than gradients. For a wide variety of gradient approximators based on finite differences, we establish asymptotic convergence to second order stationary points u…