arXiv research

Efficiently differentiate functions of large matrices using new adjoint systems.

A new method tackles bilevel optimization using Lanczos process for efficient hyper-gradient computation.

New recommendations improve Gaussian process accuracy and stability.

For applications as varied as Bayesian neural networks, determinantal point processes, elliptical graphical models, and kernel learning for Gaussian processes (GPs), one must compute a log determinant of an $n \times n$ positive definite matrix, and its derivatives - leading to prohibitive $\mathcal{O}(n^3)$ computatio…

One of the most compelling features of Gaussian process (GP) regression is its ability to provide well-calibrated posterior distributions. Recent advances in inducing point methods have sped up GP marginal likelihood and posterior mean computations, leaving posterior covariance estimation and sampling as the remaining …

The paper analyzes contraction rates for GP regression approximations.

Botnet, a group of coordinated bots, is becoming the main platform of malicious Internet activities like DDOS, click fraud, web scraping, spam/rumor distribution, etc. This paper focuses on design and experiment of a new approach for botnet detection from streaming web server logs, motivated by its wide applicability, …

We propose the Lanczos network (LanczosNet), which uses the Lanczos algorithm to construct low rank approximations of the graph Laplacian for graph convolution. Relying on the tridiagonal decomposition of the Lanczos algorithm, we not only efficiently exploit multi-scale information via fast approximated computation of…

Evaluating the log determinant of a positive definite matrix is ubiquitous in machine learning. Applications thereof range from Gaussian processes, minimum-volume ellipsoids, metric learning, kernel learning, Bayesian neural networks, Determinental Point Processes, Markov random fields to partition functions of discret…

The study examines algebraic structures of specific tensor forms in four-dimensional spacetimes.

Numerous methods for computing conformal mesh paramterizations has been developed due to the vast applications in the field of geometry processing. Spectral conformal parameterization (SCP) is one of these methods to computing a quality conformal parameterization based on the spectral technique. SCP focus on a generali…

The runtime for Kernel Partial Least Squares (KPLS) to compute the fit is quadratic in the number of examples. However, the necessity of obtaining sensitivity measures as degrees of freedom for model selection or confidence intervals for more detailed analysis requires cubic runtime, and thus constitutes a computationa…

We present a general framework for classification of sparse and irregularly-sampled time series. The properties of such time series can result in substantial uncertainty about the values of the underlying temporal processes, while making the data difficult to deal with using standard classification methods that assume …

The purpose if this master's thesis is to study and develop a new algorithmic framework for Collaborative Filtering to produce recommendations in the top-N recommendation problem. Thus, we propose Lanczos Latent Factor Recommender (LLFR); a novel "big data friendly" collaborative filtering algorithm for top-N recommend…

New Krylov subspace methods speed up mixed-effects models with crossed random effects.

We develop and analyze efficient "coordinate-wise" methods for finding the leading eigenvector, where each step involves only a vector-vector product. We establish global convergence with overall runtime guarantees that are at least as good as Lanczos's method and dominate it for slowly decaying spectrum. Our methods a…

In all dimensions and arbitrary signature, we demonstrate the existence of a new local potential -- a double (2,3)-form -- for the Weyl curvature tensor, and more generally for all tensors with the symmetry properties of the Weyl curvature tensor. The classical four-dimensional Lanczos potential for a Weyl tensor -- a …

We exploit four-dimensional tensor identities to give a very simple proof of the existence of a Lanczos potential for a Weyl tensor in four dimensions with any signature, and to show that the potential satisfies a simple linear second order differential equation, e.g., a wave equation in Lorentz signature. Furthermore,…

We analyze the Hessian spectra of large models up to 100B parameters.

The $(2k)$-th Gauss-Bonnet curvature is a generalization to higher dimensions of the $(2k)$-dimensional Gauss-Bonnet integrand, it coincides with the usual scalar curvature for $k=1$. The Gauss-Bonnet curvatures are used in theoretical physics to describe gravity in higher dimensional space times where they are known a…

We found in 2016 a few results on the mathematical structure of the conformal Killing differential sequence in arbitrary dimension $n$, in particular the rank and order changes of the successive differential operators for $n=3,n=4$ or $n\geq 5$. They were so striking that we did not dare to publish them before our form…

This paper proposes a novel profile likelihood method for estimating the covariance parameters in exploratory factor analysis of high-dimensional Gaussian datasets with fewer observations than number of variables. An implicitly restarted Lanczos algorithm and a limited-memory quasi-Newton method are implemented to deve…

Principal component analysis (PCA) is one of the most powerful tools in machine learning. The simplest method for PCA, the power iteration, requires $\mathcal O(1/Δ)$ full-data passes to recover the principal component of a matrix with eigen-gap $Δ$. Lanczos, a significantly more complex method, achieves an accelerated…

We present MLRG Deep Curvature suite, a PyTorch-based, open-source package for analysis and visualisation of neural network curvature and loss landscape. Despite of providing rich information into properties of neural network and useful for a various designed tasks, curvature information is still not made sufficient us…

Match van Stockum dust to vacuum metrics with a single parameter.

The graph Laplacian is a standard tool in data science, machine learning, and image processing. The corresponding matrix inherits the complex structure of the underlying network and is in certain applications densely populated. This makes computations, in particular matrix-vector products, with the graph Laplacian a ha…

We show that a simple randomized sketch of the matrix multiplicative weight (MMW) update enjoys (in expectation) the same regret bounds as MMW, up to a small constant factor. Unlike MMW, where every step requires full matrix exponentiation, our steps require only a single product of the form $e^A b$, which the Lanczos …

We propose a new hybrid algorithm that allows incorporating both user and item side information within the standard collaborative filtering technique. One of its key features is that it naturally extends a simple PureSVD approach and inherits its unique advantages, such as highly efficient Lanczos-based optimization pr…

Study critical metrics on Riemannian manifolds, finding new minimizers and rigidity results.

Correlated anomaly detection (CAD) from streaming data is a type of group anomaly detection and an essential task in useful real-time data mining applications like botnet detection, financial event detection, industrial process monitor, etc. The primary approach for this type of detection in previous researches is base…

HessFormer enables distributed Hessian computation for large models.

The main purpose of this paper is to revisit the well known potentials, called stress functions, needed in order to study the parametrizations of the stress equations, respectively provided by G.B. Airy (1863) for 2-dimensional elasticity, then by E. Beltrami (1892), J.C. Maxwell (1870) and G. Morera (1892) for 3-dimen…

We introduce a weighted de Rham operator which acts on arbitrary tensor fields by considering their structure as r-fold forms. We can thereby define associated superpotentials for all tensor fields in all dimensions and, from any of these superpotentials, we deduce in a straightforward and natural manner the existence …

Unified framework detects overfitting in crash classification models.

Semidefinite programming (SDP) with diagonal constraints arise in many optimization problems, such as Max-Cut, community detection and group synchronization. Although SDPs can be solved to arbitrary precision in polynomial time, generic convex solvers do not scale well with the dimension of the problem. In order to add…

We study the algorithmic problem of estimating the mean of heavy-tailed random vector in $\mathbb{R}^d$, given $n$ i.i.d. samples. The goal is to design an efficient estimator that attains the optimal sub-gaussian error bound, only assuming that the random vector has bounded mean and covariance. Polynomial-time solutio…

The purpose of this paper is to revisit the Bianchi identities existing for the Riemann and Weyl tensors in the combined framework of the formal theory of systems of partial differential equations (Spencer cohomology, differential systems, formal integrability) and Algebraic Analysis (homological algebra, differential …

Efficient approximations reduce computation of matrix-based Renyi's entropy.

New insights into learning rates and batch sizes for neural networks using random matrix theory.

This paper deals with finding an $n$-dimensional solution $x$ to a system of quadratic equations of the form $y_i=|\langle{a}_i,x\rangle|^2$ for $1\le i \le m$, which is also known as phase retrieval and is NP-hard in general. We put forth a novel procedure for minimizing the amplitude-based least-squares empirical los…

This chapter is an attempt to present a mathematical theory of compound fractional Poisson processes. The chapter begins with the characterization of a well-known Lévy process: The compound Poisson process. The semi-Markov extension of the compound Poisson process naturally leads to the compound fractional Poisson proc…

We introduce a class of hybrid marked point processes, which encompasses and extends continuous-time Markov chains and Hawkes processes. While this flexible class amalgamates such existing processes, it also contains novel processes with complex dynamics. These processes are defined implicitly via their intensity and a…

A deep Neyman-Scott process uses Poisson processes for efficient inference in complex point processes.

The study examines Hawkes processes and their long-term behavior.

Elliptical processes generalize Gaussian and Student-t models with fat tails and computational efficiency.

We show that the stick-breaking construction of the beta process due to Paisley, et al. (2010) can be obtained from the characterization of the beta process as a Poisson process. Specifically, we show that the mean measure of the underlying Poisson process is equal to that of the beta process. We use this underlying re…

We investigate the Student-t process as an alternative to the Gaussian process as a nonparametric prior over functions. We derive closed form expressions for the marginal likelihood and predictive distribution of a Student-t process, by integrating away an inverse Wishart process prior over the covariance kernel of a G…

Efficient methods for Lévy models using SINH-regular processes.