arXiv research

A new algorithm optimizes softmax units in large language models.

In this paper, we formalise order-robust optimisation as an instance of online learning minimising simple regret, and propose Vroom, a zero'th order optimisation algorithm capable of achieving vanishing regret in non-stationary environments, while recovering favorable rates under stochastic reward-generating processes.…

We consider the minimization of submodular functions subject to ordering constraints. We show that this optimization problem can be cast as a convex optimization problem on a space of uni-dimensional measures, with ordering constraints corresponding to first-order stochastic dominance. We propose new discretization sch…

We define a set of "second-order" L^(2)-signature invariants for any algebraically slice knot. These obstruct a knot's being a slice knot and generalize Casson-Gordon invariants, which we consider to be "first-order signatures". As one application we prove: If K is a genus one slice knot then, on any genus one Seifert …

Paper proposes efficient algorithms for designing SLOPE penalty sequences.

New algorithms minimize noisy, irregular functions without gradients.

Classical topological concepts are applied to understand high performance computing simulations of molecules writhing in three dimensional space. These simulations produce peta-bytes of floating point data, to describe 3 dimensional changes in molecular structure. A zero-th order analysis is achieved by viewing a compu…

Study complex structures with perturbed differential operators to compute curvature-like operators and obtain vanishing results.

The paper solves a general case of the cohomological relative index problem for foliations.

We consider $k$-dimensional random simplicial complexes that are generated from the binomial random $(k+1)$-uniform hypergraph by taking the downward-closure, where $k\geq 2$. For each $1\leq j \leq k-1$, we determine when all cohomology groups with coefficients in $\mathbb{F}_2$ from dimension one up to $j$ vanish and…

We define for arbitrary modules over a finite von Neumann algebra $\cala$ a dimension taking values in $[0,\infty]$ which extends the classical notion of von Neumann dimension for finitely generated projective $\cala$-modules and inherits all its useful properties such as additivity, cofinality and continuity. This all…

We prove the existence of positive lower bounds on the Cheeger constants of manifolds of the form $X/Γ$ where $X$ is a contractible Riemannian manifold and $Γ<\Isom(X)$ is a discrete subgroup, typically with infinite co-volume. The existence depends on the $L^2$-Betti numbers of $Γ$, its subgroups and of a uniform latt…

Negative step sizes improve second-order methods for neural networks.

First-order stochastic methods are the state-of-the-art in large-scale machine learning optimization owing to efficient per-iteration complexity. Second-order methods, while able to provide faster convergence, have been much less explored due to the high cost of computing the second-order information. In this paper we …

A new method solves complex constrained minimax problems.

Second-order methods improve differential privacy in convex optimization.

Exact second-order optimization for deep learning reduces computational cost and improves performance.

A new method predicts higher-order interactions in evolving graphs using simplicial complexes.

Review and compare model order reduction methods for process engineering.

Bayesian method learns causal orderings from heterogeneous data.

New methods using natural gradient for structured optimization.

New method explains predictive uncertainty by focusing on second-order effects.

New method for pricing options in stochastic volatility models.

We present a new Markov chain Monte Carlo method for estimating posterior probabilities of structural features in Bayesian networks. The method draws samples from the posterior distribution of partial orders on the nodes; for each sampled partial order, the conditional probabilities of interest are computed exactly. We…

Paper proposes a deep learning method to estimate fill probabilities of limit orders in LOBs.

We present a novel factor analysis method that can be applied to the discovery of common factors shared among trajectories in multivariate time series data. These factors satisfy a precedence-ordering property: certain factors are recruited only after some other factors are activated. Precedence-ordering arise in appli…

Zeroth-order optimization methods lack inherent privacy guarantees.

In this paper, we introduce the notion of motif closure and describe higher-order ranking and link prediction methods based on the notion of closing higher-order network motifs. The methods are fast and efficient for real-time ranking and link prediction-based applications such as web search, online advertising, and re…

Predicts node sequences in graphs using multi-order network models.

We provide improved convergence rates for various \emph{non-smooth} optimization problems via higher-order accelerated methods. In the case of $\ell_\infty$ regression, we achieves an $O(ε^{-4/5})$ iteration complexity, breaking the $O(ε^{-1})$ barrier so far present for previous methods. We arrive at a similar rate fo…

SOLBP extends efficient inference to uncertain Bayesian networks.

Improved algorithms for convex-concave min-max optimization and monotone variational inequalities.

A new hybrid-ordered SGD method reduces communication and complexity for non-convex optimization.

We develop high-order approximations for the Heston model.

AdamQLR optimizes Adam with K-FAC heuristics, achieving comparable performance to tuned benchmarks.

Optimal first-order methods are shown to be fundamental limits in functional estimation.

The second order method as Newton Step is a suitable technique in Online Learning to guarantee regret bound. The large data is a challenge in Newton method to store second order matrices as hessian. In this paper, we have proposed an modified online Newton step that store first and second order matrices of dimension m …

Lower bounds for higher-order methods in non-convex optimization.

Finite-sum optimization problems are ubiquitous in machine learning, and are commonly solved using first-order methods which rely on gradient computations. Recently, there has been growing interest in \emph{second-order} methods, which rely on both gradients and Hessians. In principle, second-order methods can require …

A new method for faster optimization in high dimensions.

New method detects left-orderable surgeries on knot 6_2.

Bayesian method detects Markov order in network paths more reliably.

Clustering is fundamental for gaining insights from complex networks, and spectral clustering (SC) is a popular approach. Conventional SC focuses on second-order structures (e.g., edges connecting two nodes) without direct consideration of higher-order structures (e.g., triangles and cliques). This has motivated SC ext…

In this paper, we provide an overview of first-order and second-order variants of the gradient descent method that are commonly used in machine learning. We propose a general framework in which 6 of these variants can be interpreted as different instances of the same approach. They are the vanilla gradient descent, the…

Paper proposes a new method for efficient second-order neural network training.

Representation learning on networks offers a powerful alternative to the oft painstaking process of manual feature engineering, and as a result, has enjoyed considerable success in recent years. However, all the existing representation learning methods are based on the first-order network (FON), that is, the network th…

State-of-the-art methods in convex and non-convex optimization employ higher-order derivative information, either implicitly or explicitly. We explore the limitations of higher-order optimization and prove that even for convex optimization, a polynomial dependence on the approximation guarantee and higher-order smoothn…

Paper develops a TR-SSQP method for noisy optimization with heavy-tailed noise.