arXiv research

New algorithm eliminates sign function in PGD attacks, improving performance.

Sign-based optimization methods have become popular in machine learning due to their favorable communication cost in distributed optimization and their surprisingly good performance in neural network training. Furthermore, they are closely connected to so-called adaptive gradient methods like Adam. Recent works on sign…

Unified sign-based compression for federated learning with faster convergence.

Paper solves k-sparse parity problem with sign SGD, matching SQ lower bound.

Efficiently performs robust and sparse kernel regression.

In this paper, we propose a new first-order gradient-based algorithm to train deep neural networks. We first introduce the sign operation of stochastic gradients (as in sign-based methods, e.g., SIGN-SGD) into ADAM, which is called as signADAM. Moreover, in order to make the rate of fitting each feature closer, we defi…

Estimates graph curvature and diameter using Laplacian eigenvalues.

GradPower speeds up language model training with minimal code changes.

The ADAM optimizer is exceedingly popular in the deep learning community. Often it works very well, sometimes it doesn't. Why? We interpret ADAM as a combination of two aspects: for each weight, the update direction is determined by the sign of stochastic gradients, whereas the update magnitude is determined by an esti…

Two algorithms improve federated learning efficiency and resilience.

We study the geometry at infinity of expanding gradient Ricci solitons of dimension greater than two with finite asymptotic curvature ratio without curvature sign assumptions. We mainly prove that they have a cone structure at infinity.

Fast Gradient Sign Method (FGSM) is a popular method to generate adversarial examples that make neural network models robust against perturbations. Despite its empirical success, its theoretical property is not well understood. This paper develops theory to explain the regularization effect of Generalized FGSM, a class…

This paper focuses on the problem of prescribing mean curvature on the unit ball. Assume that $f$, which is allowed to change sign, satisfies Morse index counting or certain kind of symmetry condition. By using a negative gradient flow method, we then prove that $f$ can be realized as the boundary mean curvature of som…

Novel neural network approximates exact distance for robust classification.

We study the most practical problem setup for evaluating adversarial robustness of a machine learning system with limited access: the hard-label black-box attack setting for generating adversarial examples, where limited model queries are allowed and only the decision is provided to a queried data input. Several algori…

In this paper, by slightly modifying Li-Yau's technique so that we can handle drifting Laplacians, we were able to find three different gradient estimates for the warping function, one for each sign of the Einstein constant of the fiber manifold. As an application, we exhibit a nonexistence theorem for gradient almost …

Adam outperforms gradient descent on language models due to handling heavy-tailed class imbalance.

Sign-based algorithms (e.g. signSGD) have been proposed as a biased gradient compression technique to alleviate the communication bottleneck in training large neural networks across multiple workers. We show simple convex counter-examples where signSGD does not converge to the optimum. Further, even when it does conver…

New analysis of signSGD with random reshuffling shows faster convergence rates.

Federated learning (FL) has emerged as a prominent distributed learning paradigm. FL entails some pressing needs for developing novel parameter estimation approaches with theoretical guarantees of convergence, which are also communication efficient, differentially private and Byzantine resilient in the heterogeneous da…

The main purpose of this short note is to point out that the negative gradient flow for the prescribed $\mathbf Q$-curvature problem on $S^n$ can be extended to handle the case that the $\mathbf Q$-curvature candidate $f$ may change signs.

The paper proves inequalities and growth rates for Schouten solitons.

Learning rates in stochastic neural network training are currently determined a priori to training, using expensive manual or automated iterative tuning. This study proposes gradient-only line searches to resolve the learning rate for neural network training algorithms. Stochastic sub-sampling during training decreases…

State-of-the-art adversarial attacks are aimed at neural network classifiers. By default, neural networks use gradient descent to minimize their loss function. The gradient of a classifier's loss function is used by gradient-based adversarial attacks to generate adversarially perturbed images. We pose the question whet…

A new metric learning framework for signed graphs using Gershgorin disc alignment.

Interesting theoretical associations have been established by recent papers between the fields of active learning and stochastic convex optimization due to the common role of feedback in sequential querying mechanisms. In this paper, we continue this thread in two parts by exploiting these relations for the first time …

In this paper, we look for properties of gradient Yamabe solitons on top of warped product manifolds. Utilizing the maximum principle, we find lower bound estimates for both the potential function of the soliton and the scalar curvature of the warped product. By slightly modifying Li-Yau's technique so that we can hand…

This paper resolves BIHT convergence, showing normalization is not necessary in noiseless settings but crucial for robustness.

Muon outperforms GD in associative memory learning by balancing frequency components.

A new method reduces adversarial training time without overfitting.

This paper analyzes adversarial attacks methods and their effectiveness.

Gradient flow method solves isoperimetric inequality for maps.

Estimates gradients of solutions on closed surfaces.

FedLion improves Federated Learning by speeding up convergence and reducing communication costs.

New Morse-Bott function defined on Stiefel manifolds, revealing complex critical structures.

Unified framework for distributed compressed SGD under $(L_0, L_1)$-smoothness.

New maximum score estimators using ReLU functions and deep neural networks.

Study on signed graphs with random signs, focusing on community detection.

SELO model predicts link signs better than SDGNN using subgraph encoding and linear optimization.

We argue that the standard graph Laplacian is preferable for spectral partitioning of signed graphs compared to the signed Laplacian. Simple examples demonstrate that partitioning based on signs of components of the leading eigenvectors of the signed Laplacian may be meaningless, in contrast to partitioning based on th…

Paper develops estimates for Lagrangian phase changes in 2D.

Novel GNN for signed and directed networks using magnetic signed Laplacian.

Sign equivariant networks improve model expressiveness for spectral geometric learning.

Defines signed quasiregular curves and proves growth theorem.

Stochastic gradient descent (SGD) is one of the most widely used optimization methods for parallel and distributed processing of large datasets. One of the key limitations of distributed SGD is the need to regularly communicate the gradients between different computation nodes. To reduce this communication bottleneck, …

Signed networks contain both positive and negative kinds of interactions like friendship and enmity. The task of node classification in non-signed graphs has proven to be beneficial in many real world applications, yet extensions to signed networks remain largely unexplored. In this paper we introduce the first analysi…

In this short note, we compare the combinatorial sign assignment of Manolescu, Ozsvath, Szabo and Thurston for grid homology of knots and links in 3-sphere with the sign assignment coming from a coherent system of orientations on Whitney disks. Although these constructions produce different signs, a small modification …

PyTorch Geometric Signed Directed fills the gap for GNNs on signed and directed graphs.