arXiv research

Transformers improve Alzheimer's disease progression prediction by accounting for irregular biomarker histories.

Cloud computing is becoming increasingly popular as a platform for distributed training of deep neural networks. Synchronous stochastic gradient descent (SSGD) suffers from substantial slowdowns due to stragglers if the environment is non-dedicated, as is common in cloud computing. Asynchronous SGD (ASGD) methods are i…

Proposes a neural network method to correct residual distortions in coordinate transformations.

A new model improves CT image quality from low-dose scans.

Two methods are proposed to filter correlations in DCC-GARCH residuals for foreign exchange rates.

Framework calculates positional influence in causal residual Transformers.

Researchers prove Transformers are Turing-complete and analyze their components.

In this paper, we introduce transformations of deep rectifier networks, enabling the conversion of deep rectifier networks into shallow rectifier networks. We subsequently prove that any rectifier net of any depth can be represented by a maximum of a number of functions that can be realized by a shallow network with a …

We prove the homotopy invariance of L^2 torsion for covering spaces, whenever the covering transformation group is either residually finite or amenable. In the case when the covering transformation group is residually finite and when the L^2 cohomology of the covering space vanishes, the homotopy invariance was establi…

Study Transformer layers under cross-entropy training using mean field control.

Flow-based generative models parameterize probability distributions through an invertible transformation and can be trained by maximum likelihood. Invertible residual networks provide a flexible family of transformations where only Lipschitz conditions rather than strict architectural constraints are needed for enforci…

Deep linear ResNets converge globally with certain transformations.

Researchers relax the CVF's smoothness requirement to create more flexible flow models.

The paper introduces a diagnostic method to detect grokking transitions in models before test accuracy improves.

Signal models based on sparsity, low-rank and other properties have been exploited for image reconstruction from limited and corrupted data in medical imaging and other computational imaging applications. In particular, sparsifying transform models have shown promise in various applications, and offer numerous advantag…

Large Transformer models routinely achieve state-of-the-art results on a number of tasks but training these models can be prohibitively costly, especially on long sequences. We introduce two techniques to improve the efficiency of Transformers. For one, we replace dot-product attention by one that uses locality-sensiti…

We introduce a general method of performing Residual Network inference and learning in the JPEG transform domain that allows the network to consume compressed images as input. Our formulation leverages the linearity of the JPEG transform to redefine convolution and batch normalization with a tune-able numerical approxi…

The study analyzes numerical stability in large language models using mixed-precision arithmetic.

In many video coding systems, separable transforms (such as two-dimensional DCT-2) have been used to code block residual signals obtained after prediction. This paper proposes a parametric approach to build graph-based separable transforms (GBSTs) for video coding. Specifically, a GBST is derived from a pair of line gr…

A new model classifies lightning signals more accurately across different scales.

Effective theory for Transformer initialization improves model performance.

Efficient Winograd convolution for INT8 networks using RNS.

Fusion of transformer networks using optimal transport for improved performance.

We study conformal $Spin$-subgeometry of submanifolds in a semi-Riemannian $Spin$-manifold, focusing on conformal $Spin$-manifolds $(M,[h])$ and their Poincaré-Einstein metrics $(X,g_+)$. Our approach is based on the spectral theory of Dirac operator in the ambient $Spin$-manifold, and associated spinor valued meromorp…

Our research proves neural collapse in deep ResNets and transformers is globally optimal.

Vision Transformers show different internal representations compared to CNNs.

Deviance-style normalization for sparse, jointly overdispersed count matrices

Extended LSTM improves language modeling performance.

Layer normalization placement affects training stability and warm-up stage necessity.

A simple gating mechanism improves deep learning convergence.

Study on knot $7_4$ surgeries reveals infinite residue characteristics and infinite order points.

Filters in convolutional networks are typically parameterized in a pixel basis, that does not take prior knowledge about the visual world into account. We investigate the generalized notion of frames designed with image properties in mind, as alternatives to this parametrization. We show that frame-based ResNets and De…

Consider the space $R_Δ$ of rational functions of several variables with poles on a fixed arrangement $Δ$ of hyperplanes. We obtain a decomposition of $R_Δ$ as a module over the ring of differential operators with constant coefficients. We generalize to the space $R_Δ$ the notions of principal part and of residue, and …

Residual networks with depthwise hyperparameter scaling transfer optimal hyperparameters across width and depth.

An emerging design principle in deep learning is that each layer of a deep artificial neural network should be able to easily express the identity transformation. This idea not only motivated various normalization techniques, such as \emph{batch normalization}, but was also key to the immense success of \emph{residual …

Residual networks' depth is mathematically equivalent to expanding an implicit ensemble size.

In this effort, we propose a new deep architecture utilizing residual blocks inspired by implicit discretization schemes. As opposed to the standard feed-forward networks, the outputs of the proposed implicit residual blocks are defined as the fixed points of the appropriately chosen nonlinear transformations. We show …

CAM-GAN improves GANs for continual learning with efficient feature map transformations.

We discuss several aspects of Mellin transform, including distributional Mellin transform and inversion of multiple Mellin-Barnes integrals in $\mathbb{C}^n$ and its connection to residue expansion or evaluation of Laplace integrals. These mathematical concepts are demonstrated on several option-pricing models. This in…

Unified spectral framework for μP under joint width-depth scaling.

The paper establishes a Poisson Poincaré-Dulac theorem for Poisson-flat connections.

CRC improves multivariate forecasting accuracy without risking performance degradation.

New theory explains signal propagation in normalization-free transformers.

We develop a deep autoencoder architecture that can be used to find a coordinate transformation which turns a nonlinear PDE into a linear PDE. Our architecture is motivated by the linearizing transformations provided by the Cole-Hopf transform for Burgers equation and the inverse scattering transform for completely int…

Improved DNN calibration without sacrificing accuracy.

LLapDiff models irregular multivariate time series without step-by-step integration.

Let G be a torsion free discrete group and let \bar{Q} denote the field of algebraic numbers in C. We prove that \bar{Q}[G] fulfills the Atiyah conjecture if G lies in a certain class of groups D, which contains in particular all groups which are residually torsion free elementary amenable or which are residually free.…

ImpFlows generalize normalizing flows by implicitly defining transformations.