arXiv research

ETC improves Transformer models for long and structured inputs.

Transformers handle infinite dimensional inputs effectively by feature extraction and dynamic feature selection.

Centroid Transformers reduce memory and computation by summarizing inputs into centroids.

Transformers can interpolate between arbitrary measures.

Transformer is a powerful architecture that achieves superior performance on various sequence learning tasks, including neural machine translation, language understanding, and sequence prediction. At the core of the Transformer is the attention mechanism, which concurrently processes all inputs in the streams. In this …

Deep Graph Neural Networks (GNNs) are useful models for graph classification and graph-based regression tasks. In these tasks, graph pooling is a critical ingredient by which GNNs adapt to input graphs of varying size and structure. We propose a new graph pooling operation based on compressive Haar transforms -- HaarPo…

Kernel models learn low-dimensional predictive subspaces from input data.

The characteristics (or numerical patterns) of a feature vector in the transform domain of a perturbation model differ significantly from those of its corresponding feature vector in the input domain. These differences - caused by the perturbation techniques used for the transformation of feature patterns - degrade the…

We apply neural nets with ReLU gates in online reinforcement learning. Our goal is to train these networks in an incremental manner, without the computationally expensive experience replay. By studying how individual neural nodes behave in online training, we recognize that the global nature of ReLU gates can cause und…

Transformationally invariant processors constructed by transformed input vectors or operators have been suggested and applied to many applications. In this study, transformationally identical processing based on combining results of all sub-processes with corresponding transformations at one of the processing steps or …

Transformer with denoising diffusion improves probabilistic density estimation.

This paper studies the generalization error of invariant classifiers. In particular, we consider the common scenario where the classification task is invariant to certain transformations of the input, and that the classifier is constructed (or learned) to be invariant to these transformations. Our approach relies on fa…

We find a normal form for two-input flat discrete-time systems.

We introduce a transformer-based GNN model, named UGformer, to learn graph representations. In particular, we present two UGformer variants, wherein the first variant (publicized in September 2019) is to leverage the transformer on a set of sampled neighbors for each input node, while the second (publicized in May 2021…

Mixtures of Gaussians, factor analyzers (probabilistic PCA) and hidden Markov models are staples of static and dynamic data modeling and image and video modeling in particular. We show how topographic transformations in the input, such as translation and shearing in images, can be accounted for in these models by inclu…

Transformers show strengths and weaknesses in complexity analysis.

Transformers learn algorithms for in-context learning with bounds and stability analysis.

Rough Transformers improve efficiency for medical time-series data.

Saliency methods aim to explain the predictions of deep neural networks. These methods lack reliability when the explanation is sensitive to factors that do not contribute to the model prediction. We use a simple and common pre-processing step ---adding a constant shift to the input data--- to show that a transformatio…

Deep neural network (DNN) has demonstrated its success in multiple domains. However, DNN models are inherently vulnerable to adversarial examples, which are generated by adding adversarial perturbations to benign inputs to fool the DNN model to misclassify. In this paper, we present a cross-layer strategic ensemble fra…

Rough Transformers improve time series modeling with lower costs and better performance.

I-BERT extends Transformer's self-attention to arbitrary input lengths.

New Feedback Transformer architecture improves model performance by exposing past representations to future.

Deep linear ResNets converge globally with certain transformations.

We study the first-order scattering transform as a candidate for reducing the signal processed by a convolutional neural network (CNN). We show theoretical and empirical evidence that in the case of natural images and sufficiently small translation invariance, this transform preserves most of the signal information nee…

Transformers can interpolate finite input sequences exactly.

MIMONets speed up neural network inference by processing multiple inputs in parallel.

This project improves model robustness to affine transformations.

This research discovers model architecture and training dataset characteristics through strategic input probing.

Interprets neural network classifiers for categorical inputs.

SE(3)-Transformers maintain equivariance for 3D data under rotations and translations.

Vision transformers benefit from non-smooth components in adaptation.

This paper clarifies VAE's property through geometric and information-theoretic interpretations.

Study explores how neural networks and Transformers learn modular arithmetic with multiple inputs.

Previous studies have found that an adversary attacker can often infer unintended input information from intermediate-layer features. We study the possibility of preventing such adversarial inference, yet without too much accuracy degradation. We propose a generic method to revise the neural network to boost the challe…

Transformers struggle to learn Markovian dynamics, showing NP-hard optimization challenges.

The paper improves transformer generalization bounds using rank-dependent covering number bounds.

Transformers are less sensitive to input perturbations compared to other models.

Convolutional neural networks (CNNs) have achieved state-of-the-art results on many visual recognition tasks. However, current CNN models still exhibit a poor ability to be invariant to spatial transformations of images. Intuitively, with sufficient layers and parameters, hierarchical combinations of convolution (matri…

Transformers exhibit sparse activation maps, reducing computational load and improving robustness.

Continuum transformers learn operators in context via gradient descent.

Compressive sensing is an impressive approach for fast MRI. It aims at reconstructing MR image using only a few under-sampled data in k-space, enhancing the efficiency of the data acquisition. In this study, we propose to learn priors based on undecimated wavelet transform and an iterative image reconstruction algorith…

This paper improves test-time adaptation for distribution shifts using confidence maximization and input transformation.

Proposes $σ$-PCA to learn identifiable linear transformations without whitening.

A new method reduces memory requirements for Graph Transformers by sparsely training a network.

Efficiently processes dynamic inputs in AI writing assistants with incremental computation.

New method for Transformer models to encode position information without sequential bias.

New bounds show transformers need longer training for length generalization.