SPTN uses invertible transformations to improve sum-product networks.
problem Improving inference efficiency and tractability in sum-product networks.
method Integrates invertible transformations into sum-product networks (SPNs).
result SPTNs with Gaussian leaves and affine transformations are as tractable as SPNs.
Transformers can simulate MLE for Bayesian network sequences.
problem Understanding transformers' capabilities in Bayesian network sequence generation.
method In-context maximum likelihood estimation (MLE) for autoregressive sequence generation.
result A simple transformer model can estimate Bayesian network probabilities and generate new samples.
Researchers improve transformer networks' optimization and understanding.
problem Improving the understanding and optimization of transformer networks.
method Introducing a convex alternative to the self-attention mechanism and reformulating the training problem as a convex optimization problem.
result Revealed an implicit regularization mechanism that promotes sparsity across tokens.
Unified method for deriving ridgelet transforms for various neural network architectures.
problem Deriving closed-form expressions for ridgelet transforms in modern neural network architectures.
method Unified Fourier slice method to derive ridgelet transforms for diverse neural network types.
result Systematic method to derive ridgelet transforms for various neural network architectures.
CNNs encode data augmentation transformations, especially in early layers.
problem Whether neural network features encode data augmentation transformations.
method Systematic approach using pre-trained vision models to predict augmentation transformations.
result Neural network features encode data augmentation transformations, especially in early layers.
B-cos transforms improve neural network interpretability by aligning weights.
problem Improving interpretability of deep neural networks.
method Replacing linear transforms with B-cos transforms that promote weight-input alignment during training.
result B-cos transforms lead to highly interpretable and task-relevant linear summaries of neural network computations.
Quantum Ridgelet Transform speeds up neural network learning.
problem Efficiently finding sparse trainable subnetworks in neural networks.
method Developed a quantum ridgelet transform (QRT) for linear runtime.
result Quantum Ridgelet Transform efficiently finds sparse trainable subnetworks.
This paper proposes network recasting as a general method for network architecture transformation. The primary goal of this method is to accelerate the inference process through the transformation, but there can be many other practical applications. The method is based on block-wise recasting; it recasts each source bl…
CaTs use DAGs with transformers to enforce causal constraints, improving neural network robustness.
problem Neural networks lack inherent causal structure respect, leading to reliability issues.
method Introducing Causal Transformers (CaTs) that operate under predefined causal constraints specified by DAGs.
result CaTs improve robustness and interpretability of neural networks under causal constraints.
Scattering transforms are non-trainable deep convolutional architectures that exploit the multi-scale resolution of a wavelet filter bank to obtain an appropriate representation of data. More importantly, they are proven invariant to translations, and stable to perturbations that are close to translations. This stabili…
The study analyzes the convergence rate of a large transformer model.
problem Understanding the convergence rate of over-parameterized transformer models.
method Theoretical analysis focusing on gradient descent optimization.
result Theoretical upper bound on missclassification probability.
Proposes a neural network method to correct residual distortions in coordinate transformations.
problem Nonlinear and spatially dependent distortions in coordinate transformation models.
method Residual-based neural network approach focusing on systematic distortions.
result The method improves accuracy and stability in challenging conditions.
Enhances non-life insurance pricing models using transformer models.
problem Improving predictive power of non-life insurance pricing models.
method Enhances actuarial non-life models with transformer models for tabular data.
result Transformer models outperform benchmark models in claim frequency prediction.
Data augmentation (DA) is fundamental against overfitting in large convolutional neural networks, especially with a limited training dataset. In images, DA is usually based on heuristic transformations, like geometric or color transformations. Instead of using predefined transformations, our work learns data augmentati…
Transforms between neural networks using manifold-learning techniques.
problem Establish equivalence between different neural networks.
method Diffusion maps with a Mahalanobis-like metric to construct transformations between network outputs and internal neuron activations.
result Established equivalence classes between neural networks trained on various data types.
Learning image transformations is essential to the idea of mental simulation as a method of cognitive inference. We take a connectionist modeling approach, using planar neural networks to learn fundamental imagery transformations, like translation, rotation, and scaling, from perceptual experiences in the form of image…
Top-down information plays a central role in human perception, but plays relatively little role in many current state-of-the-art deep networks, such as Convolutional Neural Networks (CNNs). This work seeks to explore a path by which top-down information can have a direct impact within current deep networks. We explore …
The wavelet transform has seen success when incorporated into neural network architectures, such as in wavelet scattering networks. More recently, it has been shown that the dual-tree complex wavelet transform can provide better representations than the standard transform. With this in mind, we extend our previous meth…
We prove that the global minimum of the backpropagation (BP) training problem of neural networks with an arbitrary nonlinear activation is given by the ridgelet transform. A series of computational experiments show that there exists an interesting similarity between the scatter plot of hidden parameters in a shallow ne…
Probabilistic STNs improve image classification and robustness.
problem Training and robustness issues in STNs.
method Probabilistic extension of STNs that estimates stochastic transformations.
result Improved classification performance, robustness, and model calibration.
Develops first robustness verification for complex Transformers.
problem Certify prediction behavior of Transformers with complex self-attention layers.
method Resolves challenges of cross-nonlinearity and cross-position dependency in Transformers.
result Certified robustness bounds are significantly tighter than Interval Bound Propagation.
Transformers can outperform feedforward and recurrent networks due to dynamic sparsity.
problem Understanding when and why Transformers outperform other neural network architectures.
method Analyzing a sequence-to-sequence data generating model with dynamic sparsity, proving sample complexity differences between feedforward, recurrent, and Transformers.
result Transformers can learn dynamic sparsity models with lower sample complexity than feedforward and recurrent networks.
Fusion of transformer networks using optimal transport for improved performance.
problem Improving performance of transformer-based models through fusion.
method Exploiting optimal transport for soft alignment of transformer components.
result Consistently outperforms vanilla fusion and individual parent models.
The key idea of current deep learning methods for dense prediction is to apply a model on a regular patch centered on each pixel to make pixel-wise predictions. These methods are limited in the sense that the patches are determined by network architecture instead of learned from data. In this work, we propose the dense…
GIT-Net uses neural networks to approximate PDE operators efficiently.
problem Approximating PDE operators for complex geometries.
method Parametrizes adaptive generalized integral transforms with deep neural networks.
result GIT-Net outperforms existing neural network operators in multiple areas.
Transformer networks approximate Hölder and Sobolev functions with fixed-depth networks.
problem Nonparametric regression with dependent observations.
method Established novel upper bounds for Transformer networks approximating Hölder and Sobolev functions under various β-mixing data assumptions. result Explicit convergence rates for nonparametric regression problems under β-mixing data assumptions. The paper explores properties of the Radon transform in relation to neural networks and ridges.
problem Understanding the Radon transform and its application to neural networks and ridges.
method Investigates properties of the Radon transform, introduces new subspaces, and characterizes ridges for any distributional profile.
result Clarifies and simplifies results on the optimality of ReLU networks using the Radon transform.
Previous research has shown that computation of convolution in the frequency domain provides a significant speedup versus traditional convolution network implementations. However, this performance increase comes at the expense of repeatedly computing the transform and its inverse in order to apply other network operati…
This thesis explores GNNs, categorizing them into local and global approaches.
problem Understanding the convergence of global GNNs and connecting local and global approaches.
method Categorization of GNNs into local and global, study of Invariant Graph Networks, connecting local and global approaches, and using local MPNN for graph coarsening.
result Established a connection between local and global GNN approaches.
An audiovisual speaker conversion method is presented for simultaneously transforming the facial expressions and voice of a source speaker into those of a target speaker. Transforming the facial and acoustic features together makes it possible for the converted voice and facial expressions to be highly correlated and f…
We present graph wavelet neural network (GWNN), a novel graph convolutional neural network (CNN), leveraging graph wavelet transform to address the shortcomings of previous spectral graph CNN methods that depend on graph Fourier transform. Different from graph Fourier transform, graph wavelet transform can be obtained …
The paper provides convergence guarantees for ODE-based generative models using transformers.
problem Theoretical guarantees for ODE-based generative models.
method A pre-trained autoencoder maps inputs to a latent space, and a transformer predicts the velocity field.
result The distribution of samples generated via estimated ODE flow converges to the target distribution in Wasserstein-2 distance.
Variational inference provides approximations to the computationally intractable posterior distribution in Bayesian networks. A prominent medical application of noisy-or Bayesian network is to infer potential diseases given observed symptoms. Previous studies focus on approximating a handful of complicated pathological…
Study presents a method to induce a generalized neural network from joint group invariant functions.
problem Encoding rule of neural network internal data representation.
method Systematic method using joint group invariant function on data-parameter domain.
result Induces a generalized neural network and its inverse operator (ridgelet transform).
This research studies affine invariance in continuous-domain convolutional neural networks.
problem Recognizing patterns and features under affine transformations in continuous domains.
method Introduces a new criterion for assessing affine invariance, embeds images into the affine Lie group, and analyzes convolution over this group.
result Extends the scope of geometrical transformations that deep-learning pipelines can handle.
Transformer-M learns molecular data in 2D or 3D formats.
problem Learning models for molecules are limited to specific data formats.
method Developed a Transformer-based model that can handle 2D and 3D molecular data.
result Transformer-M achieves strong performance on both 2D and 3D molecular tasks.
Single-layer Transformer can approximate any sequence mapping.
problem Lack of theoretical understanding of Transformers.
method Review of linear algebra, probability, and optimization concepts; detailed analysis of Transformer architecture.
result A single-layer Transformer can approximate any continuous sequence-to-sequence mapping to arbitrary precision.
Study explores how neural networks and Transformers learn modular arithmetic with multiple inputs.
problem Understanding how neural networks and Transformers learn modular arithmetic with multiple inputs.
method Analytical characterization of features learned by neural networks and Transformers, focusing on margin maximization and Fourier spectra.
result Neural networks and Transformers require a minimum neuron count of \( m \geq 2^{2k-2} \cdot (p-1) \) to solve modular addition problems with \( k \) inputs and modulus \( p \).
Deep linear ResNets converge globally with certain transformations.
problem Global convergence of training deep linear ResNets.
method Gradient descent and stochastic gradient descent for training L-hidden-layer linear ResNets. result GD and SGD can converge to global minimum for deep linear ResNets with specific transformations.
Transformers excel at sparse token selection, surpassing FCNs in both worst and average cases.
problem Sparse token selection task
method One-layer transformer trained with gradient descent
result Transformers learn sparse token selection and exhibit strong out-of-distribution length generalization
The scattering transform is a multilayered wavelet-based deep learning architecture that acts as a model of convolutional neural networks. Recently, several works have introduced generalizations of the scattering transform for non-Euclidean settings such as graphs. Our work builds upon these constructions by introducin…
We propose unitary group convolutions (UGConvs), a building block for CNNs which compose a group convolution with unitary transforms in feature space to learn a richer set of representations than group convolution alone. UGConvs generalize two disparate ideas in CNN architecture, channel shuffling (i.e. ShuffleNet) and…
How can we recognise social roles of people, given a completely unlabelled social network? We present a transfer learning approach to network role classification based on feature transformations from each network's local feature distribution to a global feature space. Experiments are carried out on real-world datasets.…
New mesh network preserves symmetries in deep learning.
problem No existing mesh processing architecture is equivariant to all symmetries.
method Equivariant attention-based mesh network using relative tangential features.
result The network achieves improved performance and is equivariant to various transformations.
Transformers approximate mean-field dynamics of indistinguishable particles.
problem Approximating the dynamics of indistinguishable particles in complex systems.
method Using transformers to model the mean-field dynamics of interacting particle systems.
result Theoretical bounds on the distance between true and transformer-obtained mean-field dynamics.
Transformers struggle to approximate smooth functions, relying on piecewise constant approximations.
problem Understanding the expressivity of Transformers for function approximation.
method Theoretical analysis and experimental validation of Transformer's ability to approximate smooth functions.
result Transformers cannot reliably approximate smooth functions, relying on piecewise constant approximations.
How can prior knowledge on the transformation invariances of a domain be incorporated into the architecture of a neural network? We propose Equivariant Transformers (ETs), a family of differentiable image-to-image mappings that improve the robustness of models towards pre-defined continuous transformation groups. Throu…
We introduce a general method of performing Residual Network inference and learning in the JPEG transform domain that allows the network to consume compressed images as input. Our formulation leverages the linearity of the JPEG transform to redefine convolution and batch normalization with a tune-able numerical approxi…