Proposes a method to improve neural architectures reproducibly.
problem Lack of reproducibility in Neural Architecture Transformer (NAT).
method Differentiable Neural Architecture Transformation (DNAT).
result DNAT outperforms NAT and is applicable to various models and datasets.
We develop a deep autoencoder architecture that can be used to find a coordinate transformation which turns a nonlinear PDE into a linear PDE. Our architecture is motivated by the linearizing transformations provided by the Cole-Hopf transform for Burgers equation and the inverse scattering transform for completely int…
Sparse transformer architecture improves accuracy and speed in generative modeling and inverse problems.
problem Improving accuracy and speed in generative modeling and inverse problems.
method Proposes a sparse transformer architecture using regularized Wasserstein proximal operator with L1 prior. result Sparse transformer achieves higher accuracy and faster convergence than classical methods.
The Transformer architecture has become increasingly popular over the past two years, owing to its impressive performance on a number of natural language processing (NLP) tasks. However, all Transformer computations occur at the level of word representations and therefore, it may be argued that Transformer models do no…
Intelligent Momentum Transformer outperforms traditional trading strategies.
problem Improving time-series momentum and mean-reversion trading strategies.
method Attention-based deep-learning architecture (Momentum Transformer) combining attention and LSTM.
result Momentum Transformer outperforms benchmarks and adapts to new market regimes.
We introduce a new function-preserving transformation for efficient neural architecture search. This network transformation allows reusing previously trained networks and existing successful architectures that improves sample efficiency. We aim to address the limitation of current network transformation operations that…
Efficient algorithm finds fast Transformer models.
problem Slow inference time of Transformer models.
method Decompose Transformer architecture into components, use sampling-based one-shot search.
result Achieved 10% to 30% speedup on pre-trained BERT and 70% on top of a previous state-of-the-art model.
GATES improves neural architecture search by modeling operations as information transformation.
problem Improving predictor-based neural architecture search efficiency.
method GATES models operations as information transformation, covering both node and edge cell search spaces.
result GATES boosts sample efficiency and improves predictor performance.
New analysis shows transformer models can't learn effectively.
problem Transformer models learning in context can't achieve general predictive accuracy.
method Empirical evidence and mathematical analysis of transformer architecture limitations.
result Transformers cannot achieve general predictive accuracy due to architectural limitations.
Transformer architecture improved with credibility mechanism for better model performance.
problem Improving predictive models in tabular data.
method Introducing a credibility mechanism to the Transformer architecture.
result Credibility Transformer leads to superior predictive models compared to state-of-the-art models.
Recent works have highlighted the strength of the Transformer architecture on sequence tasks while, at the same time, neural architecture search (NAS) has begun to outperform human-designed models. Our goal is to apply NAS to search for a better alternative to the Transformer. We first construct a large search space in…
Transformers use a unique Hessian structure that differs from classical networks, affecting optimization.
problem Understanding the unique optimization landscape of Transformers.
method Theoretical Hessian analysis of a single self-attention layer in Transformers.
result Transformers have a highly non-linear Hessian structure, distinguishing them from classical networks.
BoostTransformer uses boosting to improve transformer efficiency and accuracy.
problem Heavy computational resources and hyperparameter tuning in transformer architectures.
method Augments transformers with boosting principles through subgrid token selection and importance-weighted sampling, incorporating a least square boosting objective directly into the pipeline.
result BoostTransformer demonstrates faster convergence and higher accuracy compared to standard transformers.
This paper compares Transformers and RNNs in various tasks, showing size differences.
problem Comparing representational capabilities of Transformers and RNNs across tasks.
method Analysis of differences in tasks like index lookup, nearest neighbor, and string equality.
result Size differences in Transformers and RNNs for various tasks.
Owing to their ability to both effectively integrate information over long time horizons and scale to massive amounts of data, self-attention architectures have recently shown breakthrough success in natural language processing (NLP), achieving state-of-the-art results in domains such as language modeling and machine t…
This study examines how model architecture affects deep learning model privacy.
problem Privacy concerns in deep learning models due to potential leakage of sensitive information.
method Investigation of CNNs and Transformers, focusing on activation layers, stem layers, LN layers, and attention modules.
result Transformers generally exhibit higher vulnerability to privacy attacks than CNNs.
Designing effective architectures is one of the key factors behind the success of deep neural networks. Existing deep architectures are either manually designed or automatically searched by some Neural Architecture Search (NAS) methods. However, even a well-searched architecture may still contain many non-significant o…
MixerFlow combines MLP-Mixer with normalizing flows for efficient image modeling.
problem Efficiently modeling complex image densities using generative models.
method Proposes MixerFlow, a novel architecture based on MLP-Mixer for normalizing flows.
result Demonstrates improved density estimation and better scaling with higher image resolutions.
Transformer architecture struggles with complex tasks due to limitations in function composition.
problem Transformer architecture's limitations in handling complex tasks.
method Used Communication Complexity to prove limitations in composing functions.
result Transformer layer is incapable of handling large domain functions, even when domains are small.
New mesh network preserves symmetries in deep learning.
problem No existing mesh processing architecture is equivariant to all symmetries.
method Equivariant attention-based mesh network using relative tangential features.
result The network achieves improved performance and is equivariant to various transformations.
This research discovers model architecture and training dataset characteristics through strategic input probing.
problem Discovering model architecture and training dataset characteristics in black box models.
method Structured input probes and model outputs are used to train a deep classifier for image and text classification.
result The approach successfully distinguishes between different image and text datasets and architectures.
Transformers improve Finnish language modeling, achieving lower perplexity scores.
problem Improving language modeling for Finnish using deep learning models.
method Used BERT and Transformer-XL models in a sub-word setting, compared to LSTM.
result Transformer-XL outperforms LSTM, achieving a 27% better perplexity score.
einspace expands NAS search space to include diverse neural architectures.
problem NAS results are often limited to existing structures; new designs are rare.
method einspace uses a probabilistic context-free grammar to create a versatile search space.
result einspace discovers novel and improved architectures, including convolutions and attention.
Unified transformer-based LT-TTD improves ranking efficiency and quality.
problem Decoupled L1 and L2 models in recommendation and search systems cause irreversible error propagation and suboptimal ranking.
method LT-TTD combines two-tower models with transformer expressivity in a unified listwise learning framework, providing theoretical guarantees and UPQE evaluation.
result LT-TTD reduces irretrievable relevant items and achieves better global optimization than disjoint training.
Improved hyperparameter optimization using simplified Transformer blocks.
problem Discovering optimal architectures in high-dimensional search spaces with limited exploration budgets.
method Simplified Transformer block for modeling hyper-parameter dependencies, actor-critic style algorithm, ensembling.
result Outperformed most algorithms on NAS-Bench-101 and Random Search in discovering more accurate model architectures.
Recent advances in Neural Architecture Search (NAS) have produced state-of-the-art architectures on several tasks. NAS shifts the efforts of human experts from developing novel architectures directly to designing architecture search spaces and methods to explore them efficiently. The search space definition captures pr…
Encoder-decoder based sequence-to-sequence models have demonstrated state-of-the-art results in end-to-end automatic speech recognition (ASR). Recently, the transformer architecture, which uses self-attention to model temporal context information, has been shown to achieve significantly lower word error rates (WERs) co…
This paper proposes network recasting as a general method for network architecture transformation. The primary goal of this method is to accelerate the inference process through the transformation, but there can be many other practical applications. The method is based on block-wise recasting; it recasts each source bl…
Recurrent neural network (RNN)'s architecture is a key factor influencing its performance. We propose algorithms to optimize hidden sizes under running time constraint. We convert the discrete optimization into a subset selection problem. By novel transformations, the objective function becomes submodular and constrain…
The scattering transform is a multilayered wavelet-based deep learning architecture that acts as a model of convolutional neural networks. Recently, several works have introduced generalizations of the scattering transform for non-Euclidean settings such as graphs. Our work builds upon these constructions by introducin…
Transformer predicts Ethereum prices using cross-currency correlation and sentiment analysis.
problem Predicting Ethereum cryptocurrency prices with limited data.
method Transformer-based neural network with cross-currency correlation and sentiment analysis.
result Transformer model outperforms other models on some parameters.
ETC improves Transformer models for long and structured inputs.
problem Scaling input length and encoding structured inputs in Transformers.
method Introduces global-local attention, relative position encodings, and CPC pre-training.
result Achieves state-of-the-art results on four natural language datasets.
Transformer models perform slower than convolutional networks in learning hierarchical language structures.
problem Understanding how neural networks learn hierarchical language structures.
method Theoretical scaling laws and empirical validation of neural network performance.
result Convolutional networks outperform transformers in learning hierarchical language structures.
A new UNet variant reduces spectral artifacts in image transformations.
problem Spectral artifacts caused by traditional UNet upsampling layers.
method Introduced a Guided UNet (GUNet) architecture using a novel upsampling module.
result GUNet produces higher fidelity outputs in image transformations.
Sentinel improves time series forecasting by modeling both temporal and channel dependencies.
problem Limited effectiveness of existing transformer-based architectures in multivariate time-series forecasting.
method Proposes Sentinel, a full transformer-based architecture with multi-patch attention mechanism.
result Sentinel achieves better or comparable performance compared to state-of-the-art approaches.
Method learns equivariances from data without custom architecture design.
problem Learning equivariances for tasks without manually designed architectures.
method Reparameterization to learn equivariant parameter sharing.
result Can learn equivariances for any finite group of transformations.
The architecture of Transformer is based entirely on self-attention, and has been shown to outperform models that employ recurrence on sequence transduction tasks such as machine translation. The superior performance of Transformer has been attributed to propagating signals over shorter distances, between positions in …
In this paper, we present a new comparative study on automatic essay scoring (AES). The current state-of-the-art natural language processing (NLP) neural network architectures are used in this work to achieve above human-level accuracy on the publicly available Kaggle AES dataset. We compare two powerful language model…
New theory explains signal propagation in normalization-free transformers.
problem Understanding signal propagation in normalization-free transformers.
method Deriving recurrence relations for activation statistics and APJNs across layers.
result Transformers with elementwise tanh-like nonlinearities exhibit subcritical signal propagation.
Sumformer simplifies Transformers to handle long sequences efficiently.
problem Quadratic complexity of Transformers limits their use with long sequences.
method Introducing Sumformer, a simple architecture that universally approximates equivariant sequence-to-sequence functions.
result Sumformer achieves the first universal approximation results for Linformer and Performer.
A new transformer model accelerates training with optimization techniques.
problem Training deep neural networks efficiently and effectively.
method Interprets transformer layers as optimization steps, applying Nesterov acceleration.
result The new model outperforms existing models on benchmark datasets.
Paper develops a new model for predicting volatility surface.
problem Predicting volatility in financial markets is challenging due to its non-observable nature and complex dynamics.
method Physics-informed convolutional transformer architecture.
result The new model outperforms other deep-learning architectures in predicting volatility surface.
Measures equivariance in vision models using Lie derivative.
problem Understanding the role of equivariance in recent vision models.
method Introducing Lie derivative to measure equivariance with strong mathematical foundations and minimal hyperparameters.
result Many violations of equivariance can be linked to spatial aliasing in network layers, and larger models tend to display more equivariance.
New Transformers maintain Lipschitz continuity for robustness.
problem Ensuring robustness in Transformers for safety-sensitive applications.
method Introducing gradient-descent-type in-context Transformers with explicit Euler steps of negative gradient flows.
result Universal approximation theorem for Lipschitz continuous Transformers.
Transformers can interpolate between arbitrary measures.
problem Understanding the expressive power of Transformers as measure-to-measure maps.
method Provided an explicit choice of parameters for a single Transformer to match N arbitrary input measures to N arbitrary target measures.
result A single Transformer can interpolate between arbitrary measures.
The Transformer architecture is widely used in natural language processing. Despite its success, the design principle of the Transformer remains elusive. In this paper, we provide a novel perspective towards understanding the architecture: we show that the Transformer can be mathematically interpreted as a numerical Or…
Improved Transformer performance by addressing 'explaining away' effect.
problem Transformer's self-attention mechanism can explain away important input features.
method Proposed a doubly-normalized attention scheme to avoid 'explaining away' effect.
result Improved performance on benchmarks with the new attention scheme.
Unified model trained on images and videos using masked autoencoding.
problem Training a single model for multiple visual modalities.
method Masked autoencoding on a Vision Transformer.
result Unified model achieves comparable or better performance than single-modality models.