Study on encoding neural architectures for NAS, showing impact on performance.
problem Impact of different encodings on NAS performance.
method Formal definition and characterization of encodings, empirical study of encodings' effectiveness.
result Different encodings can significantly affect NAS performance.
Study efficient neural operator learning using variation spaces.
problem Operator learning using encoder-decoder neural networks.
method Introduce variation space for nonlinear operators, establish approximation bounds.
result Algebraic approximation and learning rates for polynomially decaying input and output encoding errors.
Binary encoding enables neural networks to extrapolate periodic functions.
problem Extrapolating periodic functions without prior knowledge of their form.
method Normalized Base-2 Encoding (NB2E) for continuous numerical values.
result MLPs using NB2E can successfully extrapolate diverse periodic signals.
Decodes neural activity to detect context effects in natural settings.
problem Detecting context-dependent changes in neural encoding.
method Decoding-based approach controlling for confounding factors.
result Demonstrates context-dependent changes in neural encoding.
CNNs encode data augmentation transformations, especially in early layers.
problem Whether neural network features encode data augmentation transformations.
method Systematic approach using pre-trained vision models to predict augmentation transformations.
result Neural network features encode data augmentation transformations, especially in early layers.
MUTE improves neural network performance with efficient target encoding.
problem Improving neural network performance with limited resources.
method MUTE optimizes Hamming distances among target encoding by understanding class confusion.
result MUTE offers better generalization and robustness with minimal overhead.
A simple encoder and complex decoder for secure image encryption and decryption.
problem Secure and efficient image encryption and decryption.
method Uses a shallow encoder neural network for encryption and a deep decoder for decryption, trained independently.
result Decrypted images are nearly identical to the original, demonstrating the effectiveness of the framework.
Neural networks encode inputs deterministically and categorically, behaving like hash encoders.
problem Understanding the encoding properties of neural networks.
method Analyzed the input space partitioned by ReLU-like activations in neural networks.
result Neural networks can be represented by unique activation patterns, similar to hash encoders.
Paper proposes LCP for structural encodings, outperforming existing methods.
problem Improving Graph Neural Networks performance through effective structural encodings.
method Geometric perspective, Local Curvature Profiles (LCP) for structural encodings, combining with global positional encodings, comparing with rewiring techniques.
result LCP significantly outperforms existing structural encodings and combining LCP with global positional encodings improves performance.
Neural machine translation is a relatively new approach to statistical machine translation based purely on neural networks. The neural machine translation models often consist of an encoder and a decoder. The encoder extracts a fixed-length representation from a variable-length input sentence, and the decoder generates…
GATES improves neural architecture search by modeling operations as information transformation.
problem Improving predictor-based neural architecture search efficiency.
method GATES models operations as information transformation, covering both node and edge cell search spaces.
result GATES boosts sample efficiency and improves predictor performance.
This work uses decision trees to encode relevant features and their interactions into neural networks, improving model performance.
problem Overfitting in neural networks with many irrelevant variables.
method Defines a mapping to encode decision tree extracted relationships into a neural network.
result The approach outperforms fully connected neural networks and tree-based methods.
The paper identifies saddlepoints in unsupervised auto-encoding neural nets.
problem The risk landscape of unsupervised least squares in auto-encoding neural nets.
method Established an equivalence between unsupervised least squares and principal manifolds, discussed regularization strategies for auto-encoders.
result All non-trivial critical points in auto-encoding are saddlepoints, which are degenerate in overcomplete auto-encoding.
A model learns stock trading rules from raw prices using encoder-decoder neural network.
problem Extracting features from long price sequences for profitable trading rules.
method Neural encoder-decoder framework combined with DRL.
result The model outperforms state-of-the-art models in dynamic environments.
Paper explores neural network approximations on sphere domains.
problem Approximating functionals on sphere domains using neural networks.
method Encoder-decoder framework with spherical harmonics for infinite-dimensional domain.
result Approximation rates of neural networks with different encoder structures.
The paper uses neural networks to forecast time series data.
problem Forecasting high-dimensional stationary processes.
method Encoder-decoder neural network structure to model past observations.
result Upper bounds for forecast error under specific assumptions.
Proposes a method to improve skull stripping accuracy in MRI images.
problem Skull stripping accuracy in MRI images is improved.
method Context-encoding method to empower 2D networks with 3D semantic information.
result Achieves superior accuracy (dice score 99.6% on NFBS, 99.09% on LPBA40, 99.17% on OASIS) compared to state-of-the-art methods.
Stochastic encoders outperform deterministic ones in 'perfect perceptual quality'.
problem Understanding when stochastic encoders outperform deterministic ones.
method Provided a toy example to illustrate performance.
result Stochastic encoders can significantly outperform deterministic ones in 'perfect perceptual quality'.
A new histogram layer improves texture analysis performance.
problem Extracting features for texture analysis from local spatial regions.
method Directly computes local spatial distribution of features during backpropagation.
result Improves performance on three material/texture datasets.
Sparse neural encoding can store more memories as targets become sparser.
problem Storing sparse input-target associations in neural networks.
method Mathematical proofs using properties of random polytopes and sub-gaussian random vector variables.
result The capacity of neural maps increases with sparsity in target layers.
We introduce a novel encoder-decoder architecture to embed functional processes into latent vector spaces. This embedding can then be decoded to sample the encoded functions over any arbitrary domain. This autoencoder generalizes the recently introduced Conditional Neural Process (CNP) model of random processes. Our ar…
Despite the superior performance of deep learning in many applications, challenges remain in the area of regression on function spaces. In particular, neural networks are unable to encode function inputs compactly as each node encodes just a real value. We propose a novel idea to address this shortcoming: to encode an …
Although neural machine translation with the encoder-decoder framework has achieved great success recently, it still suffers drawbacks of forgetting distant information, which is an inherent disadvantage of recurrent neural network structure, and disregarding relationship between source words during encoding step. Wher…
We present a memory augmented neural network for natural language understanding: Neural Semantic Encoders. NSE is equipped with a novel memory update rule and has a variable sized encoding memory that evolves over time and maintains the understanding of input sequences through read}, compose and write operations. NSE c…
Memory-augmented neural networks (MANNs) have been shown to outperform other recurrent neural network architectures on a series of artificial sequence learning tasks, yet they have had limited application to real-world tasks. We evaluate direct application of Neural Turing Machines (NTM) and Differentiable Neural Compu…
In this paper, we propose a novel neural network model called RNN Encoder-Decoder that consists of two recurrent neural networks (RNN). One RNN encodes a sequence of symbols into a fixed-length vector representation, and the other decodes the representation into another sequence of symbols. The encoder and decoder of t…
BANANAS uses Bayesian optimization with neural predictors to improve neural architecture search.
problem Improving neural architecture search (NAS) performance.
method BANANAS combines Bayesian optimization with neural predictors, focusing on architecture encoding, predictor, uncertainty calibration, acquisition function, and optimization strategy.
result BANANAS achieves state-of-the-art performance on NAS search spaces.
Proposes a method to train neural networks directly on compressed text data.
problem Training neural networks on compressed text data without decompression.
method Introduces composer modules to encode symbols from grammar compression rules into vector representations.
result Demonstrates that the proposed method can achieve both memory and computational efficiency while maintaining moderate performance.
This paper proposes CodeX, an end-to-end framework that facilitates encoding, bitwidth customization, fine-tuning, and implementation of neural networks on FPGA platforms. CodeX incorporates nonlinear encoding to the computation flow of neural networks to save memory. The encoded features demand significantly lower sto…
Performance of neural networks can be significantly improved by encoding known invariance for particular tasks. Many image classification tasks, such as those related to cellular imaging, exhibit invariance to rotation. We present a novel scheme using the magnitude response of the 2D-discrete-Fourier transform (2D-DFT)…
PE-GQNN improves spatial data prediction and uncertainty quantification.
problem Poor calibration of predictive distributions in spatial data models.
method Combines PE-GNNs with Quantile Neural Networks and recalibration techniques.
result PE-GQNN outperforms existing methods in predictive accuracy and uncertainty quantification.
A new model improves recurrent neural networks' ability to memorize long sequences.
problem Improving recurrent neural networks' ability to memorize long sequences and extract task-relevant features.
method Proposes a Linear Memory Network with an encoding-based memorization component and a specialized training algorithm.
result Improves the final performance of recurrent neural networks when memorizing long sequences is necessary.
Understanding properties of deep neural networks is an important challenge in deep learning. In this paper, we take a step in this direction by proposing a rigorous way of verifying properties of a popular class of neural networks, Binarized Neural Networks, using the well-developed means of Boolean satisfiability. Our…
Neural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that can be jointly tuned to maximize the translation performance. The models proposed recently for neural ma…
New method for Transformer models to encode position information without sequential bias.
problem Lack of flexible and learnable position encoding for Transformer models.
method Continuous dynamical model to learn position encoding.
result Consistent improvements over baselines in various NLP tasks.
The condition number predicts efficient information encoding in neural units, aiding model fine-tuning.
problem Efficient information encoding in neural units for various tasks and input modalities.
method Linking the condition number to the log-volume scaling factor and entropy of the output distribution.
result High condition number indicates efficient encoding, reducing overall information transfer.
There has been significant recent interest towards achieving highly efficient deep neural network architectures. A promising paradigm for achieving this is the concept of evolutionary deep intelligence, which attempts to mimic biological evolution processes to synthesize highly-efficient deep neural networks over succe…
Affine spiking neural networks learn efficiently and generalize well.
problem Learning with spiking neural networks, especially with positive weights.
method Affine encoders and decoders, continuous parameter dependence, gradient-based training.
result Affine spiking neural networks can approximate shallow ReLU networks and generalize well.
We study the use of a time series encoder to learn representations that are useful on data set types with which it has not been trained on. The encoder is formed of a convolutional neural network whose temporal output is summarized by a convolutional attention mechanism. This way, we obtain a compact, fixed-length repr…
Graph neural network constructs a sparse latent point cloud from dense point clouds.
problem Efficiently reconstructing and simulating point clouds with fine details.
method Irregular graph convolutional neural network with non-isotropic operations.
result The model can reconstruct dense point clouds from a sparse latent representation.
Derives formulae for general permutation equivariant layers and presents a second order graph variational encoder.
problem Tackles the limitation of previous equivariant neural networks by considering permutations of matrices.
method Derives formulae for general permutation equivariant layers, including matrix permutations. Presents a second order graph variational encoder.
result Latent distribution of equivariant generative models must be exchangeable.
MetaFun learns functional representations for meta-learning.
problem Few-shot classification on large-scale datasets.
method Functional encoder-decoder approach with iterative updates.
result State-of-the-art performance on miniImageNet and tieredImageNet.
Encoder-decoder networks using convolutional neural network (CNN) architecture have been extensively used in deep learning literatures thanks to its excellent performance for various inverse problems. However, it is still difficult to obtain coherent geometric view why such an architecture gives the desired performance…
Different neural networks trained on the same dataset often learn similar input-output mappings with very different weights. Is there some correspondence between these neural network solutions? For linear networks, it has been shown that different instances of the same network architecture encode the same representatio…
The large memory requirements of deep neural networks limit their deployment and adoption on many devices. Model compression methods effectively reduce the memory requirements of these models, usually through applying transformations such as weight pruning or quantization. In this paper, we present a novel scheme for l…
Hybrid Bayesian neural networks use function uncertainty for probabilistic inference.
problem Uncertainty in neural network weights is hard to specify and interpret.
method Integrates probabilistic layers with standard deterministic layers for function uncertainty.
result Improves probabilistic inference by encoding function uncertainty.
New method encodes function preferences into neural nets for better generalization.
problem Challenges in encoding explicit function preferences in neural network training.
method Function-space empirical Bayes (FSEB) regularization.
result FSEB leads to near-perfect semantic shift detection and improved generalization.
Improves neural network regression performance using encoder-decoder approach.
problem Suboptimal regression results from classical methods in neural networks.
method Proposes a new method using an encoder-decoder approach for regression.
result Shows improved performance on various real-world datasets.