Simplified NAS for GNN architectures improves efficiency and expressiveness.
problem Efficient and effective discovery of optimal GNN architectures.
method SNAG framework with a novel search space and reinforcement learning.
result SNAG framework outperforms human-designed and existing NAS methods.
Improved hyperparameter optimization using simplified Transformer blocks.
problem Discovering optimal architectures in high-dimensional search spaces with limited exploration budgets.
method Simplified Transformer block for modeling hyper-parameter dependencies, actor-critic style algorithm, ensembling.
result Outperformed most algorithms on NAS-Bench-101 and Random Search in discovering more accurate model architectures.
Deep learning using multi-layer neural networks (NNs) architecture manifests superb power in modern machine learning systems. The trained Deep Neural Networks (DNNs) are typically large. The question we would like to address is whether it is possible to simplify the NN during training process to achieve a reasonable pe…
In this paper, we present a deep neural network (DNN) training approach called the "DeepMimic" training method. Enormous amounts of data are available nowadays for training usage. Yet, only a tiny portion of these data is manually labeled, whereas almost all of the data are unlabeled. The training approach presented ut…
Improved model for multivariate time series prediction with simpler architecture.
problem Multivariate probabilistic time series prediction challenges.
method Simplified transformer-based attentional copulas (TACTiS) with linearly scalable parameters.
result Significantly better training dynamics and state-of-the-art performance.
Implicit deep learning prediction rules generalize the recursive rules of feedforward neural networks. Such rules are based on the solution of a fixed-point equation involving a single vector of hidden features, which is thus only implicitly defined. The implicit framework greatly simplifies the notation of deep learni…
The process of designing neural architectures requires expert knowledge and extensive trial and error. While automated architecture search may simplify these requirements, the recurrent neural network (RNN) architectures generated by existing methods are limited in both flexibility and components. We propose a domain-s…
mlr3torch simplifies deep learning in R for tabular and image data.
problem Simplifying deep learning workflows in R for various data types.
method Built on torch, integrates mlr3 ecosystem, supports custom architectures.
result Extensible framework for tabular and image data, including preprocessing and model definition.
We introduce MinimalRNN, a new recurrent neural network architecture that achieves comparable performance as the popular gated RNNs with a simplified structure. It employs minimal updates within RNN, which not only leads to efficient learning and testing but more importantly better interpretability and trainability. We…
It has been argued in the past that high-dimensional neural networks do not exhibit local minima capable of trapping an optimisation algorithm. However, the relationship between loss surface modality and the neural architecture parameters, such as the number of hidden neurons per layer and the number of hidden layers, …
MushroomRL is an open-source Python library developed to simplify the process of implementing and running Reinforcement Learning (RL) experiments. Compared to other available libraries, MushroomRL has been created with the purpose of providing a comprehensive and flexible framework to minimize the effort in implementin…
In many learning situations, resources at inference time are significantly more constrained than resources at training time. This paper studies a general paradigm, called Differentiable ARchitecture Compression (DARC), that combines model compression and architecture search to learn models that are resource-efficient a…
Ring-reservoir networks simplify graph embeddings efficiently.
problem Efficient graph embeddings using deep neural networks.
method Progressive simplification of Reservoir Computing models to ring topology.
result Ring-reservoir networks show consistent advantages in predictive performance.
Equivariant neural network simplifies particle physics models.
problem Complexity and interpretability in particle physics classification.
method Lorentz group equivariant neural network architecture.
result Simplified, interpretable models with fewer parameters.
No methods currently exist for making arbitrary neural networks fair. In this work we introduce GRAD, a new and simplified method to producing fair neural networks that can be used for auto-encoding fair representations or directly with predictive networks. It is easy to implement and add to existing architectures, has…
The study compares different neural network architectures for option pricing accuracy and training time.
problem Evaluating the impact of network architectures on option pricing accuracy and training time.
method Empirical investigation of various neural network architectures (plain feed forward, highway, DGM) on option pricing problems.
result Generalized highway network architecture achieves the best performance in terms of mean squared error and training time.
New CNN initialization scheme derived from modern architectures.
problem Stability of CNN model parameters initialization.
method Derived new initialization scheme from modern CNN architectures.
result New initialization method outperforms de facto standard schemes.
We present a pairwise learning to rank approach based on a neural net, called DirectRanker, that generalizes the RankNet architecture. We show mathematically that our model is reflexive, antisymmetric, and transitive allowing for simplified training and improved performance. Experimental results on the LETOR MSLR-WEB10…
DFM simplifies CNF training without interpolants.
problem Efficiently training CNFs with computationally expensive ODE solving.
method DFM optimizes dual vector fields for bijective transformations.
result DFM outperforms CNF trained with FM or ML objectives.
FlexServe simplifies deployment of PyTorch models as REST endpoints.
problem Lack of control over model evolution and strict security requirements in operational environments.
method Developed FlexServe, a library to deploy multi-model ensembles with flexible batching.
result Rapid deployment of PyTorch models without intermediate transformations.
Simplified LSTM models improve sentiment analysis on Twitter debate data.
problem Performing sentiment analysis on long sequence data from Twitter debates.
method Developed six parameter-reduced LSTM models (slim LSTM) for faster training and reduced computational cost.
result Slim LSTM models outperform standard LSTM model in sentiment analysis of GOP Debate Twitter dataset.
Residual Network (ResNet) is the state-of-the-art architecture that realizes successful training of really deep neural network. It is also known that good weight initialization of neural network avoids problem of vanishing/exploding gradients. In this paper, simplified models of ResNets are analyzed. We argue that good…
Monotonic neural additive models simplify machine learning for credit scoring.
problem Complex machine learning methods make models less transparent and fair.
method Introducing monotonic neural additive models that simplify neural networks while maintaining regulatory compliance.
result Monotonic neural additive models achieve similar accuracy to complex neural networks but are more transparent and fair.
Unified framework for U-Net design and analysis.
problem Understudied design and architecture of U-Nets.
method Theoretical results, Multi-ResNets, function constraints encoding.
result Competitive and superior performance in various tasks.
Unified access package for fundamental physics datasets simplifies machine learning.
problem Lack of unified access to datasets from multiple fundamental physics disciplines.
method Unified Python package with common interface and reference models.
result Graph-based neural networks perform similarly to dedicated methods on various datasets.
Novel NAS method balances performance and hardware metrics efficiently.
problem Challenging multi-objective optimization in neural architecture search.
method Parameterizes joint architectural distribution via hypernetwork conditioned on hardware features and preferences.
result Zero-shot transferability to new devices with representative and diverse architectures.
TIME network simplifies complex physical processes with interpretable models.
problem Challenges in learning coupled dynamic processes from multiple observations.
method Fully convolutional architecture capturing invariant domain structure.
result Robust and transparent in capturing process kernels and anomalies.
Sumformer simplifies Transformers to handle long sequences efficiently.
problem Quadratic complexity of Transformers limits their use with long sequences.
method Introducing Sumformer, a simple architecture that universally approximates equivariant sequence-to-sequence functions.
result Sumformer achieves the first universal approximation results for Linformer and Performer.
The standard LSTM recurrent neural networks while very powerful in long-range dependency sequence applications have highly complex structure and relatively large (adaptive) parameters. In this work, we present empirical comparison between the standard LSTM recurrent neural network architecture and three new parameter-r…
Improved continuous-time consistency models for large-scale image generation.
problem Training instability and discretization errors in existing diffusion models.
method Unified theoretical framework, improved diffusion process, and network architecture.
result Trained continuous-time CMs at 1.5B parameters, achieving state-of-the-art FID scores.
Improved neural network capacity analysis using simplified RDT.
problem Analyzing the memorization capabilities of sign perceptron neural networks.
method Developed a simplified, partially lifted Random Duality Theory (fl RDT) approach.
result Concrete capacity bounds universally improve over previous best known ones.
Paper studies simplified trisections and their equivalence classes.
problem Understanding right-left equivalence of simplified (2,0)-trisections. method Analyzes simplified trisection diagrams and upper-triangular handle-slides.
result At least two simplified (2,0)-trisections can be right-left equivalent without being related by automorphisms or handle-slides. A longstanding goal in the theory of deep learning is to characterize the conditions under which a given neural network architecture will be trainable, and if so, how well it might generalize to unseen data. In this work, we provide such a characterization in the limit of very wide and very deep networks, for which the…
The EMNLP 2018 workshop BlackboxNLP was dedicated to resources and techniques specifically developed for analyzing and understanding the inner-workings and representations acquired by neural models of language. Approaches included: systematic manipulation of input to neural networks and investigating the impact on thei…
Topology design optimization offers tremendous opportunity in design and manufacturing freedoms by designing and producing a part from the ground-up without a meaningful initial design as required by conventional shape design optimization approaches. Ideally, with adequate problem statements, to formulate and solve the…
AR algorithm simplifies backpropagation with improved scalability and biological plausibility.
problem Improving backpropagation algorithms for complex neural networks and biological plausibility.
method Introducing learnable backwards weights and avoiding nonlinear derivative computations; relaxing frozen feedforward pass assumption.
result Simplified AR algorithm maintains performance on complex CNN architectures and challenging datasets.
Pruning FCNs reveals sub-networks that match CNNs' performance.
problem Understanding the inductive bias of pruning in neural networks.
method Iterative magnitude pruning of a simple FCN followed by analysis of the resulting architecture.
result Pruned FCNs exhibit key features of CNNs, suggesting new architectural biases.
OrthoGrad improves neural calibration by constraining gradient updates orthogonally.
problem Overconfidence in neural networks, leading to poor uncertainty estimates.
method Orthogonal gradient updates to optimize for decision boundaries and reduce overconfidence.
result Significant improvements in test loss, predictive entropy, and confidence measures.
Paper proposes a new sparse Bayesian neural network for simpler, more efficient DNNs.
problem Complex and large DNN architectures require simplification for better performance and efficiency.
method Masked Bayesian Neural Networks (BNN) with nodewise sparsity and optimal posterior distributions.
result The proposed BNN yields well-condensed DNN architectures with similar accuracy and uncertainty quantification to large DNNs.
Gaussian process models simplify neural network behavior for easier understanding.
problem Understanding and predicting the behavior of deep learning systems.
method Constructing surrogate models using Gaussian processes from finite neural networks.
result Surrogate models capture phenomena like spectral bias and predict generalization well.
INNs can approximate diverse functions despite layer restrictions.
problem Can INNs approximate sufficiently diverse functions?
method Developed a theoretical framework based on differential geometry to simplify the approximation problem of diffeomorphisms.
result INNs have the universal approximation property.
Calibration of simplified vine copulas using noise contrastive estimation
problem Modeling complex multivariate dependence structures
method Noise contrastive estimation for calibration
result Improved model accuracy when simplifying assumption is violated
Recently, a lot of techniques were developed to sparsify the weights of neural networks and to remove networks' structure units, e.g. neurons. We adjust the existing sparsification approaches to the gated recurrent architectures. Specifically, in addition to the sparsification of weights and neurons, we propose sparsif…
Ensembles of neural networks improve training dynamics and performance.
problem Improving neural network performance through model size increase.
method Defining collegial ensembles (CE) as multiple independent models trained as a single model, and using theoretical results on NTK to optimize architecture search.
result CE dynamics simplify and scale favorably, resembling wide models, and can be efficiently implemented using group convolutions and block diagonal layers.
Framework detects covert financial market manipulation using LOB representations.
problem Detecting covert financial market manipulation (spoofing) from complex anomaly patterns in multilevel prices.
method Cascaded contrastive representation learning of LOB data.
result Transformer-based architectures achieve state-of-the-art results in detection performance.
PSiLON Net uses L1 weight normalization and 1-path-norm regularization for efficient learning and sparsity.
problem Efficient learning and sparsity in neural networks with limited data.
method PSiLON Net employs L1 weight normalization and 1-path-norm regularization to simplify the 1-path-norm and achieve efficient learning and near-sparse parameters. result PSiLON Net achieves reliable optimization and strong performance in the small data regime.
We simplify deep learning convergence analysis using basic math.
problem Ensuring reliable convergence of optimization algorithms in deep networks.
method Elementary arguments and computations to estimate smoothness constants.
result Systematic computation of smoothness constants for first-order algorithms.
We consider the use of look-up tables (LUT) to simplify the hardware implementation of a deep learning network for inferencing after weights have been successfully trained. The use of LUT replaces the matrix multiply and add operations with a small number of LUTs and addition operations resulting in a completely multip…