Study deep maxout networks and their equivalence to Gaussian processes.
problem Understanding neural networks with infinite width.
method Derive equivalence between deep maxout networks and Gaussian processes, characterize maxout kernel, and provide efficient numerical implementation.
result Bayesian inference based on deep maxout network kernel leads to competitive results compared to finite-width counterparts and deep neural network kernels.
We present a probabilistic variant of the recently introduced maxout unit. The success of deep neural networks utilizing maxout can partly be attributed to favorable performance under dropout, when compared to rectified linear units. It however also depends on the fact that each maxout unit performs a pooling operation…
Random maxout features improve classification and visualization.
problem Improving classification and visualization methods.
method Randomly generated vectors projecting input data, maxout of projections.
result Maxout features allow locally linear estimation and embedding.
Maxout networks study gradients and propose initialization strategies.
problem Complexity in input-output Jacobian distribution complicates stable parameter initialization.
method Obtained bounds on moments of gradients and formulated initialization strategies.
result Parameter initialization strategies improve training of deep maxout networks.
Maxout networks show similar complexity issues as ReLU networks.
problem Understanding the complexity of maxout networks and decision boundaries.
method Analyzing the parameter space and decision boundaries, obtaining lower bounds, and investigating initialization procedures.
result Maxout networks exhibit a wide range of complexity, similar to ReLU networks.
We consider the problem of designing models to leverage a recently introduced approximate model averaging technique called dropout. We define a simple new model called maxout (so named because its output is the max of a set of inputs, and because it is a natural companion to dropout) designed to both facilitate optimiz…
Motivated by an important insight from neural science, we propose a new framework for understanding the success of the recently proposed "maxout" networks. The framework is based on encoding information on sparse pathways and recognizing the correct pathway at inference time. Elaborating further on this insight, we pro…
HyCNNs improve convex function learning and optimal transport.
problem Learning and optimizing convex functions efficiently.
method Combining Maxout networks and ICNNs to create a new neural architecture.
result HyCNNs require fewer parameters and outperform existing methods in convex tasks.
IENs reduce neural network variance without increasing complexity.
problem Reducing variance in neural networks without increasing model complexity.
method IENs use ensemble parameters during training to reduce variance, removing them during testing.
result IENs reduce network variance by a factor of 1/mL−1, leading to significant error rate decreases. Expands learning paradigm to stochastic orders using Choquet-Toland distance and Variational Dominance Criterion.
problem Learning high-dimensional distributions with stochastic orders.
method Introduces Choquet-Toland distance and Variational Dominance Criterion, uses input convex maxout networks (ICMNs).
result Proposes surrogates for Choquet-Toland distance and Variational Dominance Criterion with parametric rates.
Unified framework for robust machine learning models against adversarial examples.
problem Adversarial examples that can fool deep learning models.
method Unified gradient regularization family to penalize input gradients.
result Maxout networks achieve best accuracy on MNIST data.
Deep neural-kernel models combine neural networks and kernel machines for scalable large datasets.
problem Combining neural networks and kernel machines for efficient large-scale learning.
method Hybrid neural-kernel architecture using explicit feature mapping and pooling layers.
result The deep neural-kernel models are effective and scalable on benchmark datasets.
Several machine learning models, including neural networks, consistently misclassify adversarial examples---inputs formed by applying small but intentionally worst-case perturbations to examples from the dataset, such that the perturbed input results in the model outputting an incorrect answer with high confidence. Ear…
We study the complexity of functions computable by deep feedforward neural networks with piecewise linear activations in terms of the symmetries and the number of linear regions that they have. Deep networks are able to sequentially map portions of each layer's input-space to the same output. In this way, deep models c…
Optimizes deep neural networks using splines and adaptive knots.
problem Improving the optimization of deep neural networks.
method Integrates a second-order total-variation criterion to optimize activation functions, deriving a representer theorem.
result Optimal network configurations can be achieved with nonuniform linear splines with adaptive knots.
Max-plus operators improve neural network filter selection and pruning.
problem Improving neural network efficiency and reducing redundancy.
method Exploiting Max-plus operators in neural network layers for filter selection and model pruning.
result Max-plus layers enhance filter selection and reduce redundancy without performance loss.
Study on the complexity of deep neural networks and their linear regions.
problem Understanding the complexity and structure of deep neural networks.
method Leveraging the dimension of the space defining each linear region, the study presents bounds and methods for counting linear regions.
result Tighter upper and lower bounds for the maximum number of linear regions on rectifier networks, and a method for exact enumeration.
New approach treats neural networks with piecewise linear activations using tropical geometry.
problem Upper bounds on linear regions of neural networks with ReLU or leaky ReLU activations.
method Treat neural network layers with piecewise linear activations as tropical polynomials, refining upper bounds using tropical geometry.
result Upper bounds on linear regions improved to $\min\left\{ 2^m, \sum_{j=0}^n \binom{m}{j}
ight\}$, where n,m are the number of inputs and outputs, respectively. A study on preventing catastrophic forgetting in neural networks using conditional computation.
problem Catastrophic forgetting in connectionist neural networks.
method Conditional computation framework where parameters are conditioned on each input example.
result Proposed conditional rehearsal to prevent forgetting of previously learned examples.
Enhances DenseNets with multi-scale convolutions and stochastic feature reuse.
problem Overfitting in DenseNets with dense feature reuse.
method Multi-scale Convolution Aggregation module and Stochastic Feature Reuse.
result Significant improvement in model accuracy with fewer parameters.
A grid layout method for graph classification using CNNs.
problem How to project graphs onto grids for CNNs to work effectively.
method Proposes a novel graph-preserving grid layout (GPGL) using integer programming to minimize topological loss, and solves it approximately with a regularized Kamada-Kawai algorithm.
result Demonstrates the success of the method for graph classification using multi-scale maxout CNNs.
Paper provides convergence rates for rectifier convnets.
problem Understanding why rectifier networks perform well empirically.
method Introduces gated games to capture rectifier units' gating function.
result Gradient descent on rectifier convnets converges to a critical point.
This work generalizes bounds on the number of linear regions in CPWL NNs.
problem Determining the number of linear regions in CPWL neural networks is challenging.
method Generalized bounds on the maximal number of linear regions for arbitrary CPWL activation functions.
result Depth significantly increases the number of linear regions, but not exponentially.
Adversarial training improves neural networks' robustness and accuracy.
problem Creating models that generalize to new data samples.
method Adding an adversarial gradient to the back-propagation method.
result Adversarial back-propagation increases resistance to adversarial examples and boosts classification performance.
In this paper we propose and investigate a novel nonlinear unit, called Lp unit, for deep neural networks. The proposed Lp unit receives signals from several projections of a subset of units in the layer below and computes a normalized Lp norm. We notice two interesting interpretations of the Lp unit. First…