Wider networks improve natural accuracy but worsen perturbation stability, affecting overall robustness.
problem Understanding the tradeoff between natural accuracy and perturbation stability in wider neural networks for adversarial robustness.
method Careful examination of the relationship between network width, robust regularization parameter λ, and perturbation stability using neural tangent kernels.
result Wider networks can achieve better natural accuracy but worse perturbation stability, leading to potentially worse overall model robustness.
This paper compares deeper and wider neural networks for optimal generalization error in Sobolev losses.
problem The dilemma of choosing between deeper or wider neural networks for optimal generalization error.
method Analytical investigations into the influence of sample points, parameters, and loss function regularity on neural network architecture.
result A higher number of parameters favors wider neural networks, while more sample points and greater loss function regularity favor deeper neural networks.
We theoretically study the landscape of the training error for neural networks in overparameterized cases. We consider three basic methods for embedding a network into a wider one with more hidden units, and discuss whether a minimum point of the narrower network gives a minimum or saddle point of the wider one. Our re…
This work improves the lottery ticket hypothesis by reducing over-parameterization requirement.
problem Approximating a neural network by pruning a randomly over-parameterized network.
method Connecting pruning ReLU networks to extsc{SubsetSum} problem, showing logarithmic over-parameterization sufficiency.
result Logarithmic over-parameterization is sufficient for approximating any target neural network.
Differentiable NAS frameworks grow networks wider and deeper, revealing biases in wiring evolution.
problem Understanding the evolution of neural architecture wiring in differentiable NAS methods.
method Unified view on searching algorithms, local cost minimization, empirical and theoretical analyses.
result Implicit inductive biases cause observed searching patterns in differentiable NAS methods.
Distributed implementations of mini-batch stochastic gradient descent (SGD) suffer from communication overheads, attributed to the high frequency of gradient updates inherent in small-batch training. Training with large batches can reduce these overheads; however, large batches can affect the convergence properties and…
Wider neural networks have predominantly positive curvature, aiding optimization.
problem Understanding the convex behavior of deep neural networks with varying layer widths.
method Hessian decomposition and gradient analysis of over-parameterized networks.
result For wide networks, the Hessian is dominated by the positive component G, leading to positive curvature.
Wider neural networks perform better than deeper ones with the same number of parameters.
problem Understanding the role of network width versus the number of parameters in neural network performance.
method Comparing models with different ways of increasing width while keeping the number of parameters constant, analyzing their performance and using Gaussian Process kernels for analysis.
result Network width is the determining factor for good performance, while the number of weights is secondary as long as trainability is ensured.
SAM optimizes deep networks by oscillating between sides of the minimum.
problem Improving performance of deep networks.
method Gradient-based optimization method that oscillates between sides of the minimum.
result SAM effectively performs gradient descent on the spectral norm of the Hessian, encouraging drift towards wider minima.
New method identifies parameters of wider shallow neural networks with biases.
problem Identifying parameters of wide shallow neural networks with biases from finite samples.
method Two-step pipeline: direction of weights via second order information, signs via algebraic evaluations, biases via gradient descent.
result Constructive methods and theoretical guarantees of finite sample identification for wider shallow networks with biases.
Hides the complexity of neural networks, making them more transparent.
problem Lack of transparency in Neural Networks hinders their adoption.
method Proposes Hide-and-Seek (HnS) framework for training interpretable neural networks.
result Interpretable neural networks can be trained without sacrificing predictive power.
It is shown how the well-known class of bihamiltonian structures in general position can be extended to a wider class. A generalization of the corresponding notion of a Veronese web for this wider class is presented (in the general position case Veronese webs form complete systems of local invariants for bihamiltonian …
Current UAV-recorded datasets are mostly limited to action recognition and object tracking, whereas the gesture signals datasets were mostly recorded in indoor spaces. Currently, there is no outdoor recorded public video dataset for UAV commanding signals. Gesture signals can be effectively used with UAVs by leveraging…
We investigate Relational Graph Attention Networks, a class of models that extends non-relational graph attention mechanisms to incorporate relational information, opening up these methods to a wider variety of problems. A thorough evaluation of these models is performed, and comparisons are made against established be…
We present a Statistical Mechanics (SM) model of deep neural networks, connecting the energy-based and the feed forward networks (FFN) approach. We infer that FFN can be understood as performing three basic steps: encoding, representation validation and propagation. From the meanfield solution of the model, we obtain a…
Pruning unimportant parameters can allow deep neural networks (DNNs) to reduce their heavy computation and memory requirements. A saliency metric estimates which parameters can be safely pruned with little impact on the classification performance of the DNN. Many saliency metrics have been proposed, each within the con…
Transferability of learned features between tasks can massively reduce the cost of training a neural network on a novel task. We investigate the effect of network width on learned features using activation atlases --- a visualization technique that captures features the entire hidden state responds to, as opposed to in…
Paper proposes a new binary quantization method for faster DNN inference.
problem Accelerating deep neural network inference on resource-limited devices.
method Quantized Compressed Sensing (QCS) for binary quantization.
result The proposed method preserves benefits of standard methods while reducing quantization error.
New bound explains why high-rank neural nets generalize well.
problem Understanding why high-rank neural networks generalize well.
method Using Koopman operators, group representations, and RKHSs, a new Rademacher complexity bound is derived.
result Derives a bound for a wider range of realistic models.
The Regularized Nonlinear Acceleration (RNA) algorithm is an acceleration method capable of improving the rate of convergence of many optimization schemes such as gradient descend, SAGA or SVRG. Until now, its analysis is limited to convex problems, but empirical observations shows that RNA may be extended to wider set…
Marginalising out uncertain quantities within the internal representations or parameters of neural networks is of central importance for a wide range of learning techniques, such as empirical, variational or full Bayesian methods. We set out to generalise fast dropout (Wang & Manning, 2013) to cover a wider variety of …
Previous work has questioned the conditions under which the decision regions of a neural network are connected and further showed the implications of the corresponding theory to the problem of adversarial manipulation of classifiers. It has been proven that for a class of activation functions including leaky ReLU, neur…
Machine learning improves air pollution predictions in low-cost sensor networks.
problem Invalid predictions from low-cost sensors in resource-constrained countries.
method Probabilistic machine learning to model sensor calibration as distributions or Gaussian processes.
result Improves air pollution predictions and decision-making processes.
Empirical study shows standard CNNs deviate from NTK predictions.
problem Understanding how standard finite-width CNNs behave compared to their infinite-width NTK counterparts.
method Empirical analysis of AlexNet and LeNet architectures.
result Standard CNNs deviate significantly from their NTK counterparts, but deviation decreases with wider networks.
Motivated by an important insight from neural science, we propose a new framework for understanding the success of the recently proposed "maxout" networks. The framework is based on encoding information on sparse pathways and recognizing the correct pathway at inference time. Elaborating further on this insight, we pro…
Model predicts stock prices using GAN and RoI Pooling.
problem Predicting stock prices influenced by macroeconomic factors.
method Markov Decision Process, GAN, RoI Pooling.
result Identifies macroeconomic factors' influence on stock prices.
A barrier to the wider adoption of neural networks is their lack of interpretability. While local explanation methods exist for one prediction, most global attributions still reduce neural network decisions to a single set of features. In response, we present an approach for generating global attributions called GAM, w…
Wide Bayesian neural networks have a simpler weight posterior, leading to faster MCMC sampling.
problem Sampling from the posterior of wide Bayesian neural networks is challenging.
method Introducing repriorisation, a data-dependent reparameterisation that simplifies the posterior distribution.
result The repriorisation map accelerates MCMC sampling, achieving up to 50x higher effective sample size.
The paper audits trading filters, finding a high save-to-miss ratio.
problem Improving the efficiency and accuracy of trading filters in decentralized exchanges.
method A precision audit of filter rules against real trading data, classifying rejection events.
result Conservative save-to-miss ratio of 3.7 : 1, with wider interpretation of 14.8 : 1.
MLP-Mixer achieves better performance through sparsity and wider architecture.
problem Understanding why MLP-Mixer outperforms conventional MLPs.
method Revealed sparseness as a key mechanism, showed effective expression as wider MLP, demonstrated quantitative similarities, and applied a guiding principle.
result MLP-Mixer's performance improvement through sparseness and wider architecture.
Deep neural networks are typically trained by optimizing a loss function with an SGD variant, in conjunction with a decaying learning rate, until convergence. We show that simple averaging of multiple points along the trajectory of SGD, with a cyclical or constant learning rate, leads to better generalization than conv…
Extends ONNX for quantized neural networks with new formats and operators.
problem Handling arbitrary-precision quantization in neural networks.
method Introduces new formats and operators in ONNX to represent quantized neural networks.
result Enabled representation of uniform quantization in neural networks.
Long Short-Term Memory (LSTM) is a popular approach to boosting the ability of Recurrent Neural Networks to store longer term temporal information. The capacity of an LSTM network can be increased by widening and adding layers. However, usually the former introduces additional parameters, while the latter increases the…
Improved generalization bounds for CNNs using Rademacher complexity.
problem Establishing non-vacuous generalization bounds for deep learning models.
method Rademacher complexity framework with novel contraction lemmas for high-dimensional mappings.
result Enhanced generalization bounds for a broader class of activation functions.
When applying machine learning to sensitive data, one has to find a balance between accuracy, information security, and computational-complexity. Recent studies combined Homomorphic Encryption with neural networks to make inferences while protecting against information leakage. However, these methods are limited by the…
Accurate and robust cell nuclei classification is the cornerstone for a wider range of tasks in digital and Computational Pathology. However, most machine learning systems require extensive labeling from expert pathologists for each individual problem at hand, with no or limited abilities for knowledge transfer between…
The bias-variance tradeoff tells us that as model complexity increases, bias falls and variances increases, leading to a U-shaped test error curve. However, recent empirical results with over-parameterized neural networks are marked by a striking absence of the classic U-shaped test error curve: test error keeps decrea…
Generative adversarial networks (GANs) provide an algorithmic framework for constructing generative models with several appealing properties: they do not require a likelihood function to be specified, only a generating procedure; they provide samples that are sharp and compelling; and they allow us to harness our knowl…
New learning rules for wide neural networks without backpropagation.
problem Training wide neural networks efficiently and without backpropagation.
method Input-weight alignment driven by gradient descent in the NTK regime.
result Biologically-motivated learning rules equivalent to backpropagation in wide networks.
Proposes a neural network for efficient deep hedging strategies.
problem Hard training of optimal hedging strategies due to action dependence.
method Introduces no-transaction band network, a neural architecture.
result Demonstrates faster and more precise hedging strategies.
This study optimizes quantized neural networks by considering model architecture and quantization types.
problem Optimizing quantized neural networks for low-power, high-throughput applications.
method Holistic approach including training methods and quantization-friendly architecture design.
result Deeper models are more sensitive to activation quantization, while wider models improve resilience to both weight and activation quantization.
The study analyzes deep linear networks from random initialization, capturing dynamics and hyperparameter effects.
problem Understanding training dynamics in deep linear networks from random initialization.
method Theoretical analysis of gradient descent dynamics in deep linear networks with random initialization and large data.
result Captures the 'wider is better' effect and hyperparameter transfer effects, contrasting with neural-tangent parameterization.
A neural network and evolutionary algorithm framework designs nonlinear optical molecules.
problem Designing efficient nonlinear optical materials.
method Multi-stage Bayesian neural network (msBNN) and corrected Lewis-mode group contribution method (cLGC) combined with evolutionary algorithm (EA).
result Accurately and efficiently designs molecules with different optical properties using a small data set.
The problem of video frame prediction has received much interest due to its relevance to many computer vision applications such as autonomous vehicles or robotics. Supervised methods for video frame prediction rely on labeled data, which may not always be available. In this paper, we provide a novel unsupervised deep-l…
This paper introduces early exits in neural networks for faster inference.
problem Reducing inference time and preventing overfitting in neural networks.
method Designing and training multi-output neural networks with early exits.
result Significant reductions in inference time and improved robustness.
Recently, deep residual networks have been successfully applied in many computer vision and natural language processing tasks, pushing the state-of-the-art performance with deeper and wider architectures. In this work, we interpret deep residual networks as ordinary differential equations (ODEs), which have long been s…
A natural extension of Riemannian geometry to a much wider context is presented on the basis of the iterated differential form formalism developed in math.DG/0605113 and an application to general relativity is given.
Overparametrized neural networks can generalize well with proper regularization.
problem Generalization guarantee for noisy data in overparametrized neural networks.
method Nonparametric analysis of ℓ2-regularized GD trajectories. result Achieving minimax optimal rate of L2 estimation error with ℓ2 regularization.