Study on functions computed by deep-layered machines finds same distribution in neural networks and Boolean circuits.
problem Understanding the space of functions computed by deep-layered machines.
method Investigation of Boolean functions on random-layered machines, including neural networks and Boolean circuits.
result The space of functions computed at large depth limit is characterized and the macroscopic entropy of Boolean functions is either monotonically increasing or decreasing with depth.
Paper interprets ResNets via gate-network controls and deep-layer classifications.
problem Understanding the performance mechanism of ResNets.
method Constructs typical solutions using gate-network controls and deep-layer classifications.
result Proves the universal-approximation capability of ResNets.
Deep learning depends on tuning layers near critical points.
problem Understanding how deep learning architectures depend on tuning parameters.
method Random energy approach to analyze statistical dependence in deep belief networks.
result Statistical dependence can propagate only if layers are tuned near critical points.
Recently, the deep-belief-networks (DBN) based voice activity detection (VAD) has been proposed. It is powerful in fusing the advantages of multiple features, and achieves the state-of-the-art performance. However, the deep layers of the DBN-based VAD do not show an apparent superiority to the shallower layers. In this…
Deep learning is a form of machine learning for nonlinear high dimensional pattern matching and prediction. By taking a Bayesian probabilistic perspective, we provide a number of insights into more efficient algorithms for optimisation and hyper-parameter tuning. Traditional high-dimensional data reduction techniques, …
Neural machine learning methods, such as deep neural networks (DNN), have achieved remarkable success in a number of complex data processing tasks. These methods have arguably had their strongest impact on tasks such as image and audio processing - data processing domains in which humans have long held clear advantages…
Federated learning obtains a central model on the server by aggregating models trained locally on clients. As a result, federated learning does not require clients to upload their data to the server, thereby preserving the data privacy of the clients. One challenge in federated learning is to reduce the client-server c…
In recent years, deep learning methods applying unsupervised learning to train deep layers of neural networks have achieved remarkable results in numerous fields. In the past, many genetic algorithms based methods have been successfully applied to training neural networks. In this paper, we extend previous work and pro…
Theory explains how deep nets learn features from data.
problem Understanding how deep neural networks learn features from data.
method Developed a noise-nonlinearity phase diagram and a mechanical theory.
result Links feature learning across layers to generalization.
This paper investigates the influence of different acoustic features, audio-events based features and automatic speech translation based lexical features in complex emotion recognition such as curiosity. Pretrained networks, namely, AudioSet Net, VoxCeleb Net and Deep Speech Net trained extensively for different speech…
We propose a new notion of `non-linearity' of a network layer with respect to an input batch that is based on its proximity to a linear system, which is reflected in the non-negative rank of the activation matrix. We measure this non-linearity by applying non-negative factorization to the activation matrix. Considering…
LogDet estimator improves entropy estimation in neural networks.
problem Inconsistent observations and diversified interpretation in neural networks.
method Proposes LogDet estimator for reliable entropy approximation.
result LogDet estimator overcomes distributional diversity issues.
RevDEQs improve performance on tasks with exact gradients and fewer function evaluations.
problem Inexact gradient calculation in DEQs leads to unstable training and requires regularisation or many function evaluations.
method Introduce Reversible Deep Equilibrium Models (RevDEQs) that allow for exact gradient calculation, no regularisation, and far fewer function evaluations.
result RevDEQs significantly improve performance on language modelling and image classification tasks.
We present trellis networks, a new architecture for sequence modeling. On the one hand, a trellis network is a temporal convolutional network with special structure, characterized by weight tying across depth and direct injection of the input into deep layers. On the other hand, we show that truncated recurrent network…
Supervised learning with large scale labeled datasets and deep layered models has made a paradigm shift in diverse areas in learning and recognition. However, this approach still suffers generalization issues under the presence of a domain shift between the training and the test data distribution. In this regard, unsup…
Layer-wise networks have a closed-form solution and a stopping criterion.
problem Training networks one layer at a time without backpropagation.
method Proved the Kernel Mean Embedding as the closed-form solution and developed a stopping criterion.
result Layer-wise networks converge to a highly desirable kernel for classification.
Layer-wise networks have a closed-form solution and a stopping criterion.
problem Training networks one layer at a time without backpropagation.
method Proved the closed-form solution using the kernel Mean Embedding and Neural Indicator Kernel.
result Layer-wise networks have a closed-form solution and a stopping criterion.
In this paper we develop a statistical theory and an implementation of deep learning models. We show that an elegant variable splitting scheme for the alternating direction method of multipliers optimises a deep learning objective. We allow for non-smooth non-convex regularisation penalties to induce sparsity in parame…
The paper demonstrates the use of the fully convolutional neural network for glioma segmentation on the BraTS 2019 dataset. Three-layers deep encoder-decoder architecture is used along with dense connection at encoder part to propagate the information from coarse layer to deep layers. This architecture is used to train…
DeeperGCN tackles deep GCNs by overcoming vanishing gradient and over-smoothing issues.
problem Vanishing gradient, over-smoothing, and over-fitting issues in deep GCNs.
method DeeperGCN uses differentiable generalized aggregation functions and a novel normalization layer (MsgNorm) to train deep GCNs.
result DeeperGCN significantly boosts performance on large-scale graph learning tasks.
A practical limitation of deep neural networks is their high degree of specialization to a single task and visual domain. Recently, inspired by the successes of transfer learning, several authors have proposed to learn instead universal, fixed feature extractors that, used as the first stage of any deep network, work w…
A new method generates natural-looking adversarial examples by bounding internal activation values.
problem Creating natural-looking adversarial examples that closely mimic the original input.
method Bounding internal activation values through a distribution quantile bound and polynomial barrier loss function.
result Our attack achieves similar success and confidence levels as state-of-the-art methods but with more natural-looking perturbations.
New approach to abstract neural network representations using renormalization group.
problem Developing truly abstract representations in neural networks.
method Renormalization group approach to expand representations to encompass broader data sets.
result Representations in neural networks become more abstract as data breadth increases and depth increases.
Graph transformers outperform graph convolutions by preserving community information.
problem Understanding why graph transformers perform well in node-level prediction tasks.
method Analyzing the Gaussian process limits of graph transformers with infinite width and infinite heads.
result Graph transformers maintain discriminative node representations even in deep layers, preventing oversmoothing.
The success of deep neural networks has inspired many to wonder whether other learners could benefit from deep, layered architectures. We present a general framework called forward thinking for deep learning that generalizes the architectural flexibility and sophistication of deep neural networks while also allowing fo…
CAggNet improves medical image segmentation by fusing coarse and fine features.
problem Medical image segmentation accuracy and efficiency.
method Crossing Aggregation Network with nested skip connections and weighted aggregation.
result CAggNet achieves more accurate and efficient segmentation compared to existing methods.
The paper tackles decision making problems with funnel structure in email marketing campaigns.
problem Decision making challenges in systems with funnel structure, where fewer observations are received from deeper layers.
method Formulated as a contextual bandit with funnel structure and developed a multi-task learning algorithm.
result Our algorithms offer significant improvement over previous methods in email marketing campaigns.
Localization of chest pathologies in chest X-ray images is a challenging task because of their varying sizes and appearances. We propose a novel weakly supervised method to localize chest pathologies using class aware deep multiscale feature learning. Our method leverages intermediate feature maps from CNN layers at di…
SINF models transform arbitrary PDFs to target PDFs using 1D slices.
problem Transforming arbitrary probability distributions to target distributions efficiently.
method Iterative Optimal Transport of 1D slices, maximizing Wasserstein distance.
result SINF models generate high-quality samples and competitive density estimates.
Statistical neurodynamics studies macroscopic behaviors of randomly connected neural networks. We consider a deep layered feedforward network where input signals are processed layer by layer. The manifold of input signals is embedded in a higher dimensional manifold of the next layer as a curved submanifold, provided t…
EigenGAN discovers interpretable dimensions in GAN layers for semantic control.
problem Lack of explicit dimensions to control semantic attributes in GAN layers.
method EigenGAN embeds linear subspaces with orthogonal bases into each generator layer, learning eigen-dimensions corresponding to semantic attributes via adversarial training.
result EigenGAN can produce samples with continuous changes corresponding to specific semantic attributes.
TENP prunes experts and neurons in Mixture-of-Experts models for efficient deployment.
problem Efficient deployment of large language models constrained by static parameter footprint.
method Structured Trapezoidal ExpertNeuron Pruning (TENP) identifies and retains important experts and neurons.
result DeepSeek model achieves 10% better performance on code generation tasks with 40% expert sparsity.
This study investigates how much knowledge from natural images can be transferred to pathology images.
problem Quantifying how much knowledge from natural images can be transferred to pathology images.
method Proposes a framework to quantify knowledge gain by a particular layer, conducts empirical investigation in pathology image centered transfer learning.
result Early layers of deep models can transfer knowledge to pathology image classification tasks.
ReduNet optimizes data compression by maximizing rate reduction in deep networks.
problem Optimizing deep networks for high-dimensional multi-class data.
method Maximizing rate reduction through iterative gradient ascent, leading to a multi-layer deep network.
result ReduNet achieves optimal linear discriminative representation and is more efficient in the spectral domain.
The deep layers of modern neural networks extract a rather rich set of features as an input propagates through the network. This paper sets out to harvest these rich intermediate representations for quantization with minimal accuracy loss while significantly reducing the memory footprint and compute intensity of the DN…
Feedback alignment methods need to be evaluated for accuracy and gradient cosine similarity.
problem Evaluating feedback alignment methods
method Proposed diagnostic evaluation protocol
result Identified silent failures in standard reporting pair
In a physical neural system, where storage and processing are intimately intertwined, the rules for adjusting the synaptic weights can only depend on variables that are available locally, such as the activity of the pre- and post-synaptic neurons, resulting in local learning rules. A systematic framework for studying t…
A study on the depth of graph neural networks on sparse graphs, revealing a dichotomy based on the Kesten-Stigum ratio.
problem Determining the optimal depth of graph neural networks for sparse graphs.
method Analyzing the sparse contextual stochastic block model with a message-passing classifier.
result The value of depth is governed by the Kesten-Stigum ratio, with thresholds dividing performance into geometric and branching processes.
Reservoir Memory Machines solve benchmark tasks faster than Neural Turing Machines.
problem Training Neural Turing Machines is hard and limits their applicability.
method Proposes Reservoir Memory Machines, combining neural network flexibility with Turing machine capabilities, but with faster training via alignment and linear regression.
result Reservoir Memory Machines solve benchmark tasks as well as Neural Turing Machines but are much faster to train.
The current processes for building machine learning systems require practitioners with deep knowledge of machine learning. This significantly limits the number of machine learning systems that can be created and has led to a mismatch between the demand for machine learning systems and the ability for organizations to b…
We generalize a support vector machine to a support spinor machine by using the mathematical structure of wedge product over vector machine in order to extend field from vector field to spinor field. The separated hyperplane is extended to Kolmogorov space in time series data which allow us to extend a structure of sup…
The paper reviews machine learning safety techniques for autonomous vehicles.
problem Challenges in machine learning safety for autonomous vehicles.
method Organizes practical safety techniques to complement engineering safety.
result Enhances dependability and safety of machine learning algorithms in autonomous vehicles.
The preceding paper constructed tangle machines as diagrammatic models, and illustrated their utility with a number of examples. The information content of a tangle machine is contained in characteristic quantities associated to equivalence classes of tangle machines, which are called invariants. This paper constructs …
Application of fuzzy support vector machine in stock price forecast. Support vector machine is a new type of machine learning method proposed in 1990s. It can deal with classification and regression problems very successfully. Due to the excellent learning performance of support vector machine, the technology has becom…
RBM and DBM are represented as 2D tensor networks, revealing their expressive power and efficiency.
problem Understanding and optimizing RBM and DBM models.
method Representing RBM and DBM as 2D tensor networks and developing an efficient tensor network contraction algorithm.
result The proposed algorithm for computing partition functions is more accurate than state-of-the-art methods.
We conduct an empirical study of machine learning functionalities provided by major cloud service providers, which we call machine learning clouds. Machine learning clouds hold the promise of hiding all the sophistication of running large-scale machine learning: Instead of specifying how to run a machine learning task,…
Market incentivizes parties to share high-quality data for collaborative machine learning tasks.
problem Fair revenue distribution and data replication threats in collaborative machine learning markets.
method Introduces a novel payment division function robust to replication and customized output models.
result Validated assumptions and showed approximate satisfaction for commonly used models.
A new machine learning model uses score matching to estimate probability densities efficiently.
problem Estimating probability density functions is challenging.
method Introduced a product Jacobi-Theta Boltzmann machine (pJTBM) and used score matching for efficient fitting.
result The pJTBM can fit probability densities more efficiently than the RTBM using score matching.