ConvNets can be translated into CKNs that perform similarly.
problem The distinction between ConvNets and kernel-based methods.
method Translation of ConvNets into CKNs using a new gradient algorithm.
result CKNs perform as well as ConvNets, supporting the translation.
Graph convolutional kernel networks generalize CNNs to graph data.
problem Representing graph-structured data for machine learning.
method Convolutional kernel networks applied to graph data.
result Competitive performance on graph classification benchmarks.
Proposes a method to constrain singular values of convolutional kernels in neural networks.
problem Avoiding exploding/vanishing gradient problems and improving generalizability in neural networks.
method Introduces a penalty function to constrain singular values of convolutional kernels around 1, and derives an algorithm for optimization.
result Demonstrates the effectiveness of the method through numerical examples.
VC dimensions of group CNNs are infinite for certain kernels and groups.
problem Estimating the generalization capacity of group convolutional neural networks.
method Identifying precise VC dimension estimates for simple sets of group CNNs.
result Two-parameter families of convolutional neural networks have an infinite VC dimension for infinite groups and certain kernels.
We applied pre-defined kernels also known as filters or masks developed for image processing to convolution neural network. Instead of letting neural networks find its own kernels, we used 41 different general-purpose kernels of blurring, edge detecting, sharpening, discrete cosine transformation, etc. for the first la…
Enhanced LSTM with multiple kernels and attention improves video action recognition.
problem Improving motion understanding in video analysis.
method Proposed a Network-in-LSTM approach with multiple convolutional kernels and layers, and an attention-based mechanism.
result Improves accuracy in supervised classification on UCF-101 and Sports-1M datasets.
New method bounds singular values of convolutional kernels to stabilize gradients.
problem Stable gradients in convolutional neural networks.
method Frobenius norm regularization for convolutional kernels.
result Bounded singular values of transformation matrices.
Deep convolutional networks can be understood through kernel methods, providing insights into their inductive bias.
problem Understanding the functional space and inductive bias of deep convolutional networks.
method Using kernel methods to analyze simple hierarchical kernels with convolution and pooling layers.
result The RKHS consists of additive models of interaction terms between patches, and pooling layers encourage spatial similarities.
Convolution and pooling improve kernel methods in image classification.
problem Understanding the interplay between approximation and generalization in convolutional architectures.
method Characterized RKHS of kernels with convolution, pooling, and downsampling, computed generalization error.
result Convolution and pooling operations trade off approximation with generalization power.
Kernel Quantization improves CNN compression without sacrificing performance.
problem Efficiently compressing CNN models without significant performance loss.
method Quantizes convolution kernels as the unit, learning a codebook for low-bit indexes.
result Significant compression ratio achieved with minimal accuracy loss.
A new convolutional spectral kernel network learns hierarchical and local features.
problem Lack of deep learning in non-stationary spectral kernels.
method Introduces convolutional filters and deep architectures into non-stationary spectral kernels, derives generalization error bounds, and introduces regularizers.
result Validated the effectiveness of the convolutional spectral kernel network on real-world datasets.
We describe convolutional networks using harmonic functions.
problem Understanding the function space and smoothness of convolutional networks.
method Using reproducing kernel Hilbert spaces and functional ANOVA decomposition.
result Convolutional networks can be decomposed into a sum of elementary functions.
Proposes flexible dilation networks for better time series analysis.
problem Fixed dilation limits flexibility in time series analysis.
method End-to-end learnable dilation layers and independent kernels.
result Improves efficiency and flexibility in training.
Smooth kernel regularizer improves deep neural networks' performance with less data.
problem Deep neural networks need large datasets for effective learning.
method Proposes a smooth kernel regularizer that encourages spatial correlations in convolution kernel weights, learned from previous experience.
result The smooth kernel regularizer improves visual recognition models over an L2 regularization baseline.
ROCKET speeds up time series classification without sacrificing accuracy.
problem High computational complexity and intractability of existing time series classification methods.
method Simple linear classifiers using random convolutional kernels.
result Achieves state-of-the-art accuracy with significantly reduced computational expense.
In this paper, we introduce a new image representation based on a multilayer kernel machine. Unlike traditional kernel methods where data representation is decoupled from the prediction task, we learn how to shape the kernel with supervision. We proceed by first proposing improvements of the recently-introduced convolu…
Convolutional DKMs improve kernel methods on MNIST, CIFAR-10, and CIFAR-100.
problem Improving kernel methods for image classification.
method Developed a novel inter-domain inducing point approximation and introduced various techniques to extend DKMs to convolutional networks.
result Achieved state-of-the-art performance on image classification benchmarks.
HKConv learns hyperbolic features by aggregating kernel points.
problem Challenges in learning good hyperbolic representations using Euclidean operations.
method Proposes HKConv, a trainable hyperbolic convolution that correlates local features with kernel points and aggregates them.
result HKConv learns expressive local features according to hyperbolic geometry and enjoys equivariance to permutation and invariance to parallel transport.
In this paper we study the problem of learning the weights of a deep convolutional neural network. We consider a network where convolutions are carried out over non-overlapping patches with a single kernel in each layer. We develop an algorithm for simultaneously learning all the kernels from the training data. Our app…
We show that the output of a (residual) convolutional neural network (CNN) with an appropriate prior over the weights and biases is a Gaussian process (GP) in the limit of infinitely many convolutional filters, extending similar results for dense networks. For a CNN, the equivalent kernel can be computed exactly and, u…
The paper proposes an ensemble of convolution-based methods for fault detection in gearboxes.
problem Fault detection in planetary gearboxes using vibration signals.
method Ensemble of three convolution kernel-based methods (ROCKET, 1D CNN with ResNet, FCN).
result Outperforms other approaches with over 98.8% accuracy.
Study shows deterministic equivalent for neural network kernel convergence.
problem Understanding convergence of neural network kernels.
method Analyzes empirical spectral distribution of Conjugate Kernel, proving convergence to a deterministic limit.
result Obtains a deterministic equivalent for the Stieltjes transform and resolvent of the Conjugate Kernel.
Empirical study shows standard CNNs deviate from NTK predictions.
problem Understanding how standard finite-width CNNs behave compared to their infinite-width NTK counterparts.
method Empirical analysis of AlexNet and LeNet architectures.
result Standard CNNs deviate significantly from their NTK counterparts, but deviation decreases with wider networks.
Study the inductive bias of neural networks using neural tangent kernels.
problem Understanding the generalization properties of over-parameterized neural networks.
method Analysis of the neural tangent kernel and its corresponding function space (RKHS).
result Stability properties of functions with finite norm, including stability to image deformations in convolutional networks.
New mechanism discovered for feature learning in CNNs.
problem Understanding how CNNs learn features from images.
method Proposed Convolutional Neural Feature Ansatz linking filter covariances to patch-based AGOPs.
result Deep ConvRFM algorithm learns features similar to deep CNNs, improving performance.
Symmetric CNNs improve sequential recommendation and protein structure prediction.
problem Improving prediction accuracy in sequential recommendation and protein structure inference.
method Developed a CNN architecture that preserves symmetry in convolutional layers, using parameterized convolutional kernels.
result Symmetric structured CNNs achieve better performance with fewer parameters.
New kernel-based models improve on traditional neural methods in sequence modeling.
problem Sequence modeling challenges in natural language processing and neuroscience.
method Kernel-based recurrent neural networks and convolutional neural networks.
result Kernel-based models perform on par or better than traditional neural methods.
The success of deep convolutional architectures is often attributed in part to their ability to learn multiscale and invariant representations of natural signals. However, a precise study of these properties and how they affect learning guarantees is still missing. In this paper, we consider deep convolutional represen…
End-to-end graph SVM with graph convolutions and RKHS.
problem Graph classification with complex feature spaces.
method End-to-end training of graph convolutions, kernel function, and SVM parameters.
result Outperforms existing deep learning models on graph classification tasks.
Locality helps in learning from high-dimensional data.
problem Understanding how convolutional neural networks learn from high-dimensional data.
method Teacher-student framework for kernel regression with convolutional kernels.
result Locality is key to determining the learning curve exponent in high-dimensional data.
Although the convolutional neural networks (CNNs) have become popular for various image processing and computer vision task recently, it remains a challenging problem to reduce the storage cost of the parameters for resource-limited platforms. In the previous studies, tensor decomposition (TD) has achieved promising co…
Network deconvolution removes redundant data to improve neural network performance.
problem Redundant data in neural networks makes training challenging.
method Network deconvolution optimally removes pixel-wise and channel-wise correlations before each layer.
result Network deconvolution improves performance across various datasets.
New filter bank regularization improves DCNNs by incorporating image priors.
problem Improving DCNNs' robustness and generality.
method Structured filter bank regularization of DCNN kernels.
result Filter bank regularization leads to faster convergence and better generalization.
Unified theory for adaptive image convolutions using metric perspectives.
problem Fixed kernels in convolutions limit adaptability in image processing.
method Metric perspective on images as 2D manifolds with local distances, proposing metric convolutions.
result Metric convolutions provide better generalisation and competitive performance.
Graph convolutional deep kernel machine learns representations for graph tasks.
problem Limited representation learning in infinite-width neural networks.
method Developed a graph convolutional deep kernel machine as an infinite-width limit.
result Representation learning improves performance for heterophilous node classification tasks.
A CNN-based method detects and counts corn kernels from images.
problem Manual counting of corn kernels is labor-intensive and prone to error.
method Sliding window approach with CNN for detection and NMS for overlapping removal.
result The method successfully detects and counts kernels with low error.
Efficiently learns 3D convolutions with less data.
problem High parameter and data costs in 3D convolutions.
method Temporal factorization of 3D kernels.
result Significantly reduces training data requirement and parameter count.
TextCNN, the convolutional neural network for text, is a useful deep learning algorithm for sentence classification tasks such as sentiment analysis and question classification. However, neural networks have long been known as black boxes because interpreting them is a challenging task. Researchers have developed sever…
New IP analysis for deep neural networks using Rényi's entropy and tensor kernels.
problem Estimating mutual information in high-dimensional hidden layers of deep neural networks.
method Matrix-based Rényi's entropy coupled with tensor kernels for convolutional layers.
result First comprehensive IP analysis of large-scale DNNs and CNNs.
The paper verifies deep neural networks' ability to approximate functions on spheres.
problem Theoretical verification of deep neural networks' performance on spherical functions.
method Spherical analysis using reproducing kernels and convolutional factorizations.
result Rates of uniform approximation for functions in Sobolev spaces and additive ridge forms.
In this paper, we firstly introduce a method to efficiently implement large-scale high-dimensional convolution with realistic memristor-based circuit components. An experiment verified simulator is adapted for accurate prediction of analog crossbar behavior. An improved conversion algorithm is developed to convert conv…
KM method reduces ConvNet parameters to 9% higher accuracy with minimal additional memory.
problem Expensive memory usage for training ConvNets on embedded devices.
method Kernel Modulation (KM) method that adapts all network parameters for each task.
result KM delivers up to 9% higher accuracy than other parameter-efficient methods.
AutoQ automatically optimizes quantization for CNNs, reducing latency and energy.
problem Efficiently quantizing CNN weights for low-power mobile devices.
method Hierarchical-DRL for kernel-wise quantization bitwidth selection.
result Reduces inference latency and energy consumption by 54.06% and 50.69% respectively.
Overview of structured data representation methods.
problem Structured data lacks vectorial form, complicating machine learning.
method Various approaches including kernel, distance, neural networks, and graph convolutional networks.
result New approaches like metric learning and recurrent decoder networks have emerged.
Enhanced CNN kernels improve image classification accuracy.
problem Improving classification accuracy of CNNs.
method Local Average Pooling and random image patch representation.
result 89% accuracy on CIFAR-10, matching AlexNet performance.
The paper analyzes deep ReLU CNNs' approximation properties in 2D space.
problem Establishing L2 approximation properties for deep ReLU CNNs. method Analysis based on decomposition theorem for convolutional kernels, properties of ReLU activation, and connections with one-hidden-layer ReLU NNs.
result Universal approximation theorem for deep ReLU CNNs with classic structure.
This paper explains GCNs using NTKs and improves their performance.
problem GCNs' performance degrades with depth, and skip connections marginally improve it.
method Derive NTKs for GCNs, validate with simulations, propose NTK as a surrogate model.
result Suitable normalisation can prevent drastic performance drop with depth.
Real time application of deep learning algorithms is often hindered by high computational complexity and frequent memory accesses. Network pruning is a promising technique to solve this problem. However, pruning usually results in irregular network connections that not only demand extra representation efforts but also …