SGD finds global optima in WGANs for 1-layer generators.
problem Training GANs with WGANs requires global optimality, which is hard.
method Used SGD to train 1-layer generator networks in WGANs.
result SGD converges to global solution in polynomial time and sample complexity.
We show new connections between adversarial learning and explainability for deep neural networks (DNNs). One form of explanation of the output of a neural network model in terms of its input features, is a vector of feature-attributions. Two desirable characteristics of an attribution-based explanation are: (1) $\texti…
This paper explains how deep learning performs hierarchical learning efficiently.
problem How deep learning can perform hierarchical learning efficiently.
method Backward feature correction principle and SGD training.
result Deep learning can efficiently train complex hierarchical tasks using SGD.
We generalize recent theoretical work on the minimal number of layers of narrow deep belief networks that can approximate any probability distribution on the states of their visible units arbitrarily well. We relax the setting of binary units (Sutskever and Hinton, 2008; Le Roux and Bengio, 2008, 2010; Montúfar and Ay,…
Fast Gradient Sign Method (FGSM) is a popular method to generate adversarial examples that make neural network models robust against perturbations. Despite its empirical success, its theoretical property is not well understood. This paper develops theory to explain the regularization effect of Generalized FGSM, a class…
Deep neural network models have recently achieved state-of-the-art performance gains in a variety of natural language processing (NLP) tasks (Young, Hazarika, Poria, & Cambria, 2017). However, these gains rely on the availability of large amounts of annotated examples, without which state-of-the-art performance is rare…
MCN improves deep neural networks by bettering local minima and generalizing well.
problem Bad local minima and poor generalization in deep neural networks.
method Introducing Maximum-and-Concatenation Networks (MCN) to eliminate bad local minima and improve generalization.
result MCN can autonomously improve local minima's goodness by increasing network depth.
Our work presents extensive empirical evidence that layer rotation, i.e. the evolution across training of the cosine distance between each layer's weight vector and its initialization, constitutes an impressively consistent indicator of generalization performance. In particular, larger cosine distances between final an…
A residual network (or ResNet) is a standard deep neural net architecture, with state-of-the-art performance across numerous applications. The main premise of ResNets is that they allow the training of each layer to focus on fitting just the residual of the previous layer's output and the target output. Thus, we should…
Deep GMRFs improve spatial data modeling and prediction.
problem Modeling spatial dependencies in data.
method Established connection between GMRFs and CNNs, allowing for multi-layer architectures.
result Deep GMRFs outperform state-of-the-art models in satellite temperature prediction.
Deep neural networks can grok better than shallow ones, showing multi-stage generalization.
problem Understanding the generalization behavior of deep neural networks.
method Empirical replication and analysis of grokking phenomenon in deep MLP models.
result Deep neural networks exhibit multi-stage generalization, with a secondary surge in test accuracy.
Convolutional networks struggle to learn Game of Life, even with lottery ticket weights.
problem Training convolutional networks to predict Conway's Game of Life is challenging.
method Examined small convolutional networks trained on Game of Life, focusing on weight initializations and network sizes.
result Minimal networks require significantly more parameters to converge, and their performance is sensitive to small changes in weights.
Often, more time is spent on finding a model that works well, rather than tuning the model and working directly with the dataset. Our research began as an attempt to improve upon a simple Recurrent Neural Network for answering "simple" first-order questions (QA-RNN), developed by Ferhan Ture and Oliver Jojic, from Comc…
Two networks are equivalent if they produce the same output for any given input. In this paper, we study the possibility of transforming a deep neural network to another network with a different number of units or layers, which can be either equivalent, a local exact approximation, or a global linear approximation of t…
New sparse penalty improves biclustering for gene expression data.
problem Identifying significant clusters in gene expression data.
method Prenet penalty applied to SSVD for biclustering.
result Mixed Prenet penalty effectively clusters non-overlapped data.
The paper proves universality in optimization problems with i.i.d. random vectors.
problem Optimization problems with i.i.d. random vectors and their projections.
method Proves universality of empirical risk minimization under specific conditions.
result The minimum value of the optimization problem is universal and depends only on the mean and covariance of the random vectors.
This work analyzes how overparameterization aids GANs in reaching global saddle points.
problem Understanding the role of overparameterization in GANs for convergence to global saddle points.
method Theoretical and empirical analysis of overparameterized GANs with various architectures and datasets.
result GDA converges to a global saddle point in overparameterized GANs with certain assumptions.
This paper improves deep forest models with soft routing and topology learning.
problem Expensive computational costs and lack of interpretability in deep neural networks.
method Soft routing in probabilistic trees and topology learning for joint optimization.
result Empowered deep forests achieve better performance with reduced model complexity.
Single-head transformers with a single self-attention layer can approximate any sequence-to-sequence function and are efficient under certain conditions.
problem Statistical and computational limits of prompt tuning for transformer-based models.
method Investigation of single-head transformers with a single self-attention layer, proving universality and efficiency under SETH.
result Existence of almost-linear time prompt tuning inference algorithms under certain conditions.
Study reveals neural scaling laws in random graphs and natural language models.
problem Understanding the origin of neural scaling laws in complex systems.
method Examined scaling laws in transformers trained on random walks and simplified natural language models.
result Neural scaling laws emerge in the absence of power law structure in data correlations.
RFN improves GCNs for road networks, outperforming state-of-the-art by 21%-40%.
problem Leveraging the structure of road networks effectively in machine learning tasks.
method Introducing RFN, a novel GCN specifically designed for road networks.
result RFN outperforms state-of-the-art GCNs by 21%-40% on road network tasks.
This survey clarifies dynamic network terminology and reviews GNN models for dynamic networks.
problem Ambiguity in dynamic network terminology and lack of GNN models for dynamic networks.
method Established consistent terminology and notation for dynamic networks, reviewed GNN models.
result Comprehensive survey of dynamic graph neural network models.
From the perspective of network analysis, the ubiquitous networks are comprised of regular and irregular components, which makes uncovering the complexity of network structures to be a fundamental challenge. Exploring the regular information and identifying the roles of microscopic elements in network data can help us …
DCNs mimic neuronal networks for improved neural classification.
problem Lack of topological similarity between DNNs and biological neural networks.
method Developed DCNs with topologies inspired by real-world neuronal networks.
result High classification accuracy achieved by DCNs.
Chemical networks outperform spiking neural networks in classification tasks.
problem Learning tasks with spiking neural networks require hidden layers, which are computationally expensive.
method Used deterministic mass-action kinetics to prove chemical reaction networks without hidden layers can solve tasks previously solved by spiking neural networks.
result A chemical reaction network without hidden layers outperforms a spiking neural network with hidden layers in a handwritten digit classification task.
Deep ReLU networks can be simplified to a three-layer model.
problem Understanding the behavior of deep neural networks.
method Constructive proof and algorithm to transform deep networks into shallow ones.
result Deep ReLU networks can be represented by a simpler three-layer structure.
Tackles network structure inference from time series data using GNN.
problem Inferring network structure from incomplete or no information.
method Gumbel Graph Network (GGN) model for network reconstruction and completion.
result GGN can reconstruct up to 100% network structure and infer missing parts with up to 90% accuracy.
Network embedding helps predict speed limits on incomplete Danish road network.
problem Incomplete speed limit data on Danish roads limits machine learning applications.
method Applied node2vec network embedding to Danish road network.
result Network embedding can derive useful features for predicting speed limits.
This paper explores loss landscapes of sparse neural networks, finding unique characteristics compared to dense networks.
problem Understanding the loss landscape of sparse neural networks, especially one-hidden-layer networks.
method Analyzes sparse networks with dense and sparse final layers, focusing on linear and non-linear models.
result Sparse networks can have no spurious valleys under certain conditions, but spurious valleys and minima can exist for wide sparse networks.
New approach learns latent motifs in networks for mesoscale structure analysis.
problem Understanding large-scale behavior in complex systems through mesoscale structures.
method Network dictionary learning (NDL) combining network sampling and nonnegative matrix factorization.
result Networks can be approximated using a small set of latent motifs.
A challenging problem in complex networks is the network reconstruction problem from data. This work deals with a class of networks denoted as conserved networks, in which a flow associated with every edge and the flows are conserved at all non-source and non-sink nodes. We propose a novel polynomial time algorithm to …
Social network analysis is an important problem in data mining. A fundamental step for analyzing social networks is to encode network data into low-dimensional representations, i.e., network embeddings, so that the network topology structure and other attribute information can be effectively preserved. Network represen…
Paper links network Lasso to network flow optimization.
problem Joint clustering and optimization of networked data.
method Exploration of duality between network Lasso and network flow optimization.
result nLasso is equivalent to a minimum-cost flow problem on the data network structure.
Taking inspiration from biological evolution, we explore the idea of "Can deep neural networks evolve naturally over successive generations into highly efficient deep neural networks?" by introducing the notion of synthesizing new highly efficient, yet powerful deep neural networks over successive generations via an ev…
SyNGLER generates synthetic networks efficiently while preserving key structural properties.
problem Efficiently generating realistic synthetic networks with preserved structural properties.
method SyNGLER uses latent space network models to learn and reconstruct node embeddings, then generates synthetic networks.
result SyNGLER produces synthetic networks that better preserve key network characteristics than existing approaches.
Secret neural networks hidden within trained models.
problem Excess capacity in neural networks allows embedding secret models.
method Novel framework for hiding secret neural networks within carrier networks.
result Detection of hidden networks is computationally infeasible.
Natural graph networks are a new class of graph neural networks that are more flexible and scalable.
problem Traditional graph neural networks are limited by equivariance to node permutations.
method Introduced natural graph networks, which are more flexible and scalable than conventional graph neural networks.
result Natural graph networks are as scalable as conventional message passing graph neural networks but more flexible.
The structure of complex networks has been of interest in many scientific and engineering disciplines over the decades. A number of studies in the field have been focused on finding the common properties among different kinds of networks such as heavy-tail degree distribution, small-worldness and modular structure and …
Convolutional networks outperform fully-connected ones in certain tasks.
problem Understanding the computational advantage of convolutional networks over fully-connected networks.
method Demonstrated a computational advantage through a specific problem class.
result Convolutional networks can solve certain problems that fully-connected networks cannot, even with gradient descent.
Proposes a graph neural network for traffic forecasting in WANs.
problem Traffic forecasting challenges in WANs due to dynamic and large data volumes.
method Dynamic diffusion convolutional recurrent neural networks for multistep traffic forecasting.
result Significant improvements in forecasting accuracy compared to classical methods.
Researchers derive exact priors for finite Bayesian neural networks.
problem Understanding non-Gaussian priors in finite Bayesian neural networks.
method Analytical derivation of function space priors for finite fully-connected feedforward networks.
result Exact solutions for priors of finite networks, including Meijer G-function for linear networks and mixtures for ReLU networks.
i-cNRL learns network differences with interpretability.
problem Comparing unique network characteristics.
method Contrastive network representation learning (cNRL) integrating machine learning schemes.
result i-cNRL reveals unique network patterns with interpretability.
Deep and wide networks are shown to be equivalent in terms of their capability.
problem The relationship between the width and depth of neural networks.
method Formulated transforms to map networks, used polynomial representations.
result Deep and wide networks are quasi-equivalent with an arbitrarily small error.
We show that deep networks are better than shallow networks at approximating functions that can be expressed as a composition of functions described by a directed acyclic graph, because the deep networks can be designed to have the same compositional structure, while a shallow network cannot exploit this knowledge. Thu…
DCGANs generate drainage networks quickly from samples.
problem High computational costs in generating large numbers of drainage networks.
method DCGANs trained with connectivity-informed directional information.
result Connectivity-informed DCGANs outperform other methods in reproducing accurate drainage networks.
A network embedding consists of a vector representation for each node in the network. Its usefulness has been shown in many real-world application domains, such as social networks and web networks. Directed networks with text associated with each node, such as software package dependency networks, are commonplace. Howe…
Quantum neural network and tensor network models outperform classical models in Japanese stock market predictions.
problem Improving stock return predictions using quantum and quantum-inspired machine learning.
method Evaluation of quantum neural network and tensor network models against classical models like linear and neural networks.
result Tensor network model outperforms classical models in Japanese stock market, including linear and neural network models.
Study deep maxout networks and their equivalence to Gaussian processes.
problem Understanding neural networks with infinite width.
method Derive equivalence between deep maxout networks and Gaussian processes, characterize maxout kernel, and provide efficient numerical implementation.
result Bayesian inference based on deep maxout network kernel leads to competitive results compared to finite-width counterparts and deep neural network kernels.