DREAM model improves computational efficiency for non-linear effects in relational event models.
problem Efficiently modeling non-linear effects in dynamic relational networks.
method Introduces Deep Relational Event Additive Model (DREAM) using Neural Additive Models.
result Demonstrates superior computational efficiency compared to traditional REM approaches.
A new complexity measure for neural networks improves upon classical methods.
problem Lack of a refined complexity measure for comparing different neural network architectures, especially permutation-invariant ones.
method Introduced an equivalence relation among linear functions and counted them relative to this relation.
result The new complexity measure clearly distinguishes between different models and increases exponentially with depth.
We show that gradient descent on full-width linear convolutional networks of depth L converges to a linear predictor related to the ℓ2/L bridge penalty in the frequency domain. This is in contrast to linearly fully connected networks, where gradient descent converges to the hard margin linear support vector m…
We study alignment in linear neural networks and its relation to gradient descent.
problem Understanding alignment in linear neural networks and its impact on training.
method Defined alignment for fully connected networks, analyzed alignment under gradient descent, and compared gradient descent to projected gradient descent for layer-constrained networks.
result Gradient descent can converge linearly to a global minimum when alignment is invariant, and alignment is impossible with large datasets in layer-constrained networks.
Diagonal linear networks converge to lasso regularization path during training.
problem Understanding the regularization behavior of diagonal linear networks.
method Analyzing the training trajectory of diagonal linear networks and comparing it to the lasso regularization path.
result The training trajectory of diagonal linear networks is closely related to the lasso regularization path.
New neural network models for complex functional data analysis.
problem Complex relations between functional predictors and responses.
method Function-on-Function regression models using neural networks with continuous hidden layers.
result Demonstrated power and flexibility in handling complex functional models.
Due to the success of residual networks (resnets) and related architectures, shortcut connections have quickly become standard tools for building convolutional neural networks. The explanations in the literature for the apparent effectiveness of shortcuts are varied and often contradictory. We hypothesize that shortcut…
Noise in linear networks minimizes sharpness and leads to shrinkage-thresholding.
problem Minimizing sharpness in diagonal linear networks.
method Stochastic sharpness-aware minimization (SAM) with isotropic noise.
result Noise forces shrinkage-thresholding of true parameters.
Study L2 regularization in deep networks, uncovering performance relations and proposing a training schedule.
problem Understanding and optimizing L2 regularization in deep learning models. method Empirical observations and theoretical analysis of gradient flow dynamics in infinitely wide networks.
result Empirical relations between model performance, L2 coefficient, learning rate, and training steps; optimal regularization parameter prediction; improved training schedule. Relational learning deals with data that are characterized by relational structures. An important task is collective classification, which is to jointly classify networked objects. While it holds a great promise to produce a better accuracy than non-collective classifiers, collective classification is computational cha…
Paper tackles multi-object reinforcement learning, improving skill extrapolation.
problem Learning robust manipulation tasks in multi-object settings.
method Introduces a linear relation network module to enhance skill generalization.
result Agents can extrapolate and generalize to any new object number, scaling linearly.
Learning influence pathways of a network of dynamically related processes from observations is of considerable importance in many disciplines. In this article, influence networks of agents which interact dynamically via linear dependencies are considered. An algorithm for the reconstruction of the topology of interacti…
New measure LMN explains neural network grokking.
problem Delayed generalization after memorization in neural networks.
method Defined LMN to measure network complexity, showing LMN correlates with test losses linearly.
result LMN reveals intriguing XOR network behavior and is a promising complexity measure.
Linear RNNs exhibit a bias towards shorter memory due to initialization variance.
problem Understanding the performance limitations of RNNs, especially linear ones.
method Kernel regime analysis to show equivalence to 1D-convolutional networks and analyze weightings.
result Linear RNNs with random initialization have a bias towards shorter memory periods.
Flaky performance found in GNN SSL on RDBs, leading to worse linear evaluation.
problem Downstream task performances of GNN SSL on RDBs are poor.
method Proposed InfoNode to maximize mutual information between initial and final node representations.
result InfoNode improves GNN SSL performance on RDBs, supporting conjecture of conflict between SSL and GNN message passing.
Regularization leads to balancedness in deep linear networks.
problem Balancedness in deep linear networks.
method Geometric invariant theory and Riemannian geometry of fibers.
result Balancing flows converge to the balanced manifold at a uniform exponential rate.
The study tests inferences about neural network optimization from linear interpolation of loss landscapes.
problem Understanding the difficulty of neural network optimization problems.
method Linear interpolation of neural network loss landscapes, systematic evaluation of various factors.
result Linear interpolation does not correlate with model performance, challenging prior intuition.
Improved electricity price forecasting model combining linear and non-linear structures.
problem Day-ahead electricity price forecasting in energy systems.
method Recurrent neural networks with embedded linear structures.
result Approximately 11% higher accuracy than state-of-the-art models.
We present Spectral Inference Networks, a framework for learning eigenfunctions of linear operators by stochastic optimization. Spectral Inference Networks generalize Slow Feature Analysis to generic symmetric operators, and are closely related to Variational Monte Carlo methods from computational physics. As such, the…
DANCE optimizes neural network and accelerator design for faster, more efficient DNN execution.
problem Challenges in optimizing neural network and accelerator design for efficient DNN execution.
method Differentiable approach to co-exploration of accelerator and network architecture design.
result Significantly shorter time to achieve superior accuracy and hardware cost metrics.
Study of Bayes optimal learning in high-dimensional linear regression with network side information.
problem Bayes optimal learning in high-dimensional linear regression with network side information.
method Introduce a Reg-Graph model and an iterative AMP algorithm for Bayes optimality under general conditions.
result Characterization of the limiting mutual information between latent signal and data observed.
New deep learning model for matrix completion combining linear and nonlinear relationships.
problem Matrix completion considering only linear or nonlinear relations, ignoring latent relationships.
method Combines linear and nonlinear models in a latent variables framework, using a deep neural network with two branches for columns and rows, and manifold learning as an auxiliary task.
result Experimental results show the proposed method outperforms state-of-the-art matrix completion methods.
The era of data deluge has sparked the interest in graph-based learning methods in a number of disciplines such as sociology, biology, neuroscience, or engineering. In this paper, we introduce a graph recurrent neural network (GRNN) for scalable semi-supervised learning from multi-relational data. Key aspects of the no…
The monotonic linear interpolation in deep networks often leads to plateaus, revealing biases in optimization.
problem Plateaus in the optimization landscape of deep networks during monotonic linear interpolation.
method Investigated monotonic linear interpolation on deep neural networks, focusing on biases in weights and biases.
result Interpolating weights and biases differently can lead to significant differences in loss and accuracy, revealing biases in optimization.
Given two networks with the same training loss on a dataset, when would they have drastically different test losses and errors? Better understanding of this question of generalization may improve practical applications of deep networks. In this paper we show that with cross-entropy loss it is surprisingly simple to ind…
The relational model is a ubiquitous representation of big-data, in part due to its extensive use in databases. In this paper, we propose the Equivariant Entity-Relationship Network (EERN), which is a Multilayer Perceptron equivariant to the symmetry transformations of the Entity-Relationship model. To this end, we ide…
The paper analyzes financial networks with default charges and defines a model using fixpoint problems.
problem Modeling systemic risk in interbank networks with crossholdings and default charges.
method Mixed integer-linear programming and Gaussian elimination algorithm for computing clearing pairs.
result Developed methods to compute maximal and minimal clearing pairs.
Variational auto-encoder frameworks have demonstrated success in reducing complex nonlinear dynamics in molecular simulation to a single non-linear embedding. In this work, we illustrate how this non-linear latent embedding can be used as a collective variable for enhanced sampling, and present a simple modification th…
New PEMs improve network inference from time-series data.
problem Causal inference from time-series data with trade-off between accuracy and feasibility.
method Infer networks via process motifs for lagged correlation in linear stochastic processes.
result Proposed PEMs achieve high accuracy and efficiency in network inference.
This work proposes a mathematical framework for loss landscapes and optimization in deep neural networks.
problem The effectiveness of gradient-based optimization in over-parameterized neural networks.
method A modern view and mathematical framework of loss landscapes and efficient optimization in over-parameterized machine learning models.
result Wide neural networks satisfy the PL∗ condition, explaining (S)GD convergence to a global minimum. Recent studies on the adversarial vulnerability of neural networks have shown that models trained to be more robust to adversarial attacks exhibit more interpretable saliency maps than their non-robust counterparts. We aim to quantify this behavior by considering the alignment between input image and saliency map. We h…
Gradient descent proves global convergence for deep networks with a single wide layer.
problem Proving global convergence of gradient descent for deep ReLU networks.
method Simplified proof using a single wide layer, leveraging ReLU's Lipschitz property.
result Gradient descent converges globally for networks with a single wide layer.
SGD trains ReLU networks to implement piecewise linear maps with at most 3 knot points.
problem Understanding the training dynamics of neural networks trained via SGD.
method Mean-field analysis of a two-layer ReLU network trained via SGD for a univariate regression problem.
result At convergence, SGD-trained ReLU networks implement piecewise linear maps with at most 3 knot points.
Bayesian optimization (BO) is a model-based approach for gradient-free black-box function optimization. Typically, BO is powered by a Gaussian process (GP), whose algorithmic complexity is cubic in the number of evaluations. Hence, GP-based BO cannot leverage large amounts of past or related function evaluations, for e…
Global balance index measures systemic risk in financial networks.
problem Measuring systemic risk in financial networks.
method Defined global balance index based on a diffusive process and linear system.
result Global balance index correlates with systemic risk measures.
Classifies surfaces with special curvature properties.
problem Rotational surfaces with specific curvature conditions.
method Classifies surfaces with rotationally symmetric norms and linear curvature relations.
result Rotational surfaces with linearly related curvatures are classified.
The study examines Fisher information matrices and neural tangent kernels for simple ReLU networks with random weights.
problem Understanding the relationship between Fisher information matrices and neural tangent kernels for 2-layer ReLU networks.
method Analyzes Fisher information matrices and neural tangent kernels for 2-layer ReLU networks with random hidden weights, focusing on spectral decomposition and eigenfunctions.
result Obtained an approximation formula for functions represented by 2-layer neural networks.
The recently introduced dropout training criterion for neural networks has been the subject of much attention due to its simplicity and remarkable effectiveness as a regularizer, as well as its interpretation as a training procedure for an exponentially large ensemble of networks that share parameters. In this work we …
Financial markets are complex adaptive systems, and are commonly studied as complex networks. Most of such studies fall short in two respects: they do not account for non-linearity of the studied relationships, and they create one network for the whole studied time series, providing an average picture of a very long, e…
We introduce a variational framework to learn the activation functions of deep neural networks. Our aim is to increase the capacity of the network while controlling an upper-bound of the actual Lipschitz constant of the input-output relation. To that end, we first establish a global bound for the Lipschitz constant of …
This study compares mtl architectures for renewable power generation forecasting.
problem Forecasting expected power generation for renewable power plants.
method Comparative study of mtl architectures including Hard parameter sharing, cross-stitch network, sluice network, and task embedding.
result Emerging Relation Network (ERN) and task embedding provide significant improvements in forecasting.
We establish, for the first time, connections between feedforward neural networks with ReLU activation and tropical geometry --- we show that the family of such neural networks is equivalent to the family of tropical rational maps. Among other things, we deduce that feedforward ReLU neural networks with one hidden laye…
Boltzmann-Gibbs distribution arises as the statistical equilibrium probability distribution of money among the agents of a closed economic system where random and undirected exchanges are allowed. When considering a model with uniform savings in the exchanges, the final distribution is close to the gamma family. In thi…
The implementation of artificial neural networks in hardware substrates is a major interdisciplinary enterprise. Well suited candidates for physical implementations must combine nonlinear neurons with dedicated and efficient hardware solutions for both connectivity and training. Reservoir computing addresses the proble…
Steerable E(3) Graph Neural Networks incorporate geometric and physical covariant information.
problem Incorporating covariant information like position, force, velocity, or spin in graph neural networks.
method Steerable E(3) Equivariant Graph Neural Networks (SEGNNs) that use steerable MLPs to incorporate geometric and physical covariant information.
result SEGNNs improve upon classic linear point convolutions and recent equivariant graph networks that send invariant messages.
New methods test correlation between network structure and node features.
problem Assessing correlation between network structure and node-level covariates.
method Four novel methods based on linear models and canonical correlation analysis.
result Theoretical guarantees and computational efficiency for testing network dependency.
Model captures neural activity related to behavior while separating internal computations.
problem Capturing neural activity related to behavior from complex brain recordings.
method Behavior-decomposed linear dynamical systems (b-dLDS) model.
result Improves over state-of-the-art models in disentangling behavior-related dynamics.
Estimating a constrained relation is a fundamental problem in machine learning. Special cases are classification (the problem of estimating a map from a set of to-be-classified elements to a set of labels), clustering (the problem of estimating an equivalence relation on a set) and ranking (the problem of estimating a …