Sharp limits found for storing and retrieving input-output associations in linear associative memories.
problem Understanding the fundamental limits of storing and retrieving input-output associations in neural networks.
method Study of a minimal linear associative memory model, introducing a decoupled model and using statistical physics to characterize storage capacity.
result Linear associative memory can store up to 1/2 log(p) associations, providing a sharp statistical-physics characterization.
Despite tremendous progress in outlier detection research in recent years, the majority of existing methods are designed only to detect unconditional outliers that correspond to unusual data patterns expressed in the joint space of all data attributes. Such methods are not applicable when we seek to detect conditional …
In practice it is often found that large over-parameterized neural networks generalize better than their smaller counterparts, an observation that appears to conflict with classical notions of function complexity, which typically favor smaller models. In this work, we investigate this tension between complexity and gen…
Sigmoid autoencoders can implement associative memory with certain conditions.
problem Implementing associative memory in neural networks.
method Theoretical analysis of overparameterized sigmoid autoencoders using the NTK and iterative maps.
result Overparameterized sigmoid autoencoders can have attractors in the NTK limit, leading to associative memory.
Economic systems, traditionally analyzed as almost independent national systems, are increasingly connected on a global scale. Only recently becoming available, the World Input-Output Database (WIOD) is one of the first efforts to construct the multi-regional input-output (MRIO) tables at the global level. By viewing t…
There has been a recent shift in sequence-to-sequence modeling from recurrent network architectures to convolutional network architectures due to computational advantages in training and operation while still achieving competitive performance. For systems having limited long-term temporal dependencies, the approximatio…
Unitary recurrent neural networks (URNNs) have been proposed as a method to overcome the vanishing and exploding gradient problem in modeling data with long-term dependencies. A basic question is how restrictive is the unitary constraint on the possible input-output mappings of such a network? This work shows that for …
New approach ties loss curvature to model performance in deep learning.
problem Understanding the relationship between loss curvature and model performance in deep learning.
method Empirical analysis of loss Hessians and theoretical results on input-output Jacobians.
result Novel generalization bound in terms of empirical Jacobian.
This paper investigates how economic shocks propagate and amplify through the input-output network connecting industrial sectors in developed economies. We study alternative models of diffusion on networks and we calibrate them using input-output data on real-world inter-sectoral dependencies for several European count…
We present an approach for designing correct-by-construction neural networks (and other machine learning models) that are guaranteed to be consistent with a collection of input-output specifications before, during, and after algorithm training. Our method involves designing a constrained predictor for each set of compa…
Method combines clustering and matrix completion for missing data in I/O tables.
problem Reconstructing missing entries in World Input-Output (I/O) matrices due to data collection issues.
method Hierarchical clustering and Matrix Completion with LASSO-like nuclear norm penalty.
result The method effectively predicts missing values from previous and similar countries' data.
`Biologically inspired' activation functions, such as the logistic sigmoid, have been instrumental in the historical advancement of machine learning. However in the field of deep learning, they have been largely displaced by rectified linear units (ReLU) or similar functions, such as its exponential linear unit (ELU) v…
Starting from a dataset with input/output time series generated by multiple deterministic linear dynamical systems, this paper tackles the problem of automatically clustering these time series. We propose an extension to the so-called Martin cepstral distance, that allows to efficiently cluster these time series, and a…
A new Input-Output model, called the Multi-Entity Input-Output (MEIO) model, is introduced to estimate the responsibility of entities of an ecosystem on the footprint of each other. It assumed that the ecosystem is comprised of end users, service providers, and utilities. The proposed MEIO modeling approach can be seen…
New method ranks sectors and countries using local and aggregate I-O data.
problem Ranking sectors and countries in global value chains using incomplete I-O tables.
method Rank-1 approximation to I-O tables using local and aggregate information. result Consistently good performance in reconstructing rankings of upstreamness and downstreamness.
We consider the problem of learning a realization for a linear time-invariant (LTI) dynamical system from input/output data. Given a single input/output trajectory, we provide finite time analysis for learning the system's Markov parameters, from which a balanced realization is obtained using the classical Ho-Kalman al…
Develops neural network approximations for infinite-dimensional input-output maps.
problem Approximating input-output maps between infinite-dimensional spaces.
method Combines neural networks and model reduction techniques.
result Proves convergence of the proposed approximation methodology.
Graph-to-Tree Neural Networks improve structured input-output translation in tasks like semantic parsing and math word problems.
problem Improving performance on tasks like semantic parsing and math word problem solving.
method Graph-to-Tree Neural Networks, consisting of a graph encoder and a hierarchical tree decoder.
result Graph2Tree model outperforms or matches state-of-the-art models on neural semantic parsing and math word problem tasks.
Optimizes wide low-rank neural networks for reduced parameters and cost.
problem Reducing the number of learnable parameters in wide neural networks.
method Analyzed edge-of-chaos dynamics and derived formulae for optimal weight and bias variances.
result Optimal weight and bias variances for low-rank networks follow from multiplicative scaling.
We consider the problem of joint modelling of metabolic signals and gene expression in systems biology applications. We propose an approach based on input-output factorial hidden Markov models and propose a structured variational inference approach to infer the structure and states of the model. We start from the class…
Researchers infer firm-level supply chain networks from sector-level data to assess systemic risk.
problem Estimating systemic risk in economic systems using firm-level data.
method Maximum-entropy algorithms applied to input-output tables and firm-level aggregate output data.
result The most realistic systemic risk content is retrieved by models incorporating disaggregated firm-specific inputs by sector.
Sandpile Economics explains how economies can be prone to large crises from small shocks.
problem Capitalist economies' recurrent crises disproportionate to shocks.
method Formal framework interpreting instability as geometric fragility of production networks.
result Curvature of production networks predicts medium-run output dynamics and resilience.
Within machine learning, the supervised learning field aims at modeling the input-output relationship of a system, from past observations of its behavior. Decision trees characterize the input-output relationship through a series of nested if−then−else questions, the testing nodes, leading to a set of predictions, th…
We consider the learning of algorithmic tasks by mere observation of input-output pairs. Rather than studying this as a black-box discrete regression problem with no assumption whatsoever on the input-output mapping, we concentrate on tasks that are amenable to the principle of divide and conquer, and study what are it…
Recent advances in machine learning and artificial intelligence are now being considered in safety-critical autonomous systems where software defects may cause severe harm to humans and the environment. Design organizations in these domains are currently unable to provide convincing arguments that their systems are saf…
Optimal intervention in economic networks modeled as influence maximization, with hard computational problems.
problem Optimal intervention in economic networks modeled as influence maximization.
method Transformed into influence maximization-like form, with theoretical and practical implications.
result Optimal intervention is NP-hard and cannot be approximated to a constant factor in polynomial time.
We consider the problem of learning a high-dimensional multi-task regression model, under sparsity constraints induced by presence of grouping structures on the input covariates and on the output predictors. This problem is primarily motivated by expression quantitative trait locus (eQTL) mapping, of which the goal is …
The paper finds a surprising positive correlation between upstreamness and downstreamness in global value chains.
problem The puzzling positive correlation between upstreamness and downstreamness in industries and countries.
method Analysis of a simple model of random Input/Output tables and experiments on empirical data.
result Upstreamness and downstreamness of the same industrial sector/country are positively correlated with a slope close to +1.
We consider the problem of training input-output recurrent neural networks (RNN) for sequence labeling tasks. We propose a novel spectral approach for learning the network parameters. It is based on decomposition of the cross-moment tensor between the output and a non-linear transformation of the input, based on score …
Interpretability has become an important issue in the machine learning field, along with the success of layered neural networks in various practical tasks. Since a trained layered neural network consists of a complex nonlinear relationship between large number of parameters, we failed to understand how they could achie…
Paper presents a method to accurately quantify neural network uncertainty without sampling.
problem Uncertainty quantification in neural networks for reliability and robustness.
method Sample-free moment propagation technique for mean vectors and covariance matrices.
result Analytic solution for covariance of nonlinear activation functions.
Kernel methods outperform neural nets in operator learning tasks.
problem Learning operators between Banach spaces from partial observations.
method Kernel-based framework with a priori error analysis and numerical comparisons.
result Kernel methods are competitive with neural nets in cost-accuracy trade-off.
Gaussian processes are used in machine learning to learn input-output mappings from observed data. Gaussian process regression is based on imposing a Gaussian process prior on the unknown regressor function and statistically conditioning it on the observed data. In system identification, Gaussian processes are used to …
In a physical neural system, where storage and processing are intimately intertwined, the rules for adjusting the synaptic weights can only depend on variables that are available locally, such as the activity of the pre- and post-synaptic neurons, resulting in local learning rules. A systematic framework for studying t…
New approach generates better synthetic data for neural program synthesis.
problem Current approaches to neural program synthesis generalize poorly to real data.
method Adversarial approach to control synthetic data distributions.
result Proposed method outperforms current approaches.
Maxout networks study gradients and propose initialization strategies.
problem Complexity in input-output Jacobian distribution complicates stable parameter initialization.
method Obtained bounds on moments of gradients and formulated initialization strategies.
result Parameter initialization strategies improve training of deep maxout networks.
JacNet learns Jacobians to enforce structure on derivatives for invertibility and Lipschitz functions.
problem Enforcing structure on derivatives of neural network mappings.
method Proposes using a neural network to directly learn the Jacobian of the input-output function, allowing control over derivative structure.
result Demonstrates learning invertible approximations to simple and 1-Lipschitz functions.
The paper provides results regarding the computational complexity of hybrid system identification. More precisely, we focus on the estimation of piecewise affine (PWA) maps from input-output data and analyze the complexity of computing a global minimizer of the error. Previous work showed that a global solution could b…
LESS combines local predictors for subsets to learn from heterogeneous input-output pairs.
problem Learning from heterogeneous input-output pairs in populations with varied behavior.
method LESS algorithm: generates subsets, trains local predictors, combines them.
result LESS is highly competitive compared to state-of-the-art methods.
Recurrent neural networks can learn complex transduction problems that require maintaining and actively exploiting a memory of their inputs. Such models traditionally consider memory and input-output functionalities indissolubly entangled. We introduce a novel recurrent architecture based on the conceptual separation b…
New framework improves robustness of implicit neural networks.
problem Ill-posedness and convergence instability in implicit neural networks.
method NEMON framework based on contraction theory for ℓ∞ norm, including well-posedness condition, average iteration, and input-output Lipschitz constant regularization. result Improved accuracy and robustness of implicit models with smaller input-output Lipschitz bounds.
Neural Processes (NPs) (Garnelo et al 2018a;b) approach regression by learning to map a context set of observed input-output pairs to a distribution over regression functions. Each function models the distribution of the output given an input, conditioned on the context. NPs have the benefit of fitting observed data ef…
This paper introduces Non-Autonomous Input-Output Stable Network(NAIS-Net), a very deep architecture where each stacked processing block is derived from a time-invariant non-autonomous dynamical system. Non-autonomy is implemented by skip connections from the block input to each of the unrolled processing stages and al…
We develop a statistical mechanical approach based on the replica method to study the design space of deep and wide neural networks constrained to meet a large number of training data. Specifically, we analyze the configuration space of the synaptic weights and neurons in the hidden layers in a simple feed-forward perc…
DEPLOYERS models multi-country economic systems using ABM.
problem Simulate complex multi-country economic systems with detailed data.
method Agent-based modeling framework with multi-threaded simulation.
result Simulates thousands of individuals and firms in a realistic environment.
This paper presents a method for solving the supervised learning problem in which the output is highly nonlinear and discontinuous. It is proposed to solve this problem in three stages: (i) cluster the pairs of input-output data points, resulting in a label for each point; (ii) classify the data, where the correspondin…
We compress large neural networks for quick adaptation to specific contexts.
problem How to quickly adapt a pretrained large neural network to specific contexts.
method Propose a Bayesian hypernetwork framework to compress the network and encourage sparsity.
result Generated compressed networks are significantly smaller than baseline methods.
Extensions to given-data Sobol' index estimators for large models.
problem Efficiently compute Sobol' indices for models with many inputs.
method General definition, streaming algorithm, heuristic filtering.
result Comparable accuracy and lower memory usage for large models.