arXiv research

New optimizers control network width scaling, improving stability and transfer across different model sizes.

Use simplified layerwise linear models to understand neural dynamics.

A new method for learning Bayesian neural networks using layerwise inference.

This work proves the asymptotic freeness of layerwise Jacobians in MLPs with Haar orthogonal matrices.

Efficient algorithm for CMDPs reduces to offline density estimation.

New method $μP^2$ improves neural network training by scaling perturbations layerwise.

AlphaPruning optimizes LLM pruning using HT-SR theory for better performance.

The lack of transparency of neural networks stays a major break for their use. The Layerwise Relevance Propagation technique builds heat-maps representing the relevance of each input in the model s decision. The relevance spreads backward from the last to the first layer of the Deep Neural Network. Layer-wise Relevance…

Training large deep neural networks on massive datasets is computationally very challenging. There has been recent surge in interest in using large batch stochastic optimization methods to tackle this issue. The most prominent algorithm in this line of research is LARS, which by employing layerwise adaptive learning ra…

Proposes efficient training method for deep thin networks.

Deep Gaussian Processes (DGP) are hierarchical generalizations of Gaussian Processes (GP) that have proven to work effectively on a multiple supervised regression tasks. They combine the well calibrated uncertainty estimates of GPs with the great flexibility of multilayer models. In DGPs, given the inputs, the outputs …

Estimating machine learning performance 'in the wild' is both an important and unsolved problem. In this paper, we seek to examine, understand, and predict the pointwise competence of classification models. Our contributions are twofold: First, we establish a statistically rigorous definition of competence that general…

Deep learning is also known as hierarchical learning, where the learner _learns_ to represent a complicated target function by decomposing it into a sequence of simpler functions to reduce sample and time complexity. This paper formally analyzes how multi-layer neural networks can perform such hierarchical learning _ef…

Training avoids edge of stability by aligning Jacobian matrices.

SAEs struggle with curved activation manifolds, revealing layer-dependent scaling laws.

Proves deep networks can learn hierarchical structures efficiently.

Injectivity of ReLU networks is characterized for generative models and inverse problems.

Develops methods to measure and set function-space learning rates in neural networks.

New bounds adaptively control spectral complexity of trained Transformers.

A new robust prefix-tuning framework improves model robustness against adversarial attacks.

Improved code translation by preserving structure with composed fine-tuning.

In this paper, we explicitly construct the Calabi composition of multiple affine hyperspheres possibly including some points viewing as 0-dimensional hypersheres. Then we compute all the basic affine invariants of the composed affine hyperspheres, proving that the composed affine hypersphere is symmetric one if and onl…

With the development of cloud computing and big data, the reliability of data storage systems becomes increasingly important. Previous researchers have shown that machine learning algorithms based on SMART attributes are effective methods to predict hard drive failures. In this paper, we use SMART attributes to predict…

MTL-NAS combines NAS with GP-MTL for task-agnostic multi-task learning.

Scalable method bounds Lipschitz constant of generative models.

This work proposes a method to compose visual relations more faithfully.

In this paper, we study two aspects of the variational autoencoder (VAE): the prior distribution over the latent variables and its corresponding posterior. First, we decompose the learning of VAEs into layerwise density estimation, and argue that having a flexible prior is beneficial to both sample generation and infer…

The paper proposes a deep learning technique for structured and composable representations.

MLJ offers a Julia package for composing machine learning models.

A new measure predicts deep learning model performance.

This paper investigates end-to-end learnable models for attributing composers to musical scores. We introduce several pooled, convolutional architectures for this task and draw connections between our approach and classical learning approaches based on global and n-gram features. We evaluate models on a corpus of 2,500…

Deep neural networks (DNNs) depend on the storage of a large number of parameters, which consumes an important portion of the energy used during inference. This paper considers the case where the energy usage of memory elements can be reduced at the cost of reduced reliability. A training algorithm is proposed to optim…

We propose a distributed approach to train deep neural networks (DNNs), which has guaranteed convergence theoretically and great scalability empirically: close to 6 times faster on instance of ImageNet data set when run with 6 machines. The proposed scheme is close to optimally scalable in terms of number of machines, …

Large learning rates work surprisingly well in standard parameterization, contrary to theory.

Bardo Composer generates tabletop RPG music based on player speech.

We give algorithms with provable guarantees that learn a class of deep nets in the generative model view popularized by Hinton and others. Our generative model is an $n$ node multilayer neural net that has degree at most $n^γ$ for some $γ<1$ and each edge has a random edge weight in $[-1,1]$. Our algorithm learns {\em …

Meta-materials simulation sped up with energy surrogates.

NoLimits.jl: Flexible and Composable Nonlinear Mixed-Effects Modeling in Julia

Any generalized distance-squared mapping of equidimensional case has singularities, and their singularity types are wrapped into mystery in higher dimensional cases. Any generalized distance-squared mapping of equidimensional case is not injective. Nevertheless, in this paper, it is shown that the non-singular property…

How can we build recommender systems to take into account fairness? Real-world recommender systems are often composed of multiple models, built by multiple teams. However, most research on fairness focuses on improving fairness in a single model. Further, recent research on classification fairness has shown that combin…

This paper introduces the Differentiable Algorithm Network (DAN), a composable architecture for robot learning systems. A DAN is composed of neural network modules, each encoding a differentiable robot algorithm and an associated model; and it is trained end-to-end from data. DAN combines the strengths of model-driven …

Composing previously mastered skills to solve novel tasks promises dramatic improvements in the data efficiency of reinforcement learning. Here, we analyze two recent works composing behaviors represented in the form of action-value functions and show that they perform poorly in some situations. As part of this analysi…

Model-free deep reinforcement learning has been shown to exhibit good performance in domains ranging from video games to simulated robotic manipulation and locomotion. However, model-free methods are known to perform poorly when the interaction time with the environment is limited, as is the case for most real-world ro…

New view: Deep GCNs learn to anti-oversmooth during training.

The ability to compose learned skills to solve new tasks is an important property of lifelong-learning agents. In this work, we formalise the logical composition of tasks as a Boolean algebra. This allows us to formulate new tasks in terms of the negation, disjunction and conjunction of a set of base tasks. We then sho…

New measure shows how LSTM models compose hierarchical representations.

Humans are able to perform a myriad of sophisticated tasks by drawing upon skills acquired through prior experience. For autonomous agents to have this capability, they must be able to extract reusable skills from past experience that can be recombined in new ways for subsequent tasks. Furthermore, when controlling com…

In this note we present a generative model of natural images consisting of a deep hierarchy of layers of latent random variables, each of which follows a new type of distribution that we call rectified Gaussian. These rectified Gaussian units allow spike-and-slab type sparsity, while retaining the differentiability nec…