arXiv research

This paper improves auto-augment efficiency by sharing augmentation weights.

This paper investigates weight-sharing in NAS, revealing its impact and providing solutions.

Algorithm learns which weights to share in deep multi-task learning.

Weight-sharing is rarely helpful in NAS, especially on small datasets.

We propose weight-sharing regularization for neural networks.

This paper evaluates heuristics and hyperparameters in weight-sharing NAS methods.

BONAS accelerates NAS while maintaining reliability.

The study examines how weight sharing, equivariance, and locality affect the sample complexity of neural networks.

K-FAC speeds up training of modern neural networks with linear weight-sharing.

This paper extends liquidity returns in geometric mean markets to time-varying weights.

AlphaNet improves supernets training with alpha-divergence.

Establishes connection between MTDNN and multitask GP, revealing weight correlation as key to task sharing.

The use of automatic methods, often referred to as Neural Architecture Search (NAS), in designing neural network architectures has recently drawn considerable attention. In this work, we present an efficient NAS approach, named HM- NAS, that generalizes existing weight sharing based NAS approaches. Existing weight shar…

Mirror descent with an entropic regularizer is known to achieve shifting regret bounds that are logarithmic in the dimension. This is done using either a carefully designed projection or by a weight sharing technique. Via a novel unified analysis, we show that these two approaches deliver essentially equivalent bounds …

Theoretical analysis of CNNs' inductive biases and their efficiency in approximating functions.

Proposes a new method for rank-consistent ordinal regression without weight-sharing constraints.

CNNs require fewer samples than LCNs and FCNs for image-based tasks due to locality and weight sharing.

We present a novel method of compression of deep Convolutional Neural Networks (CNNs) by weight sharing through a new representation of convolutional filters. The proposed method reduces the number of parameters of each convolutional layer by learning a 1D vector termed Filter Summary (FS). The convolutional filters ar…

We identify a phenomenon, which we refer to as multi-model forgetting, that occurs when sequentially training multiple deep networks with partially-shared parameters; the performance of previously-trained models degrades as one optimizes a subsequent one, due to the overwriting of shared parameters. To overcome this, w…

New concept of attitude towards probability introduced in risk sharing problems.

VSML unifies meta learning concepts and enables simple backpropagation.

The paper improves classification accuracy by leveraging a shared signal across domains in high-dimensional classification.

The success of deep learning in numerous application domains created the de- sire to run and train them on mobile devices. This however, conflicts with their computationally, memory and energy intense nature, leading to a growing interest in compression. Recent work by Han et al. (2015a) propose a pipeline that involve…

CLAW adapts weights to balance continual learning and catastrophic forgetting.

We propose a method for simultaneously detecting shared and unshared communities in heterogeneous multilayer weighted and undirected networks. The multilayer network is assumed to follow a generative probabilistic model that takes into account the similarities and dissimilarities between the communities. We make use of…

A framework for multilayer networks predicts links without shared structures.

Diffusion models adapt to low-dimensional structures for nonparametric density estimation.

The easy access to large data sets has allowed for leveraging methodology in network physics and complexity science to disentangle patterns and processes directly from the data, leading to key insights in the behavior of systems. Here we use to country specific food production data to study binary and weighted topologi…

New insights into choosing between two data integration methods based on SVD.

New method FedEx accelerates federated hyperparameter tuning.

Introduces Causal Energy Minimization to understand Transformer layers.

Model improves covariance estimation from shared and distinct datasets.

NetFuse merges different DNN models with varying weights for faster inference.

This paper uses graph convolutional networks to improve the accuracy of neural architecture search.

Extends inf-convolution to countable risk measures for risk sharing.

Different neural networks trained on the same dataset often learn similar input-output mappings with very different weights. Is there some correspondence between these neural network solutions? For linear networks, it has been shown that different instances of the same network architecture encode the same representatio…

Multi-task learning (MTL) allows deep neural networks to learn from related tasks by sharing parameters with other networks. In practice, however, MTL involves searching an enormous space of possible parameter sharing architectures to find (a) the layers or subspaces that benefit from sharing, (b) the appropriate amoun…

As the multi-view data grows in the real world, multi-view clus-tering has become a prominent technique in data mining, pattern recognition, and machine learning. How to exploit the relation-ship between different views effectively using the characteristic of multi-view data has become a crucial challenge. Aiming at th…

Proposes HOTA-FedGradNorm for faster and robust PFL in noisy channels.

SPIRE enables efficient federated learning for diffusion models by separating client-specific embeddings from a shared backbone.

Neural architecture search (NAS) is a promising research direction that has the potential to replace expert-designed networks with learned, task-specific architectures. In this work, in order to help ground the empirical results in this field, we propose new NAS baselines that build off the following observations: (i) …

The current trend of pushing CNNs deeper with convolutions has created a pressing demand to achieve higher compression gains on CNNs where convolutions dominate the computation and parameter amount (e.g., GoogLeNet, ResNet and Wide ResNet). Further, the high energy consumption of convolutions limits its deployment on m…

Collecting the large datasets needed to train deep neural networks can be very difficult, particularly for the many applications for which sharing and pooling data is complicated by practical, ethical, or legal concerns. However, it may be the case that derivative datasets or predictive models developed within individu…

Many machine learning algorithms are based on the assumption that training examples are drawn independently. However, this assumption does not hold anymore when learning from a networked sample because two or more training examples may share some common objects, and hence share the features of these shared objects. We …

This paper considers the scenario that multiple data owners wish to apply a machine learning method over the combined dataset of all owners to obtain the best possible learning output but do not want to share the local datasets owing to privacy concerns. We design systems for the scenario that the stochastic gradient d…

This paper tackles negative transfer in multi-task learning by introducing class-wise weights.

We provide a series of results for unsupervised learning with autoencoders. Specifically, we study shallow two-layer autoencoder architectures with shared weights. We focus on three generative models for data that are common in statistical machine learning: (i) the mixture-of-gaussians model, (ii) the sparse coding mod…

Hypernetworks are neural networks that generate weights for another neural network. We formulate the hypernetwork training objective as a compromise between accuracy and diversity, where the diversity takes into account trivial symmetry transformations of the target network. We explain how this simple formulation gener…