FedMAX tackles activation divergence in FL, improving accuracy and efficiency.
problem Activation divergence in Federated Learning due to non-IID data.
method Introduced a prior based on maximum entropy to minimize information about per-device activation vectors and make similar classes more alike.
result Significantly more similar activation vectors across multiple devices, leading to better accuracy and efficiency.
Most neural network designs for FPGAs are inflexible. In this paper, we propose a flexible VHDL structure that would allow any neural network to be implemented on multiple FPGAs. Moreover, the VHDL structure allows for testing as well as training multiple neural networks. The VHDL design consists of multiple processor …
The paper proposes methods to find a shared active subspace for multivariate vector-valued functions.
problem Minimizing the deviation between function evaluations in the original and reconstructed spaces.
method Manipulating gradients or SPD matrices to identify a shared structure.
result Summing SPD matrices often identifies the best shared active subspace.
Universal approximation for ODENet and ResNet with a single activation function.
problem Approximating complex dynamical systems with limited vector fields.
method Examined ODENet and ResNet with vector fields composed of a single activation function and affine mapping.
result ODENet and ResNet with restricted vector fields can uniformly approximate those with general vector fields.
An appropriate choice of the activation function (like ReLU, sigmoid or swish) plays an important role in the performance of (deep) multilayer perceptrons (MLP) for classification and regression learning. Prototype-based classification learning methods like (generalized) learning vector quantization (GLVQ) are powerful…
BALLAST optimizes Lagrangian observer placement for ocean vector fields.
problem Optimizing Lagrangian observer placement for time-dependent ocean vector fields.
method Bayesian active learning with look-ahead amendment for sea-drifter trajectories using a physics-informed spatio-temporal Gaussian process surrogate model.
result Noticeable benefits of BALLAST-aided observer placement strategies on synthetic and high-fidelity ocean models.
CAVs reveal latent concept distributions, but are vulnerable to adversarial attacks.
problem Understanding latent concept encodings in AI models.
method Probabilistic perspective on CAVs, deriving mean and covariance.
result CAVs can be adversarially manipulated, highlighting a vulnerability.
In this paper we study a symmetry group of vector space. Basis manifold is a homogeneous space of a symmetry group. This concept leads us to the definition of active and passive transformations on basis manifold. Active transformation can be expressed as a transformation of vector space. Passive transformation gives ab…
We study covariance matrix estimation for the case of partially observed random vectors, where different samples contain different subsets of vector coordinates. Each observation is the product of the variable of interest with a 0−1 Bernoulli random variable. We analyze an unbiased covariance estimator under this mod…
This paper analyzes the variability of Concept Activation Vectors (CAVs).
problem The variability of CAVs in explaining AI models.
method Theoretical analysis and experiments on real-life datasets to quantify CAVs variability.
result The variance of CAVs decreases as 1/N, where N is the number of random examples.
Improved video tracking accuracy with active learning.
problem Efficiently predicting object matches in videos with active learning.
method Adversarial approach for active learning with structured prediction domains.
result Better accuracy and computational efficiency for object tracking.
LCW reduces activation shift in neural networks, improving training efficiency and generalization.
problem Activation shift in neural networks leading to non-zero mean preactivation values.
method Linearly constrained weights (LCW) to reduce activation shift in fully connected and convolutional layers.
result LCW resolves the vanishing gradient problem and improves generalization of neural networks.
Active subspace is a model reduction method widely used in the uncertainty quantification community. In this paper, we propose analyzing the internal structure and vulnerability and deep neural networks using active subspace. Firstly, we employ the active subspace to measure the number of "active neurons" at each inter…
fMRI semantic category understanding using linguistic encoding models attempt to learn a forward mapping that relates stimuli to the corresponding brain activation. Classical encoding models use linear multi-variate methods to predict the brain activation (all voxels) given the stimulus. However, these methods essentia…
Unified framework for testing deep learning models with concept activation vectors.
problem Statistical instability and discontinuity in testing with concept activation vectors.
method Introducing α-TCAV, a generalized framework that replaces the indicator function with a parameterized smooth function.
result Unified probabilistic formulation that subsumes TCAV and Multi-TCAV, providing principled guidance on tuning the parameter.
RCAV quantifies model sensitivity to semantic concepts, improving interpretability methods.
problem Lack of semantic interpretability in image classification models.
method RCAV calculates concept gradients and ascent steps to assess model sensitivity to semantic concepts.
result RCAV yields more accurate and robust interpretations of model behavior.
Capsule networks improve on traditional neural networks by using vector activations.
problem Comparing capsule networks to traditional neural networks to validate their benefits.
method Deep visualization analysis, feature encoding across vector components, and instantiation parameter encoding.
result Capsule features encode information differently and provide benefits in computer vision applications.
Novel metric space magnitude and weighting vectors improve machine learning tasks.
problem Improving machine learning algorithms using novel metric space concepts.
method Metric space magnitude and weighting vectors for better machine learning.
result The weighting vector effectively detects boundaries and improves classic machine learning tasks.
We propose a mathematical procedure for finding informed trader activities in European-style options and their underlying asset. The regression model (9) with moving average component was written. Being added to it ARMA-process for log-price differences of underlying asset, the generalized model is written as Vector AR…
Manually labelling large collections of text data is a time-consuming, expensive, and laborious task, but one that is necessary to support machine learning based on text datasets. Active learning has been shown to be an effective way to alleviate some of the effort required in utilising large collections of unlabelled …
Efficient method for choosing data points in machine learning models.
problem Choosing optimal data points for semi-supervised learning.
method Computing a 'regularity tangent' vector to measure model complexity and influence.
result The method efficiently calculates a measure of model complexity and data point influence.
The steel industry has great impacts on the economy and the environment of both developed and underdeveloped countries. The importance of this industry and these impacts have led many researchers to investigate the relationship between a country's steel consumption and its economic activity resulting in the so-called i…
We present a confidence-based single-layer feed-forward learning algorithm SPIRAL (Spike Regularized Adaptive Learning) relying on an encoding of activation spikes. We adaptively update a weight vector relying on confidence estimates and activation offsets relative to previous activity. We regularize updates proportion…
This paper investigates and evaluates support vector machine active learning algorithms for use with imbalanced datasets, which commonly arise in many applications such as information extraction applications. Algorithms based on closest-to-hyperplane selection and query-by-committee selection are combined with methods …
The increasing popularity of internet, wireless technologies and mobile devices has led to the birth of mass connectivity and online interaction through Online Social Networks (OSNs) and similar environments. OSN reflects a social structure consist of a set of individuals and different types of ties like connections, r…
Label space expansion for multi-label classification (MLC) is a methodology that encodes the original label vectors to higher dimensional codes before training and decodes the predicted codes back to the label vectors during testing. The methodology has been demonstrated to improve the performance of MLC algorithms whe…
Network operators are generally aware of common attack vectors that they defend against. For most networks the vast majority of traffic is legitimate. However new attack vectors are continually designed and attempted by bad actors which bypass detection and go unnoticed due to low volume. One strategy for finding such …
The autoencoder is an effective unsupervised learning model which is widely used in deep learning. It is well known that an autoencoder with a single fully-connected hidden layer, a linear activation function and a squared error cost function trains weights that span the same subspace as the one spanned by the principa…
We propose a sparse-coding framework for activity recognition in ubiquitous and mobile computing that alleviates two fundamental problems of current supervised learning approaches. (i) It automatically derives a compact, sparse and meaningful feature representation of sensor data that does not rely on prior expert know…
Proposes PER loss to regularize neural network activations to normal distribution.
problem Improving neural network generalization and training speed.
method Regularizes activations to standard normal distribution via projected error function and Wasserstein distance.
result Minimizes Wasserstein distance between activation distribution and standard normal.
We analyze the joint probability distribution on the lengths of the vectors of hidden variables in different layers of a fully connected deep network, when the weights and biases are chosen randomly according to Gaussian distributions, and the input is in {−1,1}N. We show that, if the activation function φ sat…
We propose two optimization techniques to minimize memory usage and computation while meeting system timing constraints for real-time classification in wearable systems. Our method derives a hierarchical classifier structure for Support Vector Machine (SVM) in order to reduce the amount of computations, based on the pr…
RVFL networks can efficiently approximate Lipschitz functions in L∞ norm.
problem Efficiently approximating Lipschitz continuous functions in L∞ norm.
method Random Vector Functional Link (RVFL) network with ReLU activation functions, proving approximation in L∞ norm.
result An RVFL with ReLU activation functions can approximate Lipschitz continuous functions in L∞ norm.
This paper proposes a method for solving optimization problems in which the decision-maker cannot evaluate the objective function, but rather can only express a preference such as "this is better than that" between two candidate decision vectors. The algorithm described in this paper aims at reaching the global optimiz…
Transformationally invariant processors constructed by transformed input vectors or operators have been suggested and applied to many applications. In this study, transformationally identical processing based on combining results of all sub-processes with corresponding transformations at one of the processing steps or …
Quantized neural networks can represent all fixed-point functions under certain conditions.
problem Expressive power of quantized neural networks under fixed-point arithmetic.
method Analyzing necessary and sufficient conditions for quantized networks to represent all fixed-point functions.
result Various popular activation functions satisfy the sufficient condition for representing all fixed-point functions.
New methods for uncertainty in neural networks with leaky ReLU activations.
problem Uncertainty in feed-forward neural networks with random input perturbations.
method Analytical expressions for PDF and moments of neural network output, linearization of leaky ReLU, Gaussian copula surrogate models.
result Accurate statistical results for large input perturbations, excellent agreement with Monte Carlo simulations.
GOLS finds activation functions affect training robustness, especially ReLU.
problem Investigate how different activation functions impact GOLS in neural network training.
method Identify SNN-GPPs for GOLS, analyze activation function effects on gradient continuity.
result GOLS robust for most activation functions but sensitive to ReLU.
Hexagon grid patterns emerge from conformal isometry in grid cell neural networks.
problem Understanding the algebraic, geometric, and topological properties of grid cells.
method Investigating recurrent neural network models of grid cells, focusing on Lie group and Lie algebra representations, conformal isometry, and hexagon periodic patterns.
result Conformal isometry leads to hexagon periodic patterns in grid cell responses and accurate path integration.
EVA adapts LoRA for faster, more efficient fine-tuning.
problem Fast and efficient fine-tuning of large models for specific tasks.
method EVA uses directions capturing most activation variance for initialization, maximizing gradient signal and reducing parameters.
result EVA achieves faster convergence and higher average scores across tasks, reducing parameters.
Actively sampled data can have very different characteristics than passively sampled data. Therefore, it's promising to investigate using different inference procedures during AL than are used during passive learning (PL). This general idea is explored in detail for the focused case of AL with cost-weighted SVMs for im…
We seek to improve the data efficiency of neural networks and present novel implementations of parameterized piece-wise polynomial activation functions. The parameters are the y-coordinates of n+1 Chebyshev nodes per hidden unit and Lagrangian interpolation between the nodes produces the polynomial on [-1, 1]. We show …
Improves adversarial robustness by constraining logits with a bounded function.
problem Improving adversarial robustness in deep learning models.
method Addition of a bounded function before softmax to constrain logits.
result Our method improves adversarial robustness without requiring adversarial training.
We study discrete time dynamical systems governed by the state equation ht+1=φ(Aht+But). Here A,B are weight matrices, φ is an activation function, and ut is the input data. This relation is the backbone of recurrent neural networks (e.g. LSTMs) which have broad applications in sequential learning tasks. …
New algorithm reduces clustering cost in bandit feedback.
problem Discovering hidden groupings in stochastic bandit feedback.
method ACB algorithm for active clustering with non-asymptotic lower bound.
result ACB algorithm's budget matches lower bound in most regimes.
To compare entities of differing types and structural components, the artificial neural network paradigm was used to cross-compare structural components between heterogeneous documents. Trainable weighted structural components were input into machine-learned activation functions of the neurons. The model was used for m…
We consider networks, trained via stochastic gradient descent to minimize ℓ2 loss, with the training labels perturbed by independent noise at each iteration. We characterize the behavior of the training dynamics near any parameter vector that achieves zero training error, in terms of an implicit regularization te…
We propose two approaches of locally adaptive activation functions namely, layer-wise and neuron-wise locally adaptive activation functions, which improve the performance of deep and physics-informed neural networks. The local adaptation of activation function is achieved by introducing a scalable parameter in each lay…