New model predicts neural network performance from early training epochs, incorporating architecture impact.
problem Predicting neural network performance from early training epochs, neglecting architecture impact.
method Architecture-aware graph ordinary differential equation model.
result Model outperforms state-of-the-art methods for MLP and CNN learning curves.
The practical usage of reinforcement learning agents is often bottlenecked by the duration of training time. To accelerate training, practitioners often turn to distributed reinforcement learning architectures to parallelize and accelerate the training process. However, modern methods for scalable reinforcement learnin…
This study presents a novel, deep, fully convolutional architecture which is optimized for the task of EEG-based neonatal seizure detection. Architectures of different depths were designed and tested; varying network depth impacts convolutional receptive fields and the corresponding learned feature complexity. Two deep…
ICON-OCnet solves optimal execution problems with neural networks and few examples.
problem Optimal order execution in markets with unknown price impact.
method Transformer-based neural network architecture (ICON-OCnet) that learns price impact from few examples and applies it to optimal execution strategies.
result ICON-OCnet accurately infers price impact models and retrieves optimal execution strategies for various propagator kernels.
This work explores how neural network architecture affects robustness to noisy labels.
problem The impact of neural network architecture on robustness to noisy labels.
method Formal framework connecting robustness to architecture alignments, measured by predictive power in representations.
result Network robustness to noisy labels improves when its architecture is more aligned with the target function.
Modern audio source separation techniques rely on optimizing sequence model architectures such as, 1D-CNNs, on mixture recordings to generalize well to unseen mixtures. Specifically, recent focus is on time-domain based architectures such as Wave-U-Net which exploit temporal context by extracting multi-scale features. …
Study on encoding neural architectures for NAS, showing impact on performance.
problem Impact of different encodings on NAS performance.
method Formal definition and characterization of encodings, empirical study of encodings' effectiveness.
result Different encodings can significantly affect NAS performance.
This paper investigates weight-sharing in NAS, revealing its impact and providing solutions.
problem Reducing the time and computational cost of training neural networks.
method Comprehensive experiments on weight-sharing in NAS, analyzing variance and interference.
result Properly reducing weight sharing can reduce variance and improve model performance.
The term Neural Architecture Search (NAS) refers to the automatic optimization of network architectures for a new, previously unknown task. Since testing an architecture is computationally very expensive, many optimizers need days or even weeks to find suitable architectures. However, this search time can be significan…
NAS evaluation is hard due to lack of standard protocols.
problem Difficulty in comparing NAS methods due to varying search spaces and evaluation protocols.
method Benchmarked 8 NAS methods on 5 datasets, proposing a relative improvement metric over random architectures.
result Many NAS techniques struggle to significantly outperform a randomly generated architecture.
Bayesian optimisation with graph kernels improves neural architecture search and provides interpretability.
problem Lack of insight into why architectures perform well and how to improve them.
method Combines Bayesian optimisation with Weisfeiler-Lehman graph kernels for highly data-efficient and interpretable architecture search.
result Demonstrates state-of-the-art performance on closed- and open-domain search spaces.
Study shows refugee matching gains are robust to different evaluation methods.
problem Stability of refugee matching gains under various evaluation methods.
method Used multiple off-policy evaluation methods including IPW and AIPW.
result Impact estimates remain consistent in magnitude and statistically significant.
Higher granularity in MoE models boosts expressivity exponentially.
problem Expressivity of Mixture-of-Experts models with varying granularity.
method Comparing models with different numbers of active experts (granularity).
result Exponential separation in network expressivity based on granularity.
In this paper we present a statistical analysis about the characteristics that we intend to influence in the performance of the neural networks in terms of assertiveness in the prediction of Brazilian stock returns. We created a population of architectures for analysis and extracted the sample that had the best asserti…
Weight-sharing is rarely helpful in NAS, especially on small datasets.
problem The efficiency of weight-sharing in Neural Architecture Search (NAS) is questionable.
method Comparison of a state-of-the-art weight-sharing approach to random search on the nasbench dataset.
result Weight-sharing is only rarely significantly helpful in NAS, highlighting the importance of the search space.
Deep learning models reconstruct volatility surfaces from noisy data under no-arbitrage constraints.
problem Reconstructing implied volatility surfaces from sparse and noisy option quotes.
method Compared multiple neural architectures including Transformers, U-Nets, and variational autoencoders.
result Transformer and U-Net architectures achieve strong reconstruction accuracy, especially under sparse observation regimes.
Cell-based NAS search spaces are redundant and lack novelty.
problem Redundancy and lack of novelty in cell-based neural architecture search.
method Empirical analysis of architectures from popular cell-based search spaces.
result Randomly sampled architectures can match or outperform state-of-the-art results.
A new UNet variant reduces spectral artifacts in image transformations.
problem Spectral artifacts caused by traditional UNet upsampling layers.
method Introduced a Guided UNet (GUNet) architecture using a novel upsampling module.
result GUNet produces higher fidelity outputs in image transformations.
We provide an overview of several non-linear activation functions in a neural network architecture that have proven successful in many machine learning applications. We conduct an empirical analysis on the effectiveness of using these function on the MNIST classification task, with the aim of clarifying which functions…
Framework ranks sectors influenced by Indian Union Budgets.
problem Real-time analysis of budgetary impacts on sector-specific equity performance.
method Fine-tuned embeddings and language models for sector identification and performance ranking.
result 0.997 NDCG score in predicting sector ranks based on post-budget performances.
Paper analyzes soft tree ensembles using NTK, finding only leaf count matters.
problem Understanding impact of various tree architectures in ensemble learning.
method Formulated and analyzed Neural Tangent Kernel (NTK) for soft tree ensembles.
result Only the number of leaves at each depth is relevant for tree architecture in ensemble learning.
A hybrid ASR system using conformer architecture improves word-error-rate and training speed.
problem Improving word-error-rate and training efficiency for hybrid ASR systems.
method Used conformer architecture, applied time downsampling, and transposed convolutions.
result Conformer-based hybrid model achieves competitive results and significantly outperforms BLSTM-based hybrid model.
New method improves neural architecture search by optimizing for both performance and diversity.
problem Traditional multi-objective NAS fails to address practical constraints and niches.
method Formulated as quality diversity optimization, introduces multifidelity optimizers.
result Quality diversity NAS outperforms multi-objective NAS in quality and efficiency.
Deep Echo State Networks (DeepESNs) recently extended the applicability of Reservoir Computing (RC) methods towards the field of deep learning. In this paper we study the impact of constrained reservoir topologies in the architectural design of deep reservoirs, through numerical experiments on several RC benchmarks. Th…
Paper introduces privacy-preserving deep-learning service.
problem Privacy loss in deep-learning-as-a-service.
method Homomorphic Encryption tailored for CNNs.
result Effectiveness of proposed privacy-preserving architecture.
Develops a new framework for integrating satellite allocations in small portfolios.
problem Feasibility constraints in small portfolios, not return predictability, are the primary concerns.
method A four-layer feasibility framework: physical, economic, structural, and epistemic.
result Closed-form feasibility bounds on satellite size, turnover, and breadth without return forecasts.
This paper investigates how network width and depth affect adversarially robust DNNs.
problem Understanding architectural configurations for adversarially robust DNNs.
method Comprehensive investigation on the impact of network width and depth on adversarial robustness.
result Optimal architectural configuration for adversarial robustness exists and can improve robustness.
Bayesian Optimization (BO) is a common approach for hyperparameter optimization (HPO) in automated machine learning. Although it is well-accepted that HPO is crucial to obtain well-performing machine learning models, tuning BO's own hyperparameters is often neglected. In this paper, we empirically study the impact of o…
This study examines how model architecture affects deep learning model privacy.
problem Privacy concerns in deep learning models due to potential leakage of sensitive information.
method Investigation of CNNs and Transformers, focusing on activation layers, stem layers, LN layers, and attention modules.
result Transformers generally exhibit higher vulnerability to privacy attacks than CNNs.
Neural architecture search (NAS) has a great impact by automatically designing effective neural network architectures. However, the prohibitive computational demand of conventional NAS algorithms (e.g. 104 GPU hours) makes it difficult to \emph{directly} search the architectures on large-scale tasks (e.g. ImageNet).…
Theory explains why DARTS favors deep architectures over shallow ones.
problem DARTS selects architectures with dominated skip connections, leading to performance degradation.
method Theoretical analysis of operations' effects on network optimization; introduces sparse binary gates and path-depth-wise regularization.
result Theoretical proof that architectures with more skip connections converge faster.
NoisyDARTS injects random noise to improve neural architecture search.
problem Performance collapse in Differentiable Architecture Search (DARTS).
method Inject unbiased random noise to skip connections to impede gradient flow.
result NoisyDARTS achieves state-of-the-art results across various tasks.
Empirical study shows removing neural parameter symmetries impacts model performance.
problem Understanding the impact of neural parameter symmetries on model performance.
method Developed two methods to reduce parameter space symmetries in neural networks.
result Removing parameter symmetries can lead to faster and more effective Bayesian neural network training.
Mixed integer programming identifies critical neurons in neural networks.
problem Identifying neurons critical for network performance and generalization.
method Developed a mixed integer program (MIP) to assign importance scores to neurons, guiding pruning decisions.
result The method identifies multiple 'lucky' sub-networks resulting in optimized architectures that generalize across datasets.
This paper distills Bayesian posterior expectations for deep neural networks.
problem Improving deep neural network performance and uncertainty quantification.
method Develops a framework for distilling expectations from Bayesian posterior distributions using Monte Carlo samples.
result The framework successfully distills posterior predictive distribution and expected entropy.
Importance-weighted risk minimization is a key ingredient in many machine learning algorithms for causal inference, domain adaptation, class imbalance, and off-policy reinforcement learning. While the effect of importance weighting is well-characterized for low-capacity misspecified models, little is known about how it…
Study on shortcuts in deep networks, revealing their layer-wise distribution and impact.
problem Understudied impact of shortcuts on feature representations in deep networks.
method Layer-wise localization through counterfactual training on clean and skewed datasets.
result Shortcuts are distributed throughout the network, not localized in specific layers.
Study evaluates scalability and real-world impact of disentangled representations.
problem Scalability and real-world impact of disentangled representations.
method New high-resolution dataset and architectures for disentangled representation learning.
result Disentanglement predicts out-of-distribution task performance.
Maintaining good cardiac function for as long as possible is a major concern for healthcare systems worldwide and there is much interest in learning more about the impact of different risk factors on cardiac health. The aim of this study is to analyze the impact of systolic blood pressure (SBP) on cardiac function whil…
This paper uses graph convolutional networks to improve the accuracy of neural architecture search.
problem Improving the precision of sampled sub-networks in weight-sharing NAS.
method Training a graph convolutional network to fit the performance of sampled sub-networks.
result Achieved higher rank correlation coefficient and better final architecture performance.
This study analyzes NAS benchmarks and finds that only a subset of operations is crucial for generating high-performing architectures.
problem NAS benchmarks lack generability and provide skewed performance distributions, leading to unreliable comparisons.
method Empirical analysis of widely used NAS benchmarks (101, 201, TransNAS-Bench-101) focusing on operation importance and generability.
result Only a subset of operations is necessary to generate architectures close to the upper-bound performance range, and convolution layers have the highest impact.
Most digital cameras use sensors coated with a Color Filter Array (CFA) to capture channel components at every pixel location, resulting in a mosaic image that does not contain pixel values in all channels. Current research on reconstructing these missing channels, also known as demosaicing, introduces many artifacts, …
This paper evaluates heuristics and hyperparameters in weight-sharing NAS methods.
problem Improving the performance of weight-sharing NAS methods.
method Systematic evaluation of heuristics and hyperparameters in weight-sharing NAS algorithms.
result Some heuristics negatively impact super-net and stand-alone performance correlation.
Online retailers execute a very large number of price updates when compared to brick-and-mortar stores. Even a few mis-priced items can have a significant business impact and result in a loss of customer trust. Early detection of anomalies in an automated real-time fashion is an important part of such a pricing system.…
The presence of Long Distance Dependencies (LDDs) in sequential data poses significant challenges for computational models. Various recurrent neural architectures have been designed to mitigate this issue. In order to test these state-of-the-art architectures, there is growing need for rich benchmarking datasets. Howev…
Measures equivariance in vision models using Lie derivative.
problem Understanding the role of equivariance in recent vision models.
method Introducing Lie derivative to measure equivariance with strong mathematical foundations and minimal hyperparameters.
result Many violations of equivariance can be linked to spatial aliasing in network layers, and larger models tend to display more equivariance.
Anisotropic data structure affects learning dynamics and generalization error in linear networks.
problem Understanding the impact of data anisotropy on learning dynamics and generalization error in linear networks.
method Examined a spiked covariance structure as a model of anisotropy in a two-layer linear network in a linear regression setting.
result Learning dynamics proceed in two phases: initially driven by input-output correlation, then by other principal directions of the data structure. Derived an analytical expression for the generalization error.
Reduced numerical precision is a common technique to reduce computational cost in many Deep Neural Networks (DNNs). While it has been observed that DNNs are resilient to small errors and noise, no general result exists that is capable of predicting a given DNN system architecture's sensitivity to reduced precision. In …