New algorithm combines new and historical data with different input dimensions for linear regression.
problem Combining new and historical data with different input dimensions for improved accuracy.
method Proposes a transfer learning algorithm with rigorous theoretical robustness analysis.
result Achieves state-of-the-art performance on 9 real-life datasets.
We study the impact of input noise dimension on GAN performance.
problem Challenges in GANs like mode collapse and unstable training.
method Quantitative analysis using FID and IS on different image datasets.
result The optimal input noise dimension varies by dataset and architecture.
Over the past few years, neural networks were proven vulnerable to adversarial images: targeted but imperceptible image perturbations lead to drastically different predictions. We show that adversarial vulnerability increases with the gradients of the training objective when viewed as a function of the inputs. Surprisi…
Neural networks approximate and estimate binary classifiers with polynomial input dependence.
problem Approximating and estimating binary classification functions with neural networks in high dimensions.
method ReLU neural networks, empirical risk minimization, Barron class.
result Approximation and estimation rates are independent of input dimension, overcoming curse of dimensionality.
Proposes a neural network for handling multi-sensor time series with varying input dimensions.
problem Handling multi-sensor time series with varying input dimensions.
method Graph neural network conditioning vectors for zero-shot transfer learning.
result Better generalization in activity recognition and equipment prognostics datasets.
We extend graph neural networks to transfer performance across different input sizes.
problem Transferability of graph neural networks across varying input dimensions.
method Introduce a general framework for transferability across dimensions, showing it corresponds to continuity in a limit space.
result Transferability of graph neural networks is driven by data and learning task, and can be ensured with design principles.
New findings on hidden symmetries in ReLU networks.
problem Understanding the redundancy and symmetries in ReLU network parameter space.
method Analyzing parameter settings and function classes for various network architectures.
result For certain network architectures, there are no hidden symmetries.
Enhances multi-fidelity modeling with DGPs for different input domains.
problem Improving prediction accuracy with multi-fidelity models using different input domains.
method Extends Deep Gaussian Processes (DGPs) to handle different input domains for high and low-fidelity models.
result Demonstrates improved performance on real-world physical problems.
FoRDE uses input gradients to improve neural network ensembles.
problem Improving neural network ensembles for robustness and accuracy.
method Proposes FoRDE, an ensemble learning method based on ParVI, which repels function space by input gradients.
result FoRDE significantly outperforms DEs and other ensemble methods in accuracy and calibration.
Study on VC dimension of GCNNs with input resolution effects.
problem Understanding the generalization capabilities of GCNNs.
method Derived upper and lower bounds for VC dimension, analyzed factors affecting it.
result Extended previous results on VC dimension of GCNNs, providing insights into input resolution dependence.
Neural networks adapt to any input dimensionality.
problem Learning functions with inputs of varying dimensions.
method Equivariant neural networks using representation stability.
result Training on fixed dimensions allows extension to any input.
A new method reduces both input and output dimensions for better goal-oriented analysis.
problem Simultaneous reduction of input and output dimensions for more accurate analysis.
method Coupled input-output dimension reduction, optimizing gradient-based bounds.
result Determine most informative sensors and influential parameters efficiently.
We utilize long-term memory, fractal dimension and approximate entropy as input variables for the Efficiency Index [Kristoufek & Vosvrda (2013), Physica A 392]. This way, we are able to comment on stock market efficiency after controlling for different types of inefficiencies. Applying the methodology on 38 stock marke…
Generative adversarial networks benefit from optimal input dimension and adaptive generator architecture.
problem Minimizing generalization error in GANs through optimal input dimension.
method Introducing generalized GANs (G-GANs) with group penalty and architecture penalty for adaptive dimensionality reduction and network architecture identification.
result G-GANs achieve superior performance with 40%+ improvements in maximum mean discrepancy or Frechet inception distance compared to off-the-shelf methods.
Monotonic improvement in uncertainty estimation with Gaussian processes as dimension increases.
problem Uncertainty quantification in machine learning models, especially with Gaussian processes, is challenging and poorly understood.
method Analyzing the behavior of marginal likelihood and cross-validation metrics as input dimension increases, and exploring the effects of cold posteriors.
result The marginal likelihood improves monotonically with input dimension, while cross-validation metrics exhibit double descent behavior.
Similarity/Distance measures play a key role in many machine learning, pattern recognition, and data mining algorithms, which leads to the emergence of metric learning field. Many metric learning algorithms learn a global distance function from data that satisfy the constraints of the problem. However, in many real-wor…
Gaussian processes (GPs) provide flexible distributions over functions, with inductive biases controlled by a kernel. However, in many applications Gaussian processes can struggle with even moderate input dimensionality. Learning a low dimensional projection can help alleviate this curse of dimensionality, but introduc…
Well-established methods for the solution of stochastic partial differential equations (SPDEs) typically struggle in problems with high-dimensional inputs/outputs. Such difficulties are only amplified in large-scale applications where even a few tens of full-order model runs are impracticable. While dimensionality redu…
Recommendation problems with large numbers of discrete items, such as products, webpages, or videos, are ubiquitous in the technology industry. Deep neural networks are being increasingly used for these recommendation problems. These models use embeddings to represent discrete items as continuous vectors, and the vocab…
Deep SMOTE improves SMOTE's stability and accuracy in imbalanced classification.
problem Stability and accuracy issues in SMOTE for imbalanced classification.
method Adapting SMOTE idea in a deep neural network regression model.
result Deep SMOTE outperforms traditional SMOTE in precision, F1 score, and AUC.
Study reveals differences in medical image models' hidden representation refinement.
problem Understanding how intrinsic dimensionality changes in neural network hidden representations across different domains.
method Analysis of 11 natural and medical image datasets using 6 network architectures.
result Medical image models refine hidden representations earlier, suggesting differences in feature abstraction.
Paper reduces hyperparameters in mixed-categorical Gaussian processes for green aircraft optimization.
problem High-dimensional mixed-categorical Gaussian processes with many hyperparameters.
method Innovative dimension reduction algorithm using partial least squares regression.
result Significant reduction in fuel consumption (439 kg) for a green aircraft.
Study identifies key parameters and input dimensions making LLMs and VLMs brittle.
problem Vulnerability of large language and vision-language models to perturbations.
method Proposed FI measure based on information geometry to quantify sensitivity.
result Small subset of high FI parameters significantly contribute to brittleness.
The holy grail of deep learning is to come up with an automatic method to design optimal architectures for different applications. In other words, how can we effectively dimension and organize neurons along the network layers based on the computational resources, input size, and amount of training data? We outline prom…
Long Short-Term Memory (LSTM) Networks and Convolutional Neural Networks (CNN) have become very common and are used in many fields as they were effective in solving many problems where the general neural networks were inefficient. They were applied to various problems mostly related to images and sequences. Since LSTMs…
Characterization of lung nodules as benign or malignant is one of the most important tasks in lung cancer diagnosis, staging and treatment planning. While the variation in the appearance of the nodules remains large, there is a need for a fast and robust computer aided system. In this work, we propose an end-to-end tra…
Methods for combining predictions from different models in a supervised learning setting must somehow estimate/predict the quality of a model's predictions at unknown future inputs. Many of these methods (often implicitly) make the assumption that the test inputs are identical to the training inputs, which is seldom re…
Neural networks learn faster with correlated latent variables.
problem Efficiently learning from higher-order correlations in neural networks.
method Analytical derivation and simulations of two-layer neural networks.
result Correlations between latent variables speed up learning from higher-order correlations.
High-dimensional neural network manifolds misalign with human perception, causing adversarial examples.
problem Adversarial attacks fool neural networks, but their origin is unclear.
method Defined and analyzed a network's perceptual manifold (PM) for a class concept.
result Neural network PMs have orders of magnitude higher dimensions than natural human concepts, suggesting exponential misalignment.
High-dimensional kernel regression struggles due to rotational invariance.
problem Kernel ridge regression struggles in high dimensions due to rotational invariance.
method Analysis of kernel properties and their impact on high-dimensional data.
result Lower bound on generalization error for high-dimensional kernel regression.
We describe theoretical bounds and a practical algorithm for teaching a model by demonstration in a sequential decision making environment. Unlike previous efforts that have optimized learners that watch a teacher demonstrate a static policy, we focus on the teacher as a decision maker who can dynamically choose differ…
New L1 regularization controls neural network generalization error and sparsifies input dimensions.
problem Selecting the optimal number of hidden neurons in neural networks.
method Theoretical analysis of L1 regularization in two-layer neural networks. result Appropriate L1 regularization leads to near minimax optimal generalization risk bounds. Reservoir computing's success depends on mapping different input time series to separable states.
problem Quantifying the ability of random linear reservoirs to map different input time series.
method Mathematical framework using spectral properties of the connectivity matrix.
result Separation capacity is fully characterized by the spectral properties of the connectivity matrix.
Adversarial examples pose a threat to deep neural network models in a variety of scenarios, from settings where the adversary has complete knowledge of the model and to the opposite "black box" setting. Black box attacks are particularly threatening as the adversary only needs access to the input and output of the mode…
Recurrent neural networks (RNNs) have been drawing much attention with great success in many applications like speech recognition and neural machine translation. Long short-term memory (LSTM) is one of the most popular RNN units in deep learning applications. LSTM transforms the input and the previous hidden states to …
3-layer NTK models generalize better than 2-layer models, especially with large input dimensions.
problem Understanding the generalization of overparameterized neural networks.
method Analyzing the 3-layer NTK model's test error and comparing it to 2-layer NTK models.
result 3-layer NTK models have a faster descent in test error with respect to the number of neurons in the second hidden layer.
Automated anomaly detection is essential for managing information and communications technology (ICT) systems to maintain reliable services with minimum burden on operators. For detecting varying and continually emerging anomalies as differences from normal states, learning normal relationships inherent among cross-dom…
In this paper, we study the problem of approximately computing the product of two real matrices. In particular, we analyze a dimensionality-reduction-based approximation algorithm due to Sarlos [1], introducing the notion of nuclear rank as the ratio of the nuclear norm over the spectral norm. The presented bound has i…
Neural network predicts functional responses from scalar inputs.
problem Regression of functional responses with large scalar predictors and nonlinear relationships.
method Transform functional response to finite dimensions, design feed-forward neural network, modify output via objective functions, apply roughness penalty.
result Proposed neural network outperforms conventional methods in multiple scenarios.
A new method for creating simpler models from complex ones.
problem Creating accurate approximations of complex models at reduced costs.
method Sequential adaptive surrogate modeling based on locally spectral expansions.
result Stochastic spectral embedding (SSE) shows good approximation capabilities and scalability.
The paper is devoted to the local classification of generic control-affine systems on an n-dimensional manifold with scalar input for any n>3 or with two inputs for n=4 and n=5, up to state-feedback transformations, preserving the affine structure. First using the Poincare series of moduli numbers we introduce the intr…
Easy conditions found for simplifying complex systems.
problem Linearizing complex two-input systems.
method Endogenous dynamic feedback with a dimension of at most two.
result Necessary and sufficient conditions for linearizability.
Paper introduces new bounds linking data compressibility to generalization error.
problem Establishing data-dependent generalization bounds.
method Variable-size compressibility framework linking generalization error to compression rate of input data.
result New bounds depend on empirical data measure, subsuming existing PAC-Bayes and intrinsic dimension bounds.
In this paper, we study the system identification problem for sparse linear time-invariant systems. We propose a sparsity promoting block-regularized estimator to identify the dynamics of the system with only a limited number of input-state data samples. We characterize the properties of this estimator under high-dimen…
Proposes a stratified sampling method for high-dimensional models using neural active manifolds.
problem Uncertainty propagation in computationally expensive models with many inputs.
method Neural active manifolds for nonlinear dimensionality reduction, followed by stratification in the reduced space.
result Effective variance reduction in high-dimensional models using stratified sampling.
A new model classifies multi-lead ECGs better than single-channel models.
problem Classifying multi-lead ECGs for accurate diagnosis.
method Channel-wise attention mechanism in CNNs for multi-lead ECG classification.
result The model achieves better sensitivity and precision than plain ResNet models.
The paper analyzes dynamics of momentum in high dimensions with sparse updates.
problem Theoretical analysis of momentum dynamics in high-dimensional sparse settings.
method Theoretical analysis of two models: least squares with sparse inputs and logistic regression with a rare class.
result Characterization of high-dimensional limits of momentum dynamics and phase structure.
Test for linearizing 2-input systems with 2D feedback.
problem Linearizability of two-input systems by feedback.
method Algorithmic test for 2D endogenous feedback.
result Systematic derivation of flat outputs.