Develops a new method for nonlinear dimension reduction using random features.
problem Statistical challenges in generalizing Gaussian process-based latent variable models to non-Gaussian data.
method Random feature latent variable models (RFLVMs) that approximate nonlinear relationships with linear functions of random features.
result RFLVMs produce comparable results to state-of-the-art methods on various data types.
Three-layer networks learn complex hierarchical polynomials of multiple nonlinear features.
problem Understanding how neural networks learn hierarchical features of multiple nonlinear inputs.
method Examine a broad class of functions using three-layer neural networks, showing complete recovery and efficient learning.
result Three-layer neural networks trained via gradient descent can learn hierarchical polynomials of multiple nonlinear features efficiently.
A novel model uses ODE-based random features to model nonlinear dynamical systems.
problem Modeling highly nonlinear dynamical systems with uncertainty quantification.
method Compositions of physics-informed random features derived from ODEs, combined with deep Gaussian processes and approximate Bayesian inference.
result The model effectively captures nonlinear behavior in real-world multivariate time series data and achieves comparable performance to other models on benchmark tasks.
Optimized nonlinearities enhance generalization in random feature models.
problem Improving generalization performance in random feature models.
method Analyzed and defined optimal nonlinearities from Gaussian model parameters.
result Optimized nonlinearities achieve better generalization performance than ReLU.
A new method for efficient nonlinear process monitoring using random Bernoulli features.
problem High computational demands and real-time responsiveness in online monitoring systems.
method Random Bernoulli principal component analysis to capture nonlinear patterns efficiently.
result The proposed methods offer excellent scalability and reduced computational complexity.
Neural networks learn task-specific features, influenced by nonlinearity.
problem Understanding the nature of task-dependent feature learning in neural networks.
method Investigation of fully-connected, wide neural networks using Bayesian framework.
result The nature of internal representations depends on neuronal nonlinearity, leading to analog, redundant, or sparse coding schemes.
For various applications, the relations between the dependent and independent variables are highly nonlinear. Consequently, for large scale complex problems, neural networks and regression trees are commonly preferred over linear models such as Lasso. This work proposes learning the feature nonlinearities by binning fe…
We propose a new method for input variable selection in nonlinear regression. The method is embedded into a kernel regression machine that can model general nonlinear functions, not being a priori limited to additive models. This is the first kernel-based variable selection method applicable to large datasets. It sides…
We simplify complex regression coefficients using linearization and feature comparison.
problem Interpreting high-dimensional regression coefficients from nonlinear responses.
method Developed a linearization method to derive feature coefficients and compare them with regression coefficients.
result Shows how regression coefficients relate to linearized feature coefficients and how they change under regularization.
Kernel methods are powerful tools to capture nonlinear patterns behind data. They implicitly learn high (even infinite) dimensional nonlinear features in the Reproducing Kernel Hilbert Space (RKHS) while making the computation tractable by leveraging the kernel trick. Classic kernel methods learn a single layer of nonl…
The approximation of nonlinear kernels via linear feature maps has recently gained interest due to their applications in reducing the training and testing time of kernel-based learning algorithms. Current random projection methods avoid the curse of dimensionality by embedding the nonlinear feature space into a low dim…
New approach uses secants to improve sensor placement and feature selection for nonlinear systems.
problem Inadequacy of linear methods for minimal sensor placement and feature selection in nonlinear systems.
method Data-driven approach using secant vectors to develop greedy algorithms for robust, near-minimal reconstruction guarantees.
result Demonstrated on two problems where linear techniques fail, secant-based approach provides robust solutions.
The visibility transformation embeds data position into signature features for efficient pattern recognition.
problem Embedding absolute position into signature features for efficient pattern recognition.
method The visibility transformation is put on a theoretical footing and used to embed absolute position into signature features efficiently.
result The generated feature set simplifies pattern recognition by accommodating nonlinear functions of absolute and relative values.
Private method measures nonlinear correlations between data hosted across two entities.
problem Measuring nonlinear correlations between sensitive data hosted across multiple parties while preserving privacy.
method Differentially private estimator of distance correlation.
result First private estimator of nonlinear correlations in a multi-party setup.
SISR improves feature attribution in complex payoff schemes.
problem Distorted feature attributions due to non-additive payoff functions and high-dimensional feature spaces.
method Sparse Isotonic Shapley Regression (SISR) learns a monotonic transformation to restore additivity and enforces L0 sparsity.
result SISR achieves strong support recovery and stable attributions across various payoff schemes.
Theory explains how deep nets learn features from data.
problem Understanding how deep neural networks learn features from data.
method Developed a noise-nonlinearity phase diagram and a mechanical theory.
result Links feature learning across layers to generalization.
A novel feature selection method for SVM improves model accuracy and interpretability.
problem Feature selection in nonlinear SVM classification problems.
method Embedded min-max optimization problem, leveraging duality theory.
result Improves model accuracy and interpretability on benchmark data sets.
CDSSL improves representation quality by integrating linear and nonlinear dependencies.
problem Scarcity of labeled data and neglect of nonlinear dependencies in SSL.
method CDSSL combines linear correlations and nonlinear dependencies using HSIC in RKHS.
result CDSSL enhances representation quality on diverse benchmarks.
Machine learning methods are used to discover complex nonlinear relationships in biological and medical data. However, sophisticated learning models are computationally unfeasible for data with millions of features. Here we introduce the first feature selection method for nonlinear learning problems that can scale up t…
Nonlinear independent component analysis (ICA) provides an appealing framework for unsupervised feature learning, but the models proposed so far are not identifiable. Here, we first propose a new intuitive principle of unsupervised deep learning from time series which uses the nonstationary structure of the data. Our l…
This paper surveys various methods for dimensionality reduction and nearest neighbor search.
problem Efficiently reducing high-dimensional data to lower dimensions while preserving essential information.
method Linear and nonlinear random projections, including sparse random projections, random Fourier Features, and Random Kitchen Sinks.
result Various methods for dimensionality reduction and nearest neighbor search are explained and compared.
FEALM learns features for better nonlinear DR of hidden patterns.
problem DR misses important patterns on distorted manifolds.
method FEALM generates optimized projections using an optimization algorithm and neighbor-shape dissimilarity.
result FEALM captures important patterns on hidden manifolds.
Three RFF-based methods for nonlinear causal discovery in mixed data.
problem Nonlinear causal discovery in mixed data with computational constraints.
method FFML, TRFF, and FFCI methods for score-based, constraint-based, and hybrid causal discovery.
result FFML and TRFF methods provide complementary performance in causal discovery.
New models improve machine learning accuracy and transparency in finance.
problem Black-box machine learning models lack interpretability in regulated industries.
method Introducing generalized groves of neural additive models with clear feature categories and interactions.
result Generalized groves of neural additive models achieve high accuracy with predominantly linear and sparse nonlinear components.
Robust method learns nonlinear structures robustly to noise.
problem Learning nonlinear structures in noisy data.
method Robust Non-Linear Matrix Factorization (RNLMF).
result RNLMF achieves noticeable improvements in denoising and clustering.
Random features enhance control of complex systems.
problem Flexible nonlinear models for control-affine systems.
method Random features approximations for control-affine structure.
result Methods formalized and shown to relate to ADP and AD kernels.
Random feature maps are ubiquitous in modern statistical machine learning, where they generalize random projections by means of powerful, yet often difficult to analyze nonlinear operators. In this paper, we leverage the "concentration" phenomenon induced by random matrix theory to perform a spectral analysis on the Gr…
A conjecturing machine uncovers feature relationships in data.
problem Discovering nonlinear and boolean relationships among features.
method Proposes a conjecturing machine that suggests feature relationships using bounds and expressions.
result Reveals true underlying relationships in datasets.
Classification and regression in which the inputs are graphs of arbitrary size and shape have been paid attention in various fields such as computational chemistry and bioinformatics. Subgraph indicators are often used as the most fundamental features, but the number of possible subgraph patterns are intractably large …
This paper selects features in deep neural networks with theoretical guarantees.
problem Feature selection in deep neural networks with unknown nonlinear functions.
method Reformulate neural networks as index models, estimate feature sets using Stein's formula, and apply screening-and-selection mechanism.
result Consistent feature selection with theoretical guarantees, even in high-dimensional settings.
GraphLIME explains GNN models by selecting key features locally.
problem Explaining the effectiveness of GNN models is challenging due to complex nonlinear transformations.
method GraphLIME uses HSIC Lasso for nonlinear feature selection in GNN models.
result GraphLIME provides more descriptive explanations than existing methods.
Novel time series forecasting method using sliding window signatures.
problem Challenges in forecasting nonlinear and delayed time series data.
method Ridge regression with signature features calculated on sliding windows.
result Signature features effectively encode temporal and nonlinear dependencies, leading to accurate forecasts.
We consider the binary classification problem when data are large and subject to unknown but bounded uncertainties. We address the problem by formulating the nonlinear support vector machine training problem with robust optimization. To do so, we analyze and propose two bounding schemes for uncertainties associated to …
Flexible model for complex relationships using Bayesian nonparametrics.
problem Complex relationships between variables not well captured by simple models.
method Hierarchical generation of nonlinear features, Bayesian inference, variable selection.
result Find interpretable models with a small set of important features.
DFIV uses deep neural nets to learn nonlinear features in IV regression.
problem Learning causal relationships from observational data with nonlinear interactions.
method DFIV trains deep neural nets to define nonlinear features on instruments and treatments, alternating training to compose stages 1 and 2.
result DFIV outperforms state-of-the-art methods on IV benchmarks and off-policy policy evaluation.
KCRL learns stable policies for nonlinear systems with formal guarantees.
problem Lack of stabilization guarantees in RL methods for safety-critical systems.
method KCRL uses Krasovskii's Lyapunov functions as a stability constraint and a primal-dual approach to learn stabilizing policies.
result KCRL guarantees learning a stabilizing policy in a finite number of interactions.
Letter analyzes training dynamics of a nonlinear contrastive learning model in high dimensions.
problem Understanding training dynamics of nonlinear contrastive learning models in high-dimensional settings.
method High-dimensional analysis using McKean-Vlasov PDEs and low-dimensional ODEs.
result The model's performance evolves according to specific ODEs, revealing features like feature learnability and noise effects.
Active learning method estimates nonlinear systems efficiently.
problem Identifying nonlinear dynamical systems with continuous states and actions.
method Repeating three steps: trajectory planning, tracking, and re-estimation.
result Estimates nonlinear dynamical systems at a parametric rate.
Study shows mixtures of nonlinearities can improve deep learning performance.
problem Improving deep learning performance with large datasets and complex models.
method Analyzed random feature regression with features F=f(WX+B) for a random weight matrix W and random bias vector B. result Mixture of nonlinearities can improve both training and test errors over a single nonlinearity.
RFMs transition from linear to nonlinear under specific input-label correlation.
problem Understanding the transition from linear to nonlinear behavior in RFMs.
method Analyzing RFMs under spiked covariance designs, characterizing the interaction between anisotropy and input-label correlation.
result The RFM generalization error is governed by the strength of input-label correlation, leading to a clear nonlinear advantage above a specific boundary.
Classical methods such as Principal Component Analysis (PCA) and Canonical Correlation Analysis (CCA) are ubiquitous in statistics. However, these techniques are only able to reveal linear relationships in data. Although nonlinear variants of PCA and CCA have been proposed, these are computationally prohibitive in the …
Machine Learning improves macroeconomic forecasting by capturing nonlinearities.
problem Improving macroeconomic forecasting accuracy.
method Study four features (nonlinearities, regularization, cross-validation, loss function) in data-rich and data-poor environments.
result Nonlinearity is the key to improving forecasting accuracy.
New method selects direct causal parents from large sets of variables.
problem Inferring direct causal parents from many variables, especially nonlinear and cyclic.
method One-vs.-the-rest feature selection approach with theoretical guarantees.
result Significant improvements over existing methods.
Mamba efficiently learns low-dimensional targets in-context via feature extraction.
problem Learning low-dimensional targets in context for computational efficiency.
method Test-time feature learning of a single-index model using Mamba's pretrained linear-time sequence model.
result Mamba achieves efficient in-context learning of low-dimensional targets via feature extraction.
Study of eigenvalues in nonlinear kernels for classification of separable data.
problem Understanding the applicability of linear equivalents in nonlinearly separable data classification.
method Analysis of conjugate kernels and their quadratic equivalents for a canonical nonlinearly separable dataset (XOR problem).
result Identification of regimes where nonlinear kernels deviate from linear equivalents, leading to label-aligned eigenspaces.
This work includes all the technical details of the Sequential Principal Curves Analysis (SPCA) in a single document. SPCA is an unsupervised nonlinear and invertible feature extraction technique. The identified curvilinear features can be interpreted as a set of nonlinear sensors: the response of each sensor is the pr…
Kernel discriminant analysis uses nonlinear embeddings to improve classification.
problem Limited effectiveness of linear discriminant analysis in capturing nonlinear features.
method Study of nonlinear embeddings in kernel discriminant analysis using polynomial and Gaussian kernels, solving generalized eigenvalue problems.
result Polynomial and Gaussian discriminants capture class differences through population moments and randomized projections.
Multilayer bootstrap network builds a gradually narrowed multilayer nonlinear network from bottom up for unsupervised nonlinear dimensionality reduction. Each layer of the network is a nonparametric density estimator. It consists of a group of k-centroids clusterings. Each clustering randomly selects data points with r…