We propose semi-random features for nonlinear function approximation. The flexibility of semi-random feature lies between the fully adjustable units in deep learning and the random features used in kernel methods. For one hidden layer models with semi-random features, we prove with no unrealistic assumptions that the m…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,657 papers · 148 categories
Trend · papers per month
48 results for “Random Feature Models”
We study the approximation properties of random ReLU features through their reproducing kernel Hilbert space (RKHS). We first prove a universality theorem for the RKHS induced by random features whose feature maps are of the form of nodes in neural networks. The universality result implies that the random ReLU features…
A new model combines diffusion and random features for better interpretability and comparable performance.
problem Lack of theoretical justification and computational expense in diffusion models, and limited interpretability in random feature models.
method Developed a deep random feature model inspired by diffusion models, derived generalization bounds using score matching.
result The model achieves comparable performance to fully connected neural networks and provides theoretical generalization bounds.
One-bit quantization improves inference speed for Random Features models.
problem Efficient inference on resource-constrained devices.
method Analysis of one-bit quantization in Random Features model.
result Asymptotically, quantizing weights except the last incurs no loss in generalization error.
HARFE approximates sparse additive functions using random features and ridge regression.
problem Approximating high-dimensional sparse additive functions.
method Hard-ridge random feature expansion with sparse ridge regression and hard-thresholding pursuit.
result HARFE method converges with a given error bound and achieves lower error than other algorithms.
Random feature models approximate functions in Banach spaces efficiently.
problem Approximating functions in Banach spaces efficiently.
method Randomly initialized feature maps and linear readout training.
result Universal approximation in Bochner spaces for Banach space-valued models.
A novel model uses ODE-based random features to model nonlinear dynamical systems.
problem Modeling highly nonlinear dynamical systems with uncertainty quantification.
method Compositions of physics-informed random features derived from ODEs, combined with deep Gaussian processes and approximate Bayesian inference.
result The model effectively captures nonlinear behavior in real-world multivariate time series data and achieves comparable performance to other models on benchmark tasks.
New method improves feature importance assessment in random forests.
problem Improving feature importance measures for random forests.
method Hypothesis testing via self-normalized feature-residual correlation test (FACT).
result The method provides theoretically justified feature importance tests with controlled type I error and appealing power.
Sparse random features improve accuracy in data-scarce settings.
problem Limited accuracy of random feature methods in data-scarce applications.
method Sparse random feature expansion using compressive sensing.
result Improved generalization bounds for sparse random features.
Orthogonal random features approximate a Bessel kernel, offering sharper bounds than random Fourier features.
problem Approximating Gaussian kernel efficiently for large datasets.
method Use of Haar orthogonal matrices to construct orthogonal random features and analyze their bias and variance.
result Orthogonal random features approximate a Bessel kernel, not the Gaussian kernel, with sharper bounds.
The paper studies multiple descent in multi-component prediction models.
problem Understanding the risk curves in multi-component prediction models.
method Investigates a 'double random feature model' and 'multiple random feature model' in ridge regression.
result Risk curves of multi-component prediction models can exhibit multiple descents.
Generative model uses random convolutional features to create financial time series.
problem Generating realistic financial time series with limited data and avoiding overfitting.
method Train generators by matching random convolutional features of real and generated time series, using SOCK (SOft Competing Kernels) feature map.
result Generators trained with random SOCK features outperform baselines across various financial datasets.
Analyzes bias-variance in overparameterized linear models using random features.
problem Understanding bias-variance trade-off in overparameterized models.
method Zero-temperature cavity method and random matrix theory.
result Three phase transitions in the linear random features model.
Study loop corrections in random feature models affecting training and test errors.
problem Analyzing loop corrections in random feature models to understand training and test errors.
method Statistical physics and effective field theory approach to study loop corrections.
result Derived loop corrections to training error, test error, and generalization gap.
Random feature models can outperform a weak teacher with early stopping.
problem Generalization from a weak to a strong model in random feature networks.
method Random feature models, early stopping, proving weak-to-strong generalization.
result Random feature models can outperform a weak teacher with early stopping.
New method uses sparse random features for crashworthiness analysis.
problem Efficient surrogate modelling for uncertainty quantification.
method Sparse Random Features combined with self-supervised dimensionality reduction.
result Superiority over state-of-the-art techniques in crashworthiness analysis.
Develops a new method for nonlinear dimension reduction using random features.
problem Statistical challenges in generalizing Gaussian process-based latent variable models to non-Gaussian data.
method Random feature latent variable models (RFLVMs) that approximate nonlinear relationships with linear functions of random features.
result RFLVMs produce comparable results to state-of-the-art methods on various data types.
Deep random feature models are analyzed for their performance with exact asymptotic expressions.
problem Understanding the performance of deep random feature models.
method Established a novel universality result and used the convex Gaussian Min-Max theorem.
result Exact asymptotic expressions for the performance of deep random feature models are derived.
The randomized-feature approach has been successfully employed in large-scale kernel approximation and supervised learning. The distribution from which the random features are drawn impacts the number of features required to efficiently perform a learning task. Recently, it has been shown that employing data-dependent …
Study compares random and learned features in deep Bayesian linear models.
problem Understanding how feature learning affects generalization in deep learning.
method Comparing deep random feature models to deep networks with trained layers.
result Random feature models can display double-descent behavior, while deep networks do not.
Random feature maps are ubiquitous in modern statistical machine learning, where they generalize random projections by means of powerful, yet often difficult to analyze nonlinear operators. In this paper, we leverage the "concentration" phenomenon induced by random matrix theory to perform a spectral analysis on the Gr…
Framework combines random features with CDEs for efficient time-series learning.
problem Efficient training of time-series models with strong inductive bias.
method Random Fourier CDEs and Random Rough DEs using continuous-time reservoirs and log-ODE discretization.
result Unified perspective on random-feature reservoirs and path-signature theory.
Random Fourier features model reconstructs wind fields from sparse measurements.
problem Reconstructing wind fields from limited data.
method Random Fourier features approximating velocity field with adaptive sampling.
result Random Fourier features model outperforms benchmarks.
Random feature matrices' singular values concentrate near their full expectation in high dimensions.
problem Characterizing the spectra of random feature matrices for regression problems.
method Analyzing two settings of input variables (random or well-separated) with conditions on dimension, complexity ratio, and sampling variance.
result The singular values of random feature matrices concentrate near their full expectation and near one with high probability.
A new feature selection method using random forest and Kolmogorov filter.
problem Ultra-high dimensional data feature selection.
method Fused Kolmogorov filter with random forest based recursive feature elimination.
result Selection and consistency under weak conditions.
Model, sample, and epoch-wise descents: exact solution of gradient flow in the random feature modelcs.LG
Analyzes the generalization and training errors of the random feature model over time.
problem Understanding the temporal behavior of generalization and training errors in deep learning.
method Uses Cauchy complex integral representations and random matrix methods based on linear pencils.
result Analytical solution of the full time-evolution path of generalization and training errors.
Two ANOVA-based algorithms boost random Fourier feature models for function approximation.
problem Approximating high-dimensional functions with low-order interactions.
method Utilizes ANOVA decomposition to learn low-order functions and index sets of important variables.
result Significantly reduces approximation error compared to existing methods.
Enhances random forest performance with exogenous randomness.
problem Improving random forest performance through exogenous randomness.
method Developed non-asymptotic MSE expansions for individual trees and forests, identified two types of randomness, and conducted simulations.
result Exogenous randomness, particularly feature subsampling, reduces both bias and variance of random forests.
Improves efficiency of random feature approximations for dot product kernels.
problem Efficiency of random feature approximations for dot product kernels.
method Generalization of existing random feature approximations using complex-valued random features, theoretical analysis of variances, data-driven optimization approach.
result Complex-valued random features can significantly reduce the variances of approximations.
Random matrix theory explains how neural networks adapt to data.
problem Understanding how neural networks learn and generalize from data.
method Random matrix analysis of two-layer neural networks.
result Sharp characterization of feature spectrum and generalization error.
Random features and KRR generalize similarly when N is large enough.
problem Understanding the generalization error of random features and KRR methods.
method Analyzing spectral conditions and hypercontractivity on kernel eigenfunctions.
result The test error of random features is larger than KRR when N is small, but they achieve the same error when N is large.
A new method for measuring conditional feature importance using generative models.
problem Challenges in evaluating feature importance given other feature values.
method Adversarial Random Forest (ARF) for generating on-manifold data points.
result cARFi method yields robust importance scores adaptable for various feature importance notions.
Survey on random features for kernel approximation, focusing on algorithms, theory, and practical applications.
problem Efficiently approximating kernel methods for large-scale problems.
method Random features techniques to speed up kernel methods.
result Need for a high number of random features for good approximation quality.
TRF uses ternary random features to improve ML performance without extra computation.
problem Improving ML performance with less computation and storage.
method Proposes Ternary Random Features (TRF) for random features compression.
result TRF asymptotically yields the same limiting kernel as original matrices, with improved efficiency.
Random feature model approximates PDE solutions efficiently.
problem Approximating solutions to PDEs with high-dimensional inputs and outputs.
method Random feature model applied to infinite-dimensional operators.
result Efficient and accurate approximation of PDE solutions.
Sparser Random Feature Models via IMP (ShRIMP) efficiently learns sparse models for high-dimensional data.
problem Learning sparse models for high-dimensional data with sparse variable dependencies.
method Iterative Magnitude Pruning applied to Random Feature Models.
result ShRIMP achieves better or competitive test accuracy compared to state-of-the-art methods.
Random feature model shows slow self-correction of generalization gap.
problem Slow deterioration of generalization error in random feature model.
method Examined the dynamic behavior of gradient descent in the model's resonance regime.
result Gradient descent exhibits a self-correction mechanism, reducing generalization gap over time.
Random feature maps improve forecasting with cheaper computation.
problem Improving forecasting accuracy with random feature maps.
method Developed a hit-and-run algorithm to select optimal internal weights.
result Optimal internal weights lead to superior forecasting skill.
Attention layers are sensitive to single words, improving generalization over random features.
problem Understanding why attention layers are effective in NLP tasks.
method Study of word sensitivity in random features using BERT-Base word embeddings.
result Attention layers have high word sensitivity, improving generalization over random features.
We analyze learning curves of RF models with convex regularization and derive precise asymptotic expressions.
problem Understanding the learning curves of RF models with general convex regularization.
method Novel multi-level application of the convex Gaussian min max theorem (CGMT) to compute precise asymptotic expressions.
result Precise asymptotic expressions for learning curves of RF models with separable strongly convex regularization or regularization.
CTRF combines logged data and randomized experiments for robust prediction.
problem Robust prediction models to handle distributional shifts between training and testing data.
method CTRF uses existing training data and a small amount of randomized experiment data to train a robust model.
result CTRF produces robust predictions and outperforms baseline methods in the presence of feature shifts.
Kernel ridgeless regression with random features shows good generalization without explicit regularization.
problem Generalization of kernel ridgeless regression without explicit regularization.
method Investigation of ridgeless regression with random features and stochastic gradient descent, exploring the effect of random features error and spectral density optimization.
result Random features error exhibits the double-descent curve, leading to improved generalization.
RFRBoost uses random features to boost deep residual neural networks, improving performance and computational efficiency.
problem Improving performance of deep residual neural networks (RFNNs) while preserving convex optimization benefits.
method Random Feature Representation Boosting (RFRBoost) using boosting theory and random features at each layer.
result RFRBoost significantly outperforms RFNNs and end-to-end trained MLP ResNets in small- to medium-scale tabular datasets.
Paper analyzes error bounds for learning with vector-valued RF, improving existing analyses.
problem Learning with vector-valued random features in infinite-dimensional settings.
method Direct analysis of risk functional, avoiding random matrix theory.
result Strong consistency and minimax optimal convergence rates established.
New insights into how randomization affects greedy model selection.
problem Understanding the impact of feature subsampling on greedy model selection.
method Investigated greedy forward selection with feature subsampling, proving effects on bias and variance.
result Ensembling with feature subsampling reduces both bias and variance, unlike convex base learners.
Graph neural networks (GNNs) are powerful machine learning models for various graph learning tasks. Recently, the limitations of the expressive power of various GNN models have been revealed. For example, GNNs cannot distinguish some non-isomorphic graphs and they cannot learn efficient graph algorithms. In this paper,…
Paper introduces a method for operator learning using random features.
problem Estimating maps between infinite-dimensional spaces using input-output pairs.
method Function-valued random features method, building a linear combination of random operators.
result The method provides convergence guarantees and error bounds for nonlinear problems.
Paper extends RUMs with features to handle incomplete preferences and proves identifiability.
problem Learning preferences from incomplete and possibly noisy data.
method Generalized Mixture of Random Utility Models with features, proving identifiability and consistency of MLE.
result Proves consistency of Maximum Likelihood Estimation (MLE) for PL with features under certain conditions.