Kernel ridgeless regression with random features shows good generalization without explicit regularization.
problem Generalization of kernel ridgeless regression without explicit regularization.
method Investigation of ridgeless regression with random features and stochastic gradient descent, exploring the effect of random features error and spectral density optimization.
result Random features error exhibits the double-descent curve, leading to improved generalization.
Study shows topological features improve time series classification.
problem Classifying stochastic processes with varying noise and sampling.
method Topological data analysis features compared to statistical and raw features.
result Topological features lead to better classification performance.
This work extends neural networks to automatically select features by stochastically penalizing feature involvement.
problem Feature selection in machine learning models.
method Stochastic regularization to select features instead of layer weights.
result Superior efficiency compared to classical methods with minimal computational overhead.
Study on test risk dynamics in learning theory with stochastic gradient flow.
problem Understanding test risk in stochastic gradient flow dynamics.
method Path integral formulation for small learning rates, explicit computation for weak features.
result Explicit corrections due to stochastic term in dynamics, good agreement with simulations.
New method selects features for sequential decision making.
problem Dynamic feature selection for instance-wise decisions.
method Latent variable model trained in a supervised manner; reasoning across stochastic latent space.
result Outperforms existing methods on various datasets.
Binary Stochastic Filtering (BSF), the algorithm for feature selection and neuron pruning is proposed in this work. The method defines filtering layer which penalizes amount of the information involved in the training process. This information could be the input data or output of the previous layer, which directly lead…
A neural network approach for feature selection using mutual information.
problem Feature ranking and selection leading to sub-optimal solutions for class separability.
method Stochastic mutual information gradient estimation for dimensionality reduction.
result The network projects features onto an output space maximizing mutual information with class labels.
We consider the binary classification problem when data are large and subject to unknown but bounded uncertainties. We address the problem by formulating the nonlinear support vector machine training problem with robust optimization. To do so, we analyze and propose two bounding schemes for uncertainties associated to …
Deep neural networks solve stochastic control problems with delay.
problem Challenges in stochastic control problems with delay due to path-dependence and high dimensions.
method Employing recurrent neural networks (RNNs) to parameterize policies and optimize objectives.
result RNNs, especially LSTMs, efficiently capture path-dependence and outperform feedforward networks in training and performance.
RoSHAP stabilizes feature attribution in machine learning models.
problem Stochastic variation in feature attribution measures.
method Modeling feature attribution score distribution and estimating it through bootstrap resampling and kernel density estimation.
result RoSHAP provides stable feature rankings and improves model performance.
SGD in DLNs reveals feature learning dynamics.
problem Understanding SGD dynamics in DLNs during saddle-to-saddle training.
method Stochastic Langevin dynamics with anisotropic, state-dependent noise; one-dimensional per-mode SDEs; Boltzmann distribution approximation.
result SGD noise encodes feature learning progression but does not alter saddle-to-saddle dynamics.
Contrastive learning struggles with class collapse and feature suppression, revealing bias towards simpler solutions.
problem Contrastive learning struggles with class collapse and feature suppression, especially in supervised and unsupervised settings.
method Unified theoretical framework to determine which features are learnt by CL, revealing bias towards simpler solutions.
result Bias towards simpler solutions is a key factor in class collapse and feature suppression.
Feature selection (FS) has become an indispensable task in dealing with today's highly complex pattern recognition problems with massive number of features. In this study, we propose a new wrapper approach for FS based on binary simultaneous perturbation stochastic approximation (BSPSA). This pseudo-gradient descent st…
Unified derivation of high-dimensional linear models using stochastic gradient descent.
problem Performance analysis of high-dimensional linear models trained with stochastic gradient descent.
method Derivation of a deterministic equivalence for the two-point function of a random matrix resolvent.
result Unified understanding of model performance including previously known and novel results.
Study predicts SGD test loss for structured features.
problem Understanding test loss dynamics in SGD for structured data.
method Solveable model of SGD on mean square loss for arbitrary covariance structure.
result Simpler Gaussian model accurately predicts test loss of nonlinear models.
Paper analyzes TD(λ) convergence rates for arbitrary features.
problem Convergence rates for linear TD(λ) under arbitrary features. method Developed a novel stochastic approximation result for arbitrary features.
result Established L2 convergence rates for linear TD(λ) without linearly independent features assumption. Study reveals conditions for neural networks to forget learned features.
problem Understanding feature unlearning in neural networks.
method Infinite-width limit analysis with stochastic gradient descent, fast-slow dynamics.
result Conditions for feature unlearning are determined by the strength of nonlinear terms and initial weights.
Study models deep learning training dynamics using locally elastic SDEs to reveal feature separability.
problem Understanding how deep learning models separate features from different classes during training.
method Modeling deep learning training using locally elastic SDEs with a drift term reflecting backpropagation impact.
result Local elasticity in SDEs leads to linear separability of features, resulting in vanishing training loss.
As the size of modern data sets exceeds the disk and memory capacities of a single computer, machine learning practitioners have resorted to parallel and distributed computing. Given that optimization is one of the pillars of machine learning and predictive modeling, distributed optimization methods have recently garne…
SIM-Shapley improves SV approximation efficiency and stability.
problem High computational costs of Shapley value methods in high-dimensional settings.
method Stochastic Iterative Momentum for Shapley Value Approximation (SIM-Shapley).
result Reduced computation time by up to 85% while maintaining feature attribution quality.
There is a long history of using meta learning as representation learning, specifically for determining the relevance of inputs. In this paper, we examine an instance of meta-learning in which feature relevance is learned by adapting step size parameters of stochastic gradient descent---building on a variety of prior w…
Quantum computing improves feature selection in machine learning.
problem Optimizing feature selection in machine learning problems.
method Formulated feature selection as a QUBO problem and compared quantum and classical methods.
result Quantum computing can outperform classical methods in feature selection, depending on data set.
Feature attribution methods, or saliency maps, are one of the most popular approaches for explaining the decisions of complex machine learning models such as deep neural networks. In this study, we propose a stochastic optimization approach for the perturbation-based feature attribution method. While the original optim…
This paper explains CART random forests using stochastic control theory.
problem Understanding the inner workings of CART random forests.
method Developed a stochastic-control perspective on CART random forests, interpreting feature subsampling as a random feasible action set and the split rule as a policy.
result Established that the CART policy is locally stabilizing but globally suboptimal for the forest objective.
Large SGD step sizes lead to sparse feature learning in neural networks.
problem Sparse feature learning in neural networks with large step sizes.
method Empirical observations and theoretical analysis of SGD dynamics.
result Large step sizes induce implicit regularization leading to sparse predictors.
New algorithm tackles nested bi-level optimization problems for robust feature learning.
problem Nested compositional bi-level optimization problems in machine learning.
method Stochastic approximation algorithms for solving nested compositional bi-level optimization problems without matrix inversions.
result Achieves an ε-stationary solution with an oracle complexity of approximately O_T(1/ε^2).
Enhances DGPs with adaptive RKHS Fourier features for better non-stationary pattern modeling.
problem Capturing complex non-stationary patterns in non-linear dynamical systems.
method Integrates ODE-based RKHS Fourier features into DGPs using convolution operations for adaptive amplitude and phase modulation. Uses a doubly stochastic variational inference framework.
result Improved predictive performance across various regression tasks.
This paper explores a new form of the linear bandit problem in which the algorithm receives the usual stochastic rewards as well as stochastic feedback about which features are relevant to the rewards, the latter feedback being the novel aspect. The focus of this paper is the development of new theory and algorithms fo…
Stochastic zeroth-order (SZO), or gradient-free, optimization allows to optimize arbitrary functions by relying only on function evaluations under parameter perturbations, however, the iteration complexity of SZO methods suffers a factor proportional to the dimensionality of the perturbed function. We show that in scen…
Large learning rates cause oscillations in NN weights that improve generalization.
problem Improving generalization of neural networks trained with large learning rates.
method Theoretical analysis and feature-noise data generation model.
result Oscillating SGD with large learning rates benefits NN generalization by effectively learning weak features.
FDSKL algorithm trains vertically partitioned data with kernels securely and efficiently.
problem Training vertically partitioned data with kernels while maintaining privacy.
method FDSKL algorithm using random features and doubly stochastic gradients for federated learning.
result FDSKL achieves sublinear convergence and guarantees data security.
Although kernel methods are widely used in many learning problems, they have poor scalability to large datasets. To address this problem, sketching and stochastic gradient methods are the most commonly used techniques to derive efficient large-scale learning algorithms. In this study, we consider solving a binary class…
Improves posterior approximation speed for Dirichlet process mixture models.
problem Inefficiency of stochastic variational inference in large datasets.
method Uses stochastic gradient ascent with adaptive stepsize optimization.
result Adaptive stepsize improves speed and performance of posterior approximation.
Proposes efficient Bayesian logistic regression for large sparse datasets.
problem Infeasibility of theoretical Bayesian methods for large sparse feature sets.
method Low complexity analytical approximations for sparse online logistic and probit regressions.
result Empirical results show superior performance compared to more complex methods.
In risk management it is desirable to grasp the essential statistical features of a time series representing a risk factor. This tutorial aims to introduce a number of different stochastic processes that can help in grasping the essential features of risk factors describing different asset classes or behaviors. This pa…
New robust algorithms improve learning with feature feedback.
problem Interactive learning with discriminative feature feedback.
method Developed new robust interactive learning algorithms with improved mistake bounds.
result Achieved significantly lower mistake bounds in adversarial and stochastic settings.
Consider a two-player zero-sum stochastic game where the transition function can be embedded in a given feature space. We propose a two-player Q-learning algorithm for approximating the Nash equilibrium strategy via sampling. The algorithm is shown to find an ε-optimal strategy using sample size linear to the number …
Sketching and stochastic gradient methods are arguably the most common techniques to derive efficient large scale learning algorithms. In this paper, we investigate their application in the context of nonparametric statistical learning. More precisely, we study the estimator defined by stochastic gradient with mini bat…
We propose a single neural probabilistic model based on variational autoencoder that can be conditioned on an arbitrary subset of observed features and then sample the remaining features in "one shot". The features may be both real-valued and categorical. Training of the model is performed by stochastic variational Bay…
Recently, Convolution Neural Networks (CNNs) obtained huge success in numerous vision tasks. In particular, DenseNets have demonstrated that feature reuse via dense skip connections can effectively alleviate the difficulty of training very deep networks and that reusing features generated by the initial layers in all s…
Study online linear regression with paid noise reduction.
problem Online linear regression with noisy features and the ability to pay for reduced noise.
method Analyzes regret against optimal predictor, uses matrix martingale concentration.
result Optimal regret rates for known and unknown noise covariance.
Leveraging advances in variational inference, we propose to enhance recurrent neural networks with latent variables, resulting in Stochastic Recurrent Networks (STORNs). The model i) can be trained with stochastic gradient methods, ii) allows structured and multi-modal conditionals at each time step, iii) features a re…
Sliced Inverse Regression reduces parameter space for estimating complex financial models.
problem High-dimensional parameter space in stochastic differential equations.
method Sliced Inverse Regression for dimension reduction.
result Reduced computational costs in estimating parameters.
Stochasticity is key for machine learning's robustness and generalizability.
problem Machine learning's need for robustness and generalizability.
method Review of ML literature and biological intelligence.
result Stochasticity is a critical ingredient for intelligent systems in ML.
Deep neural networks are a powerful tool for feature learning and extraction given their ability to model high-level abstractions in highly complex data. One area worth exploring in feature learning and extraction using deep neural networks is efficient neural connectivity formation for faster feature learning and extr…
We develop a general term structure framework taking stochastic discontinuities explicitly into account. Stochastic discontinuities are a key feature in interest rate markets, as for example the jumps of the term structures in correspondence to monetary policy meetings of the ECB show. We provide a general analysis of …
Two algorithms achieve optimal logarithmic regret in feature-based dynamic pricing.
problem Optimal pricing for products with features based on online learning.
method Developed EMLP and ONSP algorithms for stochastic and adversarial settings, proving O(dlogT) regret bounds. result Achieved optimal O(dlogT) regret, improving over existing bounds. Optimizes pruning masks for neural networks using probabilistic fine-tuning and PAC-Bayes bounds.
problem Improving neural network performance through adaptive pruning of weights.
method Optimizes stochastic pruning masks by minimizing expected loss, considering data-adaptive regularization and feature alignment.
result Probabilistic fine-tuning leads to improved test error over baseline methods in neural networks.