Scaling feature values is an important step in numerous machine learning tasks. Different features can have different value ranges and some form of a feature scaling is often required in order to learn an accurate classifier. However, feature scaling is conducted as a preprocessing task prior to learning. This is probl…
On-device research index
arXiv research
A locally-built, LLM-digested index of recent arXiv papers in quant finance, geometry/topology, and statistical ML — keyword search served straight from SQLite on this machine.
168,695 papers · 148 categories
Trend · papers per month
48 results for “feature scaling”
Model shows feature learning can improve neural scaling laws for hard tasks.
problem Understanding and improving neural network scaling laws for various task difficulties.
method Developed a solvable model of neural scaling laws, identified three scaling regimes, and demonstrated feature learning's impact on scaling exponents.
result Feature learning can improve scaling with training time and compute for hard tasks, nearly doubling the exponent.
This paper improves conditional multidimensional scaling for incomplete data.
problem Handling missing data in known features for multidimensional scaling.
method Proposes a method to learn low-dimensional configurations with missing known feature values.
result Can learn low-dimensional configurations and impute missing values.
Improves speaker verification for variable-duration utterances using a feature pyramid module.
problem Improving robustness for variable-duration utterances in speaker verification.
method Integrates a feature pyramid module into multi-scale aggregation to enhance speaker-discriminative information from multiple layers.
result Improves performance for both short and long utterances compared to state-of-the-art approaches.
Feature selection is an important challenge in machine learning. It plays a crucial role in the explainability of machine-driven decisions that are rapidly permeating throughout modern society. Unfortunately, the explosion in the size and dimensionality of real-world datasets poses a severe challenge to standard featur…
Irregular features disrupt the desired classification. In this paper, we consider aggressively modifying scales of features in the original space according to the label information to form well-separated clusters in low-dimensional space. The proposed method exploits spectral clustering to derive scaling factors that a…
This work bridges two views of feature learning in neural networks.
problem The relationship between kernel scale changes and data-adaptive feature learning in neural networks remains unresolved.
method Using statistical mechanics, the work derives analytical expressions for network output statistics across scaling regimes.
result Kernel adaptation can be reduced to an effective kernel rescaling, but multi-scale adaptive approach provides richer insights.
Sharp feature-learning transitions and Bayes-optimal neural scaling laws in extensive-width networksstat.ML
Sharp feature transitions revealed in extensive-width networks.
problem Learning hierarchical features from noisy queries in large networks.
method Information-theoretic analysis and heuristic decoupling argument.
result Sequential phase transitions in feature learnability and effective width.
Wide neural networks with asymmetrical node scaling converge globally and learn features.
problem Global convergence and feature learning in over-parameterised shallow networks.
method Gradient-based optimisation of wide, shallow neural networks with asymmetrical node scaling.
result Gradient flow and gradient descent converge to a global minimum and learn features, unlike in the NTK parameterisation.
A hierarchical model shows how scaling laws emerge from sequential feature recovery.
problem Emergence of scaling laws from feature learning in multi-layer networks.
method Layer-wise spectral algorithm adapted to compositional structure, sequential feature detection.
result Sequential detection of latent features, leading to explicit power-law decay of prediction error.
Spectral dimensionality reduction methods enable linear separations of complex data with high-dimensional features in a reduced space. However, these methods do not always give the desired results due to irregularities or uncertainties of the data. Thus, we consider aggressively modifying the scales of the features to …
The Impact of Feature Scaling In Machine Learning: Effects on Regression and Classification Taskscs.LG
This study evaluates feature scaling across 14 datasets and 12 techniques in ML.
problem Impact of feature scaling on machine learning performance and computational costs.
method Systematic evaluation of 12 scaling techniques across 14 datasets and 16 ML algorithms.
result Wide variation in model performance due to feature scaling, especially for non-ensemble models.
Image Super-Resolution with Cross-Scale Non-Local Attention and Exhaustive Self-Exemplars Miningcs.CV
This paper improves image super-resolution by integrating cross-scale non-local attention.
problem Improving image super-resolution by leveraging long-range and cross-scale feature correlations.
method Proposes a Cross-Scale Non-Local (CS-NL) attention module integrated into a recurrent neural network.
result Significantly improved performance on SISR benchmarks.
Nearest Neighbors Algorithm is a Lazy Learning Algorithm, in which the algorithm tries to approximate the predictions with the help of similar existing vectors in the training dataset. The predictions made by the K-Nearest Neighbors algorithm is based on averaging the target values of the spatial neighbors. The selecti…
Efficiently applies NTK to large-scale datasets using random features.
problem Computational limitations of kernel methods for large-scale datasets.
method Proposes a sketching-based algorithm combining random features of arc-cosine kernels to construct an efficient feature map of the NTK.
result Achieves comparable error bounds to exact kernel methods but with significantly reduced feature dimensionality.
Transformers can scale both context and task, but MLPs can only scale task.
problem Understanding and scaling In-Context Learning in transformers.
method Simplified transformer architecture, feature map, and MLP combination.
result Simplified transformer can perform ICL and context-scaling but not task-scaling.
Study shows optimal model performance at critical level of feature learning.
problem Catastrophic forgetting in neural networks, especially in non-stationary environments.
method Systematic study on model scale and feature learning, using dynamical mean field theory.
result Optimal performance achieved at a critical level of feature learning, dependent on task non-stationarity and model scale.
Large-scale Hierarchical Classification (HC) involves datasets consisting of thousands of classes and millions of training instances with high-dimensional features posing several big data challenges. Feature selection that aims to select the subset of discriminant features is an effective strategy to deal with large-sc…
DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systemscs.IR
DCN-V2 improves deep & cross network for web-scale learning to rank systems.
problem Efficiently learning feature interactions in large-scale recommender systems.
method Proposes DCN-V2, an improved framework for deep & cross network learning.
result DCN-V2 outperforms state-of-the-art algorithms on benchmark datasets.
Study analyzes seasonal hydroclimatic features across climates and continents.
problem Lack of seasonal hydroclimatic feature analysis for Koppen-Geiger climates and continents.
method Global-scale analysis of 13,000 time series using 7 features.
result Notable differences in feature magnitudes across Koppen-Geiger climate classes and continental regions.
Graph cross network improves graph classification accuracy.
problem Improving graph classification accuracy.
method Graph cross network (GXN) with vertex infomax pooling (VIPool) and feature-crossing layer.
result Improves graph classification accuracy by 2.12% and 1.15%.
Kernel methods are powerful and flexible approach to solve many problems in machine learning. Due to the pairwise evaluations in kernel methods, the complexity of kernel computation grows as the data size increases; thus the applicability of kernel methods is limited for large scale datasets. Random Fourier Features (R…
Paper compares ML methods for credit scoring, highlighting feature selection and scaling impacts.
problem Determining default risk in credit scoring models.
method Eight ML methods (SVM, Naive Bayes, DT, RF, XGBoost, KNN, MLP, LR) with feature selection and scaling.
result Feature selection and scaling improve model performance in credit scoring.
SignSGD outperforms SGD in linear regression with optimal scaling laws under PLRF model.
problem Improving linear regression performance with signSGD under power-law random features.
method Analysis of signSGD risk under PLRF model, comparison with SGD, identification of unique effects.
result SignSGD can have a steeper compute-optimal slope than SGD in noisy regimes, especially with WSD schedule.
New PINN architectures learn high-frequency features using Fourier features.
problem PINNs struggle with high-frequency or multi-scale features.
method Employ spatio-temporal and multi-scale random Fourier features.
result Effective PINN models for multi-scale PDEs.
MCFNet recovers spatial detail and fuses it with semantic information for real-time segmentation.
problem Recovering spatial detail information and fusing it with semantic information in real-time.
method Proposes a new architecture (MCFNet) with feature refinement and fusion modules, and a gating unit.
result Achieves competitive performance with high speed (75.5% mIOU, 151.3 FPS on Cityscapes).
Efficient knockoffs for large-scale feature selection.
problem Large-scale feature selection problems.
method Gaussian model-X knockoffs with efficient methods for solving semidefinite programs.
result Efficient knockoffs can be generated with linear complexity in the dimension.
Adaptive regularization prevents overfitting in large-scale sparse feature models.
problem Overfitting in models with large-scale sparse categorical features.
method Adaptive regularization of embedding layers' norm budget.
result Improves model performance within a single epoch and prevents multi-epoch performance degradation.
Ensembles of random-feature models can't outperform a single large model.
problem Finding the optimal balance between model size and ensemble size.
method Deterministic equivalent risk estimates and scaling laws analysis.
result Ensembles of random-feature models achieve near-optimal performance only under specific conditions.
Large models follow power laws in performance with dataset size or parameters.
problem Understanding neural scaling laws in large language models.
method Joint generative data model and random feature model.
result Modeling and solving the dual limit reveals insights into scaling laws.
Recently, Convolution Neural Networks (CNNs) obtained huge success in numerous vision tasks. In particular, DenseNets have demonstrated that feature reuse via dense skip connections can effectively alleviate the difficulty of training very deep networks and that reusing features generated by the initial layers in all s…
New method uses tensor decompositions to overcome the curse of dimensionality for large-scale learning.
problem Large-scale machine learning problems with kernel methods.
method Deterministic Fourier features combined with low-rank tensor decomposition for tensor product structure.
result Demonstrated consistent performance and superior results compared to random Fourier features.
Optimizer choice affects neural scaling laws, changing the exponent .
problem The exponent in neural scaling laws varies with the optimizer used.
method Controlled random-feature regression experiments with five optimizer variants and six spectral conditions.
result Preconditioned optimizers yield steeper scaling (larger ), with the -shift increasing across most of the tested spectral range.
We introduce a method called multi-scale local shape analysis, or MLSA, for extracting features that describe the local structure of points within a dataset. The method uses both geometric and topological features at multiple levels of granularity to capture diverse types of local information for subsequent machine lea…
Defines complexity measure for neural networks and feature representations, revealing scaling patterns.
problem Understanding the nonlinearity and dimensionality of neural network computations and feature representations.
method Introduces complexity and effective dimension measures, investigates their dynamics during training, and analyzes their scaling properties.
result Power law scaling of complexity and effective dimension during training, revealing hidden structure of datasets.
The choice of normalization affects the coefficients in regularized regression models.
problem The impact of normalization on the coefficients of regularized regression models.
method Investigated lasso, ridge, and elastic net regression with different normalization methods for binary and mixed features.
result Normalization affects the coefficients of regularized regression models, and specific scaling methods can mitigate this effect.
A feature selection algorithm should ideally satisfy four conditions: reliably extract relevant features; be able to identify non-linear feature interactions; scale linearly with the number of features and dimensions; allow the incorporation of known sparsity structure. In this work we propose a novel feature selection…
RFFNet scales kernel methods to large datasets by learning kernel relevance.
problem Scaling kernel methods to large datasets while maintaining interpretability.
method Designs random Fourier features for ARD kernels and uses first-order stochastic optimization for learning kernel relevances.
result RFFNet achieves low prediction error and identifies relevant features, leading to more interpretable solutions.
Large scale online kernel learning aims to build an efficient and scalable kernel-based predictive model incrementally from a sequence of potentially infinite data points. A current key approach focuses on ways to produce an approximate finite-dimensional feature map, assuming that the kernel used has a feature map wit…
This study presents a new lossy image compression method that utilizes the multi-scale features of natural images. Our model consists of two networks: multi-scale lossy autoencoder and parallel multi-scale lossless coder. The multi-scale lossy autoencoder extracts the multi-scale image features to quantized variables a…
Gaussian equivalence fails for simple polynomial embeddings in quadratic scaling RF models.
problem Failure of Gaussian equivalence in polynomial feature embeddings under quadratic scaling.
method Introduced Conditional Gaussian Equivalent (CGE) model to capture non-Gaussian behavior.
result Correct asymptotics derived for training and test errors in CGE model.
Simplifies deep learning scaling analysis without sacrificing accuracy.
problem Interpreting feature learning mechanisms and determining network implicit bias in high-dimensional settings.
method Developed a heuristic approach for predicting data and width scales of feature learning patterns.
result Predictions align with known results and extend to complex architectures.
String kernels are attractive data analysis tools for analyzing string data. Among them, alignment kernels are known for their high prediction accuracies in string classifications when tested in combination with SVM in various applications. However, alignment kernels have a crucial drawback in that they scale poorly du…
Kernel improves Gaussian process scalability for wide datasets.
problem Under-performance of Gaussian processes on wide data.
method Introduces Bézier Gaussian Process kernel with exponential summarising variables growth and linear cost.
result Empirically demonstrates scalability to both tall and wide datasets.
New framework for analyzing hydroclimatic time series across multiple scales.
problem Understanding geophysical processes and evaluating stochastic models across different time scales.
method A novel feature compilation method for multi-scale hydroclimatic analyses.
result Identified similarities and differences in time series types across various temporal resolutions.
We propose a new method for input variable selection in nonlinear regression. The method is embedded into a kernel regression machine that can model general nonlinear functions, not being a priori limited to additive models. This is the first kernel-based variable selection method applicable to large datasets. It sides…
Study reveals conditions for neural networks to forget learned features.
problem Understanding feature unlearning in neural networks.
method Infinite-width limit analysis with stochastic gradient descent, fast-slow dynamics.
result Conditions for feature unlearning are determined by the strength of nonlinear terms and initial weights.
A new method scales sparse machine learning to ultra-high dimensional problems.
problem Sparse and interpretable machine learning in ultra-high dimensional data.
method Two-phase approach: backbone set determination followed by reduced problem solving.
result The backbone set contains truly relevant features with high probability.