Transforms improve CNNs' invariance to image transformations.
problem Current CNN models lack robustness to spatial transformations.
method Randomly transform feature maps during training to learn invariant representations.
result Significant improvements on benchmark tasks, including image recognition and retrieval.
Scale-invariant algorithms for unconstrained online learning.
problem Designing online algorithms invariant to arbitrary linear transformations of input vectors.
method Exploiting scale invariance symmetry, developing algorithms for coordinate-wise and general invariance.
result Achieved optimal regret bound for coordinate-wise invariance, and almost achieved it for general invariance with logarithmic overhead.
The paper matches features in images using centro-affine invariants and heat flow.
problem Feature matching in images with invariant algorithms.
method Developed an invariant algorithm using centro-affine invariants and heat flow.
result The algorithm compares favorably with existing feature matching methods.
IsoGCNs learn invariant and equivariant graph features for efficient simulations.
problem Learning isometric transformation invariant and equivariant features in graphs for simulations.
method Transformation invariant and equivariant Graph Convolutional Networks (IsoGCNs).
result IsoGCNs outperform state-of-the-art methods on geometrical and physical simulation tasks.
Paper proposes CNN with SIFT for rotation invariant feature extraction.
problem Max-pooling layer discards rotational information, leading to rotation invariance issues.
method Uses SIFT descriptor to capture orientation and spatial relationships.
result Improves feature extraction on MNIST and fashionMNIST datasets.
Transformers can scale both context and task, but MLPs can only scale task.
problem Understanding and scaling In-Context Learning in transformers.
method Simplified transformer architecture, feature map, and MLP combination.
result Simplified transformer can perform ICL and context-scaling but not task-scaling.
Learning invariant representations is an important problem in machine learning and pattern recognition. In this paper, we present a novel framework of transformation-invariant feature learning by incorporating linear transformations into the feature learning algorithms. For example, we present the transformation-invari…
New method selects causal features from diverse data types.
problem Discovering causal relationships from non-continuous data types.
method Transformation-Model (TRAM) based Invariant Causal Prediction (TRAM-ICP) with TRAM-GCM and TRAM-Wald tests.
result Improved power and type I error control for diverse response types.
In order to study the application of artificial intelligence (AI) to dental imaging, we applied AI technology to classify a set of panoramic radiographs using (a) a convolutional neural network (CNN) which is a form of an artificial neural network (ANN), (b) representative image cognition algorithms that implement scal…
Proposes a new feature preprocessing method using kernel density integral transformation.
problem Feature preprocessing for tabular data in machine learning and statistics.
method Kernel density integral transformation as a drop-in replacement or improved alternative to min-max scaling and quantile transformation.
result Frequently outperforms min-max scaling and quantile transformation with hyperparameter tuning.
Improved SVMs learn from few samples with composition and multiple scales.
problem Learning with small sample sizes.
method Transformation-invariant SVMs with composition and locality at multiple scales.
result Kernels based on maximum similarity are positive definite and yield superior accuracy.
We propose a simple and efficient time-series clustering framework particularly suited for low Signal-to-Noise Ratio (SNR), by simultaneous smoothing and dimensionality reduction aimed at preserving clustering information. We extend the sparse K-means algorithm by incorporating structured sparsity, and use it to exploi…
We propose a new high dimensional semiparametric principal component analysis (PCA) method, named Copula Component Analysis (COCA). The semiparametric model assumes that, after unspecified marginally monotone transformations, the distributions are multivariate Gaussian. COCA improves upon PCA and sparse PCA in three as…
PARD generates graphs efficiently and invariantly to node ordering.
problem Graph generation sensitivity to node ordering.
method Integrates autoregressive and diffusion models with a partial order for nodes and edges.
result PARD achieves state-of-the-art performance on molecular and non-molecular datasets.
We add prior knowledge to deep networks to make them invariant to transformations.
problem Creating deep networks invariant to transformations like rotation.
method A novel layer based on invariant integration to enforce feature space invariances.
result State-of-the-art performance on the Rotated-MNIST dataset.
Improves contrastive learning invariance with novel training objectives and feature averaging.
problem Contrastive learning's implicit invariance is insufficient for robust performance.
method Introduces a novel training objective and feature averaging approach to enforce invariance.
result Improved performance and robustness to transformations on downstream tasks.
A method uses ITD and XGBoost for precise power transformer fault diagnosis.
problem Fault diagnosis of power transformers using DGA data.
method Ranking DGA parameters by skewness, extracting ITD features, and using an XGBoost classifier.
result The method achieves over 95% accuracy in classification.
TradeFM learns market microstructure from trade events, improving financial model accuracy.
problem Lack of generalizable models for market microstructure.
method Generative Transformer model trained on billions of trade events, using scale-invariant features and universal tokenization.
result TradeFM generates rollouts that match key stylized facts of financial returns and outperforms existing models.
Deep convolutional neural networks have led to breakthrough results in numerous practical machine learning tasks such as classification of images in the ImageNet data set, control-policy-learning to play Atari games or the board game Go, and image captioning. Many of these applications first perform feature extraction …
Study examines infinite limits of transformer dynamics, identifying key parameterizations.
problem Understanding the training dynamics of transformer models in the feature learning regime.
method Analysis of infinite scaling limits using dynamical mean field theory.
result Identified parameterizations that admit well-defined infinite width and depth limits.
This work tackles OOD generalization by leveraging causal invariance without needing to recover causal features.
problem Learning models that perform well on out-of-distribution (OOD) data.
method Causal invariant transformations to modify non-causal features while preserving causal parts.
result Theoretical and practical methods to learn a minimax optimal model across domains using single domain data.
This study evaluates feature scaling across 14 datasets and 12 techniques in ML.
problem Impact of feature scaling on machine learning performance and computational costs.
method Systematic evaluation of 12 scaling techniques across 14 datasets and 16 ML algorithms.
result Wide variation in model performance due to feature scaling, especially for non-ensemble models.
CNNs encode data augmentation transformations, especially in early layers.
problem Whether neural network features encode data augmentation transformations.
method Systematic approach using pre-trained vision models to predict augmentation transformations.
result Neural network features encode data augmentation transformations, especially in early layers.
New method for curve comparison using iterated integrals and moving frames.
problem Comparing curves robustly to noise and transformations.
method Moving frame method paired with log-signature transform.
result Algorithmic construction of invariants for curve equivalence under rigid motions.
Employing the Klein-Gordon equation, we propose a generalized Black-Scholes equation. In addition, we found a limit where this generalized equation is invariant under conformal transformations, in particular invariant under scale transformations. In this limit, we show that the stock prices distribution is given by a C…
Wavelet scattering spectra model non-Gaussian time-series, proving scale invariance for self-similar processes.
problem Modeling non-Gaussian time-series with stationary increments.
method Complex wavelet transform for scale variations, joint correlation matrix for scale dependencies, second wavelet transform for diagonalization, maximum entropy models conditioned by scattering spectra coefficients.
result Scattering spectra of self-similar processes are scale invariant, allowing statistical testing and generation of new time-series.
We introduce multiscale invariant dictionaries to estimate quantum chemical energies of organic molecules, from training databases. Molecular energies are invariant to isometric atomic displacements, and are Lipschitz continuous to molecular deformations. Similarly to density functional theory (DFT), the molecule is re…
Conventional mutual information (MI) based feature selection (FS) methods are unable to handle heterogeneous feature subset selection properly because of data format differences or estimation methods of MI between feature subset and class label. A way to solve this problem is feature transformation (FT). In this study,…
Sparse coding is a common approach to learning local features for object recognition. Recently, there has been an increasing interest in learning features from spatio-temporal, binocular, or other multi-observation data, where the goal is to encode the relationship between images rather than the content of a single ima…
Scales attention for long contexts in LLMs.
problem Development of attention mechanisms for long context inference.
method Scale-invariant total attention and sparsity conditions, with a position-dependent transformation of logits.
result Scale-invariant attention scheme improves validation loss and long-context retrieval.
The execution flow drives market dynamics, validated on real data.
problem Understanding the fundamental driving force of market dynamics.
method Developed a numerical framework using the Radon-Nikodym derivative to calculate execution flow and determined thresholds and characteristic time scales.
result Execution flow is the fundamental driving force of market dynamics.
New algorithm optimizes ReLU networks by transforming to a scale-invariant space.
problem Optimization issues in ReLU networks due to mismatch between optimization space and network's scale-invariance.
method Developed G-SGD algorithm operating in a new G-space, invariant under positive scaling. result Significant improvement in optimizing ReLU networks compared to conventional SGD.
New algorithms predict without tuning for varying feature scales.
problem Learning linear models online with unknown feature scales.
method Adaptive scale-invariant online algorithms without tuning parameters.
result Achieves regret bounds matching OGD with optimal tuning, comparable runtime.
New models exploit invariance to reduce model complexity.
problem Reducing model complexity for tasks with inherent invariances.
method Invariant random features and kernel methods.
result Exploiting invariance saves a dα factor in model complexity. End-to-end models perform better with learned log-scaled mel-spectrogram features.
problem End-to-end neural network models struggle with performance compared to models using high-level data representations.
method Trained first layers of a CNN model on log-scaled mel-spectrogram transformation and then used these learned features to initialize an end-to-end CNN classifier.
result Convergence and performance on ESC-50 dataset are similar to a model trained on pre-processed log-scaled mel-spectrogram features.
We generate transformation-invariant CNNs using context-aware filters.
problem Creating transformation-invariant neural networks for image recognition.
method Input-conditioned convolution filters combined with max-pooling and multi-instance learning.
result Significantly improved error rates on MNIST variations (1.13% on MNIST-rot-12k, 1.12% on Half-rotated MNIST, 0.68% on Scaling MNIST).
A new feature coding method for invariant features using tensor products.
problem Learning invariant features for transformations represented by orthogonal matrices.
method Group-invariant feature vector using tensor-product representations of basic representations.
result Group-invariant feature vector contains sufficient discriminative information for linear classifiers.
Unified spectral framework for μP under joint width-depth scaling.
problem Challenges in stable feature learning and HP transfer for width-depth scaled models.
method Developed a simple and unified spectral framework for μP under joint width-depth scaling.
result Unified and generalized μP formulation for practical architectures with multi-transformation branches.
Set-Sequence model learns cross-sectional dynamics directly from time series data.
problem Predicting large cross-sections of time series data with latent cross-sectional dynamics.
method A model that learns cross-sectional structure directly, enhancing expressivity and eliminating manual feature engineering.
result Significantly outperforms strong baselines in equity portfolio optimization and loan risk prediction.
The study uncovers invariant features in healthcare models that traditional methods overlook.
problem Discovering overlooked invariant features in healthcare models.
method Empirical learning of transformations minimizing Wasserstein distance and adding similarity regularization.
result LSTM models and BioBERT reveal invariant features not previously recognized.
Invariance to nuisance transformations is one of the desirable properties of effective representations. We consider transformations that form a \emph{group} and propose an approach based on kernel methods to derive local group invariant representations. Locality is achieved by defining a suitable probability distributi…
This work explains scaling laws as redundancy laws in deep learning.
problem The mathematical origins of scaling laws in deep learning models remain unclear.
method Kernel regression and analysis of data covariance spectra.
result Scaling laws can be explained as redundancy laws, revealing the learning curve's slope depends on data redundancy.
For many tasks and data types, there are natural transformations to which the data should be invariant or insensitive. For instance, in visual recognition, natural images should be insensitive to rotation and translation. This requirement and its implications have been important in many machine learning applications, a…
This research studies affine invariance in continuous-domain convolutional neural networks.
problem Recognizing patterns and features under affine transformations in continuous domains.
method Introduces a new criterion for assessing affine invariance, embeds images into the affine Lie group, and analyzes convolution over this group.
result Extends the scope of geometrical transformations that deep-learning pipelines can handle.
Proposes a method for forecasting large-scale interval-valued time series.
problem Modeling and forecasting large-scale interval-valued time series.
method Feature extraction procedure involving auto-segmentation, clustering, and precision matrix estimation.
result The method enhances forecasting performance for large-scale interval-valued time series.
We analyze in this paper a random feature map based on a theory of invariance I-theory introduced recently. More specifically, a group invariant signal signature is obtained through cumulative distributions of group transformed random projections. Our analysis bridges invariant feature learning with kernel methods, as …
DRFormer uses dynamic tokenization and multi-scale transformer to forecast long time series.
problem Forecasting long-term time series data across diverse scales.
method Dynamic tokenizer, multi-scale transformer, dynamic sparse learning, rotary position encoding.
result DRFormer outperforms existing methods in forecasting accuracy.
Representations in the auditory cortex might be based on mechanisms similar to the visual ventral stream; modules for building invariance to transformations and multiple layers for compositionality and selectivity. In this paper we propose the use of such computational modules for extracting invariant and discriminativ…