Study excess risk in statistical inference with transformations.
problem Excess risk in estimating random variables from feature vectors and transformations.
method Characterize lossless transformations, develop test statistics, and information-theoretic bounds.
result Strongly consistent partitioning test statistic for lossless transformations.
Study shows statistical biases can mislead transformer models, impairing their generalization.
problem Statistical biases in transformers affect their ability to generalize.
method Evaluated transformer models on synthetic algorithmic tasks with varying statistical biases.
result Statistical biases lead to overestimation of transformer models' generalization capabilities.
Novel power transform unifies various mathematical functions.
problem Normalizing and standardizing datasets.
method Presented a novel power transform.
result Unified various mathematical functions.
A new method improves likelihood-free Bayesian inference by transforming summary statistics and using efficient Variational Bayes.
problem Incorrectly assuming normally distributed summary statistics in likelihood-free Bayesian inference.
method Wasserstein Gaussianization transformation combined with robust BSL and efficient Variational Bayes.
result Highly efficient and reliable approximate Bayesian inference for likelihood-free problems.
The abstract discusses transformations on statistical and semi-Weyl manifolds with torsion.
problem Invariance and structure preservation under conformal-projective transformations.
method Proof of invariance and preservation of structures under conformal-projective transformations.
result Semi-Weyl and statistical structures with torsion are invariant under conformal-projective transformations.
Efficiently transforms samples from various statistical models.
problem Approximately transforming samples from one statistical model to another without knowing the source model's parameters.
method Constructs computationally efficient procedures to reduce uniform, Erlang, and Laplace models to general target families.
result Establishes nonasymptotic reductions between canonical high-dimensional problems, such as mixtures of experts, phase retrieval, and signal denoising.
Researchers improve imputation of missing data using diffusion transformers, quantifying uncertainty.
problem Enhancing the quality of time-series data with missing values.
method Conditional diffusion transformers for imputation, with statistical sample complexity bounds and uncertainty quantification.
result Theoretical insights into the efficiency and accuracy of imputation, influenced by missing patterns.
Cookbook transforms constrained statistical inference into unconstrained problems.
problem Transforming constrained statistical inference into unconstrained problems.
method Bijective and diffeomorphisms parametrizations.
result Maintains statistical inference properties like identifiability.
Distance-based tests, also called "energy statistics", are leading methods for two-sample and independence tests from the statistics community. Kernel-based tests, developed from "kernel mean embeddings", are leading methods for two-sample and independence tests from the machine learning community. A fixed-point transf…
Study aggregation of statistical evidence under unknown dependence using group-invariance.
problem Aggregating statistical evidence under unknown and complex dependence structures.
method Develops a framework using group-invariance and permutation-based constructions to aggregate evidence across transformed datasets.
result Shows uniform improvement in critical values for single-batch aggregation over deterministic calibrations, adapting to unknown dependence structures.
Transformers learn to integrate information from past positions incrementally, specializing heads in distinct patterns.
problem How transformers learn to integrate information from multiple past positions with varying statistical significance.
method High-order Markov chain task, incremental learning, sparse attention patterns, simplified differential equations, stage-wise convergence, early stopping as regularizer.
result Transformers learn to specialize heads in distinct patterns, shifting from competitive to cooperative learning dynamics.
Analyzes how BPE tokenisation affects corpus statistics and model entropy in transformer models.
problem Understanding how natural language properties relate to tokenisation schemes in transformer models.
method Analyzes Shannon entropy of corpora under Zipfian distribution, investigates BPE transformations, trains language models, and uses attention diagnostics.
result Transformer models trained on BPE-tokenised corpora increasingly agree with Zipfian predictions as BPE depth increases, indicating reduced local token dependencies.
The study explains transformer scaling laws using statistical and approximation theories.
problem Understanding why transformer scaling laws exist for large models trained on low-dimensional data.
method Established statistical estimation and mathematical approximation theories for transformers on low-dimensional manifolds.
result Predicted a power law between generalization error and model and data sizes, with power depending on intrinsic data dimension.
Proposes a new feature preprocessing method using kernel density integral transformation.
problem Feature preprocessing for tabular data in machine learning and statistics.
method Kernel density integral transformation as a drop-in replacement or improved alternative to min-max scaling and quantile transformation.
result Frequently outperforms min-max scaling and quantile transformation with hyperparameter tuning.
Breiman discusses two statistical cultures, advocating for more research on 'before' and 'after' the black box.
problem Statistical modeling lacks exploration of processes before and after the 'black box'.
method Analyzes Breiman's visual metaphor of two statistical cultures.
result Promotes the importance of studying the 'before' and 'after' of data transformations.
Study proposes a statistical test for Vision Transformer's attention mechanisms.
problem ViT's attention mechanisms may focus on irrelevant regions, leading to unreliable evidence.
method Selective inference framework to quantify statistical significance of attentions as p-values.
result Proposed method enables reliable quantification of false positive detection probability of attentions.
Firm size data usually do not show the normality that is often assumed in statistical analysis such as regression analysis. In this study we focus on two firm size data: the number of employees and sale. Those data deviate considerably from a normal distribution. To improve the normality of those data we transform them…
We propose a new class of transforms that we call {\it Lehmer Transform} which is motivated by the {\it Lehmer mean function}. The proposed {\it Lehmer transform} decomposes a function of a sample into their constituting statistical moments. Theoretical properties of the proposed transform are presented. This transform…
Transformers simplify modeling of small longitudinal cohort data by reducing parameters and incorporating attention mechanisms.
problem Challenges in modeling longitudinal cohort data due to complex temporal dependencies and large dataset requirements.
method Simplified transformer architecture with attention mechanism, autoregressive model, and kernel-based temporal decay.
result The approach recovers contextual dependencies even with small datasets, identifying temporal patterns in stress and mental health.
In statistical analysis, measuring a score of predictive performance is an important task. In many scientific fields, appropriate scores were tailored to tackle the problems at hand. A proper score is a popular tool to obtain statistically consistent forecasts. Furthermore, a mathematical characterization of the proper…
Deep learning uses layers of transformations to predict structured data with uncertainty.
problem Predicting structured high-dimensional data efficiently and with uncertainty.
method Applying layers of semi-affine input transformations to find features for probabilistic statistical methods.
result Achieves scalable prediction rules with uncertainty quantification and feature selection.
The study reveals how attention paths in Transformers influence learning outcomes.
problem Understanding the theoretical basis of Transformers' performance.
method Developed a statistical mechanics theory for a simplified attention network.
result The predictor statistics are influenced by the combination of attention paths.
Transformers recall from long distributions with statistical guarantees.
problem Designing Transformers that can recall from arbitrarily long, distributional contexts.
method Recast associative memory as probability measures, decomposing the task into recall and prediction.
result A shallow measure-theoretic Transformer learns the recall-and-predict map under spectral assumptions.
The paper derives statistics of multi-factor functions from their Fourier transforms.
problem Deriving statistics of multi-factor functions from Fourier transforms.
method Developed an m-Coefficient/Index Annihilation Theorem to analyze the moments of a function from its Fourier transform.
result The mth moment of a function becomes a series of terms, each with precisely m Fourier coefficients, and the indices sum to zero.
Transformers can learn spectral methods and perform unsupervised learning.
problem Learning spectral methods using unsupervised learning.
method Using multi-layered Transformers, pre-trained on a large set of instances, to learn and perform statistical estimation tasks.
result Proven that pre-trained Transformers can learn spectral methods and perform tasks like PCA and clustering.
Transformers learn low-dimensional target functions efficiently in-context.
problem Efficiently learning nonlinear target functions in-context using transformers.
method Nonlinear MLP layer in transformers optimized by gradient descent, focusing on single-index target functions.
result Transformers can learn target functions with low-dimensional structures efficiently in-context.
Generative models improve image restoration from unknown transformations.
problem Restoring images distorted by unknown transformations.
method Combining maximum a-posteriori probability with maximum likelihood estimation.
result Restores images without requiring exact knowledge of transformations.
Develops a framework for consistent loss functions with variable transformations.
problem Lack of theoretical understanding of variable transformations in consistent loss functions.
method Formal characterizations of consistency for transformed loss functions in two cases: realization and prediction variables.
result Establishes new identifiable and elicitable functionals for complex predictive tasks.
OLS is a special case of Transformer, revealing its linear nature.
problem Understanding the statistical essence of Transformer architecture.
method Algebraic proof and spectral decomposition of covariance matrix.
result Attention mechanism in Transformers is mathematically equivalent to OLS.
Transformer improves parameter estimation without needing closed-form solutions.
problem Parameter estimation in statistics, especially for complex distributions.
method Transformer-based approach for parameter estimation without closed-form solutions or derivations.
result Transformer-based approach achieves similar or better accuracy than maximum likelihood estimation.
Transformers solve Gaussian Mixture Models without supervision.
problem Solving Gaussian Mixture Models (GMMs) unsupervised.
method Proposes TGMM, a transformer-based framework for GMM tasks.
result Transformers can effectively solve GMM tasks, improving upon classical methods.
Develops a deep learning approach for statistical arbitrage.
problem Temporal price differences between similar assets.
method Constructs arbitrage portfolios using latent asset pricing factors and a convolutional transformer for time series signals.
result High risk-adjusted returns and Sharpe ratios with optimal trading policy.
A new Bayesian modeling method is proposed by combining the maximization of the marginal likelihood with a momentum-space renormalization group transformation for Gaussian graphical models. Moreover, we present a scheme for computint the statistical averages of hyperparameters and mean square errors in our proposed met…
Utilizing recently introduced concepts from statistics and quantitative risk management, we present a general variant of Batch Normalization (BN) that offers accelerated convergence of Neural Network training compared to conventional BN. In general, we show that mean and standard deviation are not always the most appro…
Relational data representations have become an increasingly important topic due to the recent proliferation of network datasets (e.g., social, biological, information networks) and a corresponding increase in the application of statistical relational learning (SRL) algorithms to these domains. In this article, we exami…
This study shows neural nets can approximate Turing machines with meaningful statistical properties.
problem Theoretical limitations in approximating Turing machines with neural networks.
method Formal definition of statistically meaningful approximation, analysis of boolean circuits and Turing machines using neural nets.
result Transformers can statistically meaningfully approximate Turing machines with polynomial sample complexity.
Transformers learn random walks optimally with gradient descent.
problem Transformer interpretability and learning random walks.
method Theoretical analysis and gradient descent training.
result Transformers can predict random walks optimally with gradient descent.
Proposes Deep LTMLE for estimating dynamic treatment effects in longitudinal studies.
problem Estimating counterfactual mean outcomes under dynamic treatment policies in longitudinal settings.
method Uses a transformer architecture with temporal-difference learning for initial estimation, followed by TMLE correction and statistical inference.
result Demonstrates superior performance in complex, long-term scenarios compared to existing methods.
Study identifies and analyzes three types of errors in learning Fourier operators.
problem Statistical, discretization, and truncation errors in learning Fourier operators.
method Analysis of a Discrete Fourier Transform (DFT) based least squares estimator.
result Established upper and lower bounds on statistical, discretization, and truncation errors.
Squared families are a new model class derived from linear transformations, offering convenient properties and universal approximation.
problem Developing a new class of probability models that are easier to handle and have useful properties.
method Introducing squared families as families of probability densities obtained by squaring a linear transformation of a statistic, and showing their properties and applications.
result Squared families have convenient properties and can approximate target densities well.
Study on DiTs' rates of approximation and estimation under various data assumptions.
problem Investigating statistical rates of conditional diffusion transformers.
method Discretization and Taylor expansion of conditional diffusion score function under Hölder smooth data assumption.
result Establishes statistical limits for conditional and unconditional DiTs, offering practical guidance.
This paper provides a new unimodality test with application in hierarchical clustering methods. The proposed method denoted by signature test (Sigtest), transforms the data based on its statistics. The transformed data has much smaller variation compared to the original data and can be evaluated in a simple proposed un…
New method detects bearing faults using multivariate statistical process control.
problem Early detection of bearing faults in rotating machinery.
method Multivariate statistical process control charts applied to Fourier transform features of fixed-time batches.
result Effectiveness in detecting bearing faults across different conditions.
Transform non-private e-values into differentially private ones.
problem Leaking sensitive data through non-private e-values.
method Developed a novel biased multiplicative noise mechanism.
result Differentially private e-values maintain strong statistical power and asymptotic equivalence to non-private ones.
Single-head transformers with a single self-attention layer can approximate any sequence-to-sequence function and are efficient under certain conditions.
problem Statistical and computational limits of prompt tuning for transformer-based models.
method Investigation of single-head transformers with a single self-attention layer, proving universality and efficiency under SETH.
result Existence of almost-linear time prompt tuning inference algorithms under certain conditions.
A new method tests conditional independence by transforming it into an unconditional problem using transport maps.
problem Testing conditional independence between two random vectors given a third.
method Constructing transport maps to transform conditional independence into unconditional independence, estimating these maps from data using conditional continuous normalizing flow models.
result The proposed method is validated through simulations and real-data analysis, demonstrating practical effectiveness.
Paper proposes PMformer for better cryptocurrency price forecasting.
problem Huge volatility and trade-off between univariate and multivariate models.
method Partial-multivariate approach using PMformer.
result PMformer achieves significant statistical accuracy in forecasting.
Extends SW and GSW to compare heterogeneous joint distributions.
problem Limited applicability of SW and GSW to heterogeneous joint distributions.
method Introduces HHRT and PGRT to extend SW and GSW.
result H2SW distance for heterogeneous joint distributions.