arXiv research

Efficient algorithms find optimal monotone transforms for calibration under strictly convex losses.

Transformers can learn optimal regression mixtures efficiently.

Researchers improve transformer networks' optimization and understanding.

Study on SignGD optimization of two-layer transformer on noisy data.

A new transformer model accelerates training with optimization techniques.

Transformers can approximate Newton's method for logistic regression.

Transformers struggle to learn Markovian dynamics, showing NP-hard optimization challenges.

Transformers achieve near-optimal dynamic regret in non-stationary reinforcement learning.

FineMorphs models smooth transformations for multivariate regression.

Transformer models align words through attention weights, closely approximating Optimal Transport.

Pion optimizes LLMs by preserving weight matrix singular values.

Optimizing data movement significantly improves transformer training efficiency.

Transformers learn a mesa-optimizer to implement in-context learning.

Kernel estimator optimally recovers function from noisy exponential Radon transform.

In many state-of-the-art compression systems, signal transformation is an integral part of the encoding and decoding process, where transforms provide compact representations for the signals of interest. This paper introduces a class of transforms called graph-based transforms (GBTs) for video compression, and proposes…

Fusion of transformer networks using optimal transport for improved performance.

Transformers use a unique Hessian structure that differs from classical networks, affecting optimization.

Standard Transformers approximate Hölder functions and achieve optimal nonparametric regression rate.

Solves optimal stopping for Gauss-Markov bridges using time-space transformation.

Transformers can efficiently approximate nonparametric regression with minimal parameters and sequences.

Transforms robust algorithms into private ones with optimal error rates.

Transforms game optimization dynamics into frequency domain for precise hyperparameter analysis.

Transformers can learn optimal variable selection in group-sparse classification.

Continuum transformers learn operators in context via gradient descent.

Transformers converge linearly to optimal models for Gaussian mixtures classification.

Enhanced Transformer models predict ETF portfolio performance by optimizing covariance and semi-covariance matrices.

PPT optimizes transformer behavior by steering its latent posterior using prior samples.

This work introduces a transformation-based learner model for classification forests. The weak learner at each split node plays a crucial role in a classification tree. We propose to optimize the splitting objective by learning a linear transformation on subspaces using nuclear norm as the optimization criteria. The le…

Transformers can handle endogeneity in linear regression using IV methods.

Sparse transformer architecture improves accuracy and speed in generative modeling and inverse problems.

This paper considers fair probabilistic binary classification where the outputs of primary interest are predicted probabilities, commonly referred to as scores. We formulate the problem of transforming scores to satisfy fairness constraints that are linear in conditional means of scores while minimizing a cross-entropy…

Recurrent neural network (RNN)'s architecture is a key factor influencing its performance. We propose algorithms to optimize hidden sizes under running time constraint. We convert the discrete optimization into a subset selection problem. By novel transformations, the objective function becomes submodular and constrain…

A new model improves CT image quality from low-dose scans.

We prove that the global minimum of the backpropagation (BP) training problem of neural networks with an arbitrary nonlinear activation is given by the ridgelet transform. A series of computational experiments show that there exists an interesting similarity between the scatter plot of hidden parameters in a shallow ne…

Improved hyperparameter optimization using simplified Transformer blocks.

Algorithm finds optimal affine transformation to minimize overall distortion.

Derives an empirical capacity model for self-attention neural networks.

Transforms offline greedy algorithms to online algorithms for combinatorial problems.

Transformer RL optimizes A/B testing for time series experiments.

Transformers learn new tasks from few examples via optimal approximation.

Transformers reduce redundancy by focusing on invariant relational quantities.

Many problems in machine learning involve calculating correspondences between sets of objects, such as point clouds or images. Discrete optimal transport provides a natural and successful approach to such tasks whenever the two sets of objects can be represented in the same space, or at least distances between them can…

Transforms ensemble predictions to maintain interpretability.

OptFormer learns universal HPO from diverse datasets.

Proposes a differentiable STFT for more efficient optimization of hop length.

The paper solves a complex financial optimization problem using a novel mathematical technique.

SAMformer improves transformer performance in time series forecasting.

Recent research on accelerated gradient methods of use in optimization has demonstrated that these methods can be derived as discretizations of dynamical systems. This, in turn, has provided a basis for more systematic investigations, especially into the geometric structure of those dynamical systems and their structur…