Paper analyzes distributed learning with non-i.i.d. samples.
problem Learning rate analysis for distributed kernel ridge regression with dependent samples.
method Integral operator approach and covariance inequality for strong mixing sequences.
result Derives optimal learning rates for distributed kernel ridge regression.
Unified framework for sequence models using test-time regression.
problem Designing efficient sequence models with associative memory.
method Formalizing associative recall as regression over input tokens, deriving various sequence models.
result Clarifies the effectiveness of query-key normalization in softmax attention and offers new generalizations.
New algorithms for regression with adversarial responses on various metric spaces.
problem Regression with adversarial responses under non-i.i.d. sequences.
method Proves universal consistency for a wide range of non-stationary processes.
result Achieves universal consistency for a broader class of sequences than stationary processes.
Transformers can efficiently approximate nonparametric regression with minimal parameters and sequences.
problem Efficiently approximating nonparametric regression functions with transformers.
method Kernel-weighted polynomial basis and gradient descent.
result Achieves minimax optimal rate of convergence with fewer parameters and sequences.
R2T hybrid model improves robust regression for asymmetric noise.
problem Least-squares regression fails with asymmetric structured noise.
method Transformer encoder, compression NN, fixed symbolic equation.
result Median regression MSE of 6e-6 to 3.5e-5 on synthetic data.
GNL addresses dynamic network regression by learning dynamic graph structures and capturing sequence information.
problem Dynamic network regression of multiple inter-connected data entities.
method Graph Neural Lasso (GNL) using gated diffusive units and attention mechanism.
result GNL outperforms existing methods in dynamic network regression tasks.
This paper applies deep learning to ordinal regression, modeling it as a binary search.
problem Ordinal regression with deep learning models.
method Formulated ordinal regression as a binary search problem, using recurrent neural networks.
result Deep learning model shows comparable or better predictive power compared to traditional methods.
A new framework predicts multi-step time series with stability and efficiency.
problem General probabilistic multi-step time series regression challenges.
method Sequence-to-Sequence Neural Networks, Quantile Regression, Direct Multi-Horizon Forecasting, Forking-sequences training scheme.
result Framework accommodates temporal and static covariates, learning across multiple series.
New bounds for sequential linear regression with square loss, uniformly over all features.
problem Uniform regret bounds for online linear regression with square loss over arbitrary deterministic sequences.
method Derives new regret bounds for online linear regression with square loss, using variants of online non-linear ridge regression.
result Optimal constant of 1 in front of the d B 2 ln T d B^2 \ln T d B 2 ln T term for known features, and asymptotic regret bound of d B 2 ln T d B^2 \ln T d B 2 ln T for sequentially revealed features. Language models can predict numeric values as strings.
problem Regression tasks with numeric predictions.
method Causal sequence decoding models trained for next-token prediction.
result Decoder-based heads perform as well as standard heads in numeric regression tasks.
FineMorphs models smooth transformations for multivariate regression.
problem Efficiently modeling complex transformations for multivariate regression.
method Optimal control of affine and diffeomorphic transformations using smooth vector fields.
result FineMorphs can reduce dimensionality and adapt to large datasets.
Paper proposes a flexible framework for joint modeling of sequence features and summary response variables.
problem Joint modeling of latent sequence features and summary response variables in sequential observations.
method Flexible framework with compatibility for various mapping functions.
result Improves annotation of sequential observations and yields good prediction performance.
New model controls memory in seq2seq tasks, revealing learning regimes.
problem Understanding memory in seq2seq tasks using neural networks.
method Introducing a stochastic switching-Ornstein-Uhlenbeck (SSOU) model to control memory and a measure of non-Markovianity.
result Two learning regimes emerge from the interplay of time scales in the SSOU process.
Attention forcing improves sequence-to-sequence model training stability.
problem Training auto-regressive sequence-to-sequence models with attention mechanism is challenging.
method Attention forcing guides the model with generated output history and reference attention.
result Attention forcing trains models to recover from mistakes without requiring a schedule or classifier.
Transformer networks approximate Hölder and Sobolev functions with fixed-depth networks.
problem Nonparametric regression with dependent observations.
method Established novel upper bounds for Transformer networks approximating Hölder and Sobolev functions under various β β β -mixing data assumptions. result Explicit convergence rates for nonparametric regression problems under β β β -mixing data assumptions. Stochastic models fail to outperform standard recurrent networks in sequence modeling.
problem Discrepancy in performance between stochastic and standard recurrent models in sequence modeling.
method Re-examine roles of latent variables, remove restriction on fully factorized output distribution, compare auto-regressive models.
result Standard recurrent models consistently outperform stochastic models in sequence modeling.
We consider the problem of online linear regression on arbitrary deterministic sequences when the ambient dimension d can be much larger than the number of time rounds T. We introduce the notion of sparsity regret bound, which is a deterministic online counterpart of recent risk bounds derived in the stochastic setting…
Study connects Gaussian processes and regularization for sequence-function mappings.
problem Understanding and interpreting sequence-function maps in biology.
method Relates Gaussian process priors, regularization, and gauge fixing in overparameterized weight space.
result Established the relationship between regularized regression and Gaussian processes in function space.
Seq-U-Net improves sequence modeling efficiency with dilated U-Net.
problem Efficiently modeling long-term dependencies in sequences.
method Causal U-Net architecture with dilated filters and slow feature hypothesis.
result Seq-U-Net achieves comparable performance with speed-ups of over 4x in audio generation.
Universal algorithm learns unknown distribution for various decision-making problems.
problem Various statistical measures in contextual sequential decision-making.
method Infinite-dimensional functional regression oracle for cumulative distribution functions.
result Utility regret rate bounded by polynomial decay of eigenvalue sequence.
YASENN interprets neural networks by partitioning activation sequences.
problem Interpreting complex neural network decisions.
method YASENN uses layer-wise gradient boosting decision trees to distill and partition neuron activation sequences.
result YASENN provides interpretable partitions of the input space, revealing neural network decision artifacts.
Proposes a novel network-based neighborhood regression for biological systems.
problem Lack of comprehensive analysis on biological modules using both global and local network data.
method Develops a community-wise least square optimization approach to analyze gene modules and their regulatory strength.
result Achieves exact minimax optimality and linear consistency in identifying gene module associations.
Improves adaptivity in sequence models by over-parameterizing.
problem Adaptivity and generalization in sequence models.
method Over-parameterized gradient descent using eigenfunctions.
result Over-parameterization enhances model adaptivity and generalization.
Paper learns dynamic generator models for video sequences.
problem Modeling spatial-temporal processes like dynamic textures and actions.
method Alternating back-propagation through time algorithm to learn latent state vectors and generator model.
result Trains realistic models for dynamic textures and actions.
A new method improves text generation quality and diversity.
problem Exposure bias in Maximum Likelihood Estimation for text generation.
method ψ-MLE, a new training scheme based on density ratio estimation.
result ψ-MLE outperforms Maximum Likelihood Estimation and other models in text generation quality and diversity.
This work investigates how multi-round reasoning improves LLM performance.
problem Improving problem-solving abilities in complex tasks with LLMs.
method Investigates approximation, learnability, and generalization properties of multi-round auto-regressive models.
result Transformers with finite context windows are universal approximators for Turing-computable functions and can approximate any Turing-computable sequence-to-sequence function through multi-round reasoning.
Proposes using prior variable importance information in high-dimensional regression.
problem Using vague prior information on variable importance in high-dimensional settings.
method Fit a sequence of models indicated by the prior importance orderings, using ridge or Lasso regression.
result Cross-validation can select the best estimator from a sequence of models, with a logarithmic cost compared to the unknown best.
New algorithm reduces online regression error in RKHS.
problem Online regression with time-varying functions in RKHS.
method Hierarchical Vovk-Azoury-Warmuth with discounting.
result Achieves optimal dynamic regret with O ( T 2 / 3 P T 1 / 3 + T ln T ) O(T^{2/3}P_T^{1/3} + \sqrt{T}\ln T) O ( T 2/3 P T 1/3 + T ln T ) regret bound. Overview of high-dimensional time series regression methods.
problem Estimation and inference with high-dimensional time series data.
method Limit theory for high-dimensional dependent data, asymptotic theory for time series regression, statistical learning methods.
result Main limit theory results and asymptotic theory for high-dimensional time series regression.
Two methods preserve tensor structure for reduced dimensionality in tensor regression.
problem Reducing dimensionality of tensor predictors for improved interpretation and accuracy.
method Developed two tensor dimension reduction methods using Tucker and CP decompositions.
result Substantial improvement in accuracy over existing methods in simulations and applications.
Kernel ridge regression inference for nonstandard data.
problem Inferential theory for kernel ridge regression with nonstandard data.
method Constructs valid and sharp confidence sets using anti-symmetric multipliers.
result Develops a test for match effects in school matching mechanisms.
We study nonlinear regression of real valued data in an individual sequence manner, where we provide results that are guaranteed to hold without any statistical assumptions. We address the convergence and undertraining issues of conventional nonlinear regression methods and introduce an algorithm that elegantly mitigat…
Seglearn segments time series data for machine learning tasks.
problem Handling multivariate sequence and contextual data for classification, regression, and forecasting.
method Sliding window segmentation approach within a scikit-learn compatible pipeline.
result Efficient learning of time series data for various machine learning tasks.
STARK improves denoising of low-depth spatial transcriptomics images.
problem Denoising spatial transcriptomics images at ultra-low sequencing depths.
method Adaptive regularization with kernel ridge regression and graph Laplacian.
result STARK optimizes denoising performance over competing methods.
Study of supervised learning from multiple non-independent sequences.
problem Efficient learning from many non-independent sequences.
method Generalizes conditions for efficient learning from independent examples and single auto-correlated sequences.
result Error rate changes from Θ ( n / m T ) Θ(n / m T) Θ ( n / m T ) to Ω ( n 2 / m 2 T ) Ω(n^2 / m^2 T) Ω ( n 2 / m 2 T ) as the number of trajectories increases. Sharp convergence analysis for nonconvex regression models.
problem Nonconvex optimization in regression models with normally distributed covariates.
method Gaussian comparison theorems for analyzing iterative algorithms.
result Sharp global convergence rates for various statistical models.
We study online prediction of bounded stationary ergodic processes. To do so, we consider the setting of prediction of individual sequences and build a deterministic regression tree that performs asymptotically as well as the best L-Lipschitz constant predictors. Then, we show why the obtained regret bound entails the …
Constructs bivariate quantiles using vine copulas for multivariate analysis.
problem Need for research in multivariate quantiles, especially for bivariate responses.
method Constructs bivariate (conditional) quantiles using vine copula based bivariate regression model with a novel tree sequence graph structure.
result Avoids typical shortfalls of regression like transformations, interactions, collinearity, and quantile crossings.
The paper extends sequences while preserving statistical properties using a mixture model.
problem Extending sequences while retaining their statistical properties.
method Auto-regressive Sequence Extension Mixture Model (SEMM) using deep learning.
result The mixture model outperforms traditional neural networks in sequence extension with statistical property retention.
CAFLOW uses auto-regressive flows to translate images efficiently.
problem Image-to-image translation tasks.
method Transforms conditioning image into latent encodings using normalizing flows, models conditional distribution with auto-regressive distributions.
result Outperforms former conditional flow designs.
Study on RL from human bandit feedback for sequence-to-sequence learning, showing reliability and learnability.
problem Reliability and learnability of human bandit feedback for RL from sequence-to-sequence learning.
method Investigated reliability of human bandit feedback, analyzed influence on reward estimator learnability, and tested improvements with regression-based reward estimator.
result Improvements of over 1 BLEU can be achieved by integrating a regression-based reward estimator trained on cardinal feedback into RL for NMT.
Overparameterization helps prevent forgetting in sequential learning tasks.
problem Catastrophic forgetting in continual learning systems.
method Analytical study of gradient descent with linear regression model.
result Overparameterization can mitigate forgetting in a two-task setting.
New algorithm reduces dynamic regret for noisy gradient feedback with piecewise polynomial comparators.
problem Online estimation of piecewise polynomial trends with noisy feedback.
method Introduces variational constraint for piecewise polynomial comparators, designs adaptive algorithm.
result Achieves nearly optimal dynamic regret of $ ilde{O}(n^{ rac{1}{2k+3}}C_n^{ rac{2}{2k+3}})$ .
Tree-based LSTM improves sequential regression with missing data.
problem Regression for variable-length sequential data with missing samples.
method Tree architecture of LSTM networks, selecting LSTM networks based on presence-pattern of previous inputs.
result Significant performance improvements on financial and real-life datasets.
New approach predicts tokens in context, explaining how ICL emerges.
problem Limited understanding of in-context learning emergence.
method Auto-regressive next-token prediction (AR-NTP) with prompt token-dependency and a two-level expectation.
result ICL emerges from the generalization of sequences and topics.
Develops an MS-inspired algorithm for regression mode finding and space partitioning.
problem Finding local modes of regression functions and partitioning input space.
method Mean-shift-inspired algorithm for iterative gradient ascent.
result Proves convergence and rates of convergence for estimated local modes.
Improved text generation with constraints using discrete auto-regressive biasing.
problem Balancing fluency and constraint satisfaction in LLM outputs.
method Discrete Auto-regressive Biasing, leveraging gradients in discrete text space.
result Significantly improved constraint satisfaction with comparable fluency.
The paper explains why combining Sobol sequences and polynomials improves LSMC stability.
problem Improving numerical stability in LSMC algorithms.
method Theoretical justification and derivation of a bound for numerical stability.
result Explicit bound for the number of outer scenarios for numerical stability.