Study compares LSTM and Transformer models in financial time series prediction.
problem Comparing LSTM and Transformer models for financial time series prediction.
method Various LSTM-based and Transformer-based models compared on financial tasks; DLSTM and new Transformer architecture designed.
result Transformer-based models show limited advantage in absolute price sequence prediction, while LSTM-based models perform better on difference sequences.
This paper proposes a new RV prediction model using neural distributional transformation and co-training.
problem Predicting skewed and fat-tailed realized volatility (RV) is challenging.
method The paper uses a neural distributional transformation and co-training to predict RV. It jointly trains the transformation and prediction model using a maximum-likelihood objective function.
result The proposed method significantly outperforms other methods on a dataset of 100 stocks.
Transforms ensemble predictions to maintain interpretability.
problem Loss of interpretability in deep ensembles.
method Proposes transformation ensembles that aggregate predictions while preserving interpretability.
result Transformation ensembles yield better predictions than individual models and maintain interpretability.
Paper uses Time Series Transformer for bank stability prediction.
problem Predicting bank stability using complex financial data.
method Time Series Transformer model with self-attention mechanism.
result Time Series Transformer model outperforms other models in MSE and MAE.
Transformer pre-training improves stock return prediction accuracy.
problem Improving stock price prediction accuracy for better investment decisions.
method Pre-trained transformer models on TSX index, fine-tuned for individual stocks, compared to LSTM and XGBoost.
result Transformer model achieved lower mean squared error than benchmarks.
ST-BCP narrows the coverage gap in BCP by transforming nonconformity scores.
problem The looseness in BCP's coverage guarantee due to Markov's inequality.
method Introduces a data-dependent transformation of nonconformity scores.
result Reduces the average coverage gap from 4.20% to 1.12% on benchmarks.
Looped transformers with LN converge to power method for principal component prediction.
problem Understanding how transformers learn algorithmic procedures.
method Study of principal component prediction with looped linear transformers and layer normalization.
result Gradient descent trains looped transformers with LN to implement the power method for principal component prediction.
Transformers can approximate posterior predictive distributions through in-context learning.
problem Bayesian prediction tasks, especially beyond point predictions.
method Gradient descent algorithm targeting posterior predictive mean and variance, followed by nonlinear mappings.
result Transformers can implement algorithms to approximate posterior predictive distributions.
Study uses XAI and transformers for stock price prediction of top 100 BIST banks.
problem Enhancing interpretability and accuracy of stock price predictions.
method Combines transformer-based time series models with XAI techniques.
result Transformer models show strong predictive capabilities and provide feature transparency.
CNNs encode data augmentation transformations, especially in early layers.
problem Whether neural network features encode data augmentation transformations.
method Systematic approach using pre-trained vision models to predict augmentation transformations.
result Neural network features encode data augmentation transformations, especially in early layers.
Transformer model predicts stock prices in Bangladesh's stock market.
problem Predicting volatile stock prices in the Bangladesh stock market.
method Transformer model applied to time series data for stock price prediction.
result Transformer model shows promising results in predicting stock price movements.
Paper proposes integrating wavelet transform, channel attention, and LSTM for better stock price prediction.
problem Inherently difficult stock price prediction due to low signal-to-noise ratio.
method Wavelet transform convolution, channel attention, and LSTM integration.
result Robust performance in post-pandemic market conditions.
Deep learning model estimates uncertainty in complex regression tasks.
problem Uncertainty quantification in probabilistic regression predictions.
method Combines statistical and deep learning transformation models using gradient descent.
result State-of-the-art performance on small datasets and complex image data.
We study the invariance characteristics of pre-trained predictive models by empirically learning transformations on the input that leave the prediction function approximately unchanged. To learn invariant transformations, we minimize the Wasserstein distance between the predictive distribution conditioned on the data i…
Preformer improves Transformer for long-term time series forecasting.
problem Transformer's quadratic complexity and lack of context-awareness for long-term forecasting.
method Introduces Multi-Scale Segment-Correlation mechanism for efficient time series segmentation and context-aware attention.
result Preformer outperforms other Transformer-based methods in long-term time series forecasting.
This paper explores how Transformers predict next tokens in autoregressive tasks.
problem Understanding the success of Transformers in autoregressive learning.
method Trained a Transformer on a next-token prediction task, focusing on commuting orthogonal matrices.
result Trained Transformers can be seen as implementing gradient descent for a specific objective function.
We propose the first qualitative hypothesis characterizing the behavior of visual transformation based self-supervision, called the VTSS hypothesis. Given a dataset upon which a self-supervised task is performed while predicting instantiations of a transformation, the hypothesis states that if the predicted instantiati…
Paper uses Transformers to predict intraday volume ratio with high accuracy.
problem Accurate prediction of intraday volume ratio for VWAP strategies.
method Transformer architecture with log-normal transformation and external features.
result Probabilistic forecasting captures mean and standard deviation of volume ratios.
Transformer predicts Ethereum prices using cross-currency correlation and sentiment analysis.
problem Predicting Ethereum cryptocurrency prices with limited data.
method Transformer-based neural network with cross-currency correlation and sentiment analysis.
result Transformer model outperforms other models on some parameters.
CTLNet predicts Shanghai Composite Index with improved accuracy.
problem Predicting Shanghai Composite Index for investors and researchers.
method Proposes CNN-Transformer-LSTM Networks (CTLNet) for multivariate time series forecasting.
result CTLNet outperforms state-of-the-art baselines in Shanghai Composite Index prediction.
Transformer models predict financial time series movements accurately.
problem Applying transformer models to financial time series prediction.
method Transformer architecture applied to synthetic and real S&P500 data.
result Transformer models predict financial time series movements accurately.
New analysis shows transformer models can't learn effectively.
problem Transformer models learning in context can't achieve general predictive accuracy.
method Empirical evidence and mathematical analysis of transformer architecture limitations.
result Transformers cannot achieve general predictive accuracy due to architectural limitations.
Transformers can predict pseudo-random sequences from LCGs with unseen parameters and moduli.
problem Learning pseudo-random number sequences from linear congruential generators with unknown parameters and moduli.
method Investigated the ability of Transformers to learn LCG sequences with varying complexity and moduli. Analyzed embedding layers and attention patterns.
result Transformers can predict pseudo-random sequences from LCGs with unseen parameters and moduli, up to mexttest=216, using a two-step strategy. This paper proposes a deep learning model combining CNN and Transformer for improved credit default prediction.
problem Traditional machine learning models struggle with complex financial data and risk patterns.
method Combines CNN for local feature extraction and Transformer for global dependency modeling.
result The CNN+Transformer model outperforms traditional models in accuracy, AUC, and KS value.
A new transformer model uses Monte Carlo methods for sequence prediction.
problem Predicting sequences from observations with uncertainty.
method Integrates Monte Carlo methods into a transformer architecture to model stochastic sequences.
result Proposes a generative model with predictive distributions.
Predictive Sparse Manifold Transform learns dynamic video sequences.
problem Learning and predicting natural dynamics in video sequences.
method Two-layer framework: sparse coding and manifold learning.
result PSMT with dynamic embedding space outperforms static baselines in future frame prediction.
SGPA calibrates transformer uncertainty for safety-critical tasks.
problem Uncertainty estimation in transformer models for safety-critical domains.
method Bayesian inference in transformer's output space using sparse Gaussian processes.
result SGPA-based Transformers improve in-distribution calibration and out-of-distribution robustness.
Enhances stock movement prediction using Higher Order Transformers for multimodal time-series data.
problem Predicting stock movements in financial markets with complex dynamics.
method Introduced Higher Order Transformers, extending self-attention and transformer architecture to capture complex market dynamics. Employed low-rank tensor decomposition and kernel attention to manage computational complexity. Integrated technical and fundamental analysis from historical prices and tweets.
result Demonstrated effectiveness of the method on the Stocknet dataset, improving stock movement prediction.
The study explores how Transformers predict the next token in a sequence.
problem Understanding the mechanism behind Transformers' autoregressive learning ability.
method Exploring the approximation ability of Transformers for next-token prediction through specific instances and a causal kernel descent method.
result Transformer models can learn context-dependent functions f for next-token prediction based on past and current observations. This paper compares LSTM, GRU, and Transformer models for stock price prediction.
problem Improving stock price prediction accuracy in fast-paced financial markets.
method Training models on Tesla stock data from 2015 to 2024, comparing LSTM, GRU, and Transformer.
result LSTM model achieved 94% accuracy in predicting stock prices.
Timer-XL predicts multidimensional time series using a unified Transformer approach.
problem Unified time series forecasting across various tasks and contexts.
method Decoder-only Transformers with a universal TimeAttention mechanism and deft position embedding.
result State-of-the-art performance across multiple forecasting benchmarks.
A new method for discrete data normalizing flows using latent transformations.
problem Challenges in parameterizing bijective transformations for discrete data.
method Predict a distribution over latent transformations to make the marginal likelihood differentiable.
result Discrete-data normalizing flows can be trained using gradient-based learning with unbiased score function estimation.
Transformers can learn optimal regression mixtures efficiently.
problem Limited adoption of tailored regression methods due to their model-specific nature.
method Constructed a generative process for a mixture of linear regressions and used transformers to learn optimal predictors.
result Transformers achieve low mean-squared error and make predictions close to the optimal procedure.
Random forest models systematically bias predictions; a numerical transform corrects this.
problem Systematic bias in random forest regression predictions.
method Defined a numerical transformation using training data to correct bias.
result Improved predictions across real-world and synthetic datasets.
Develops a framework for consistent loss functions with variable transformations.
problem Lack of theoretical understanding of variable transformations in consistent loss functions.
method Formal characterizations of consistency for transformed loss functions in two cases: realization and prediction variables.
result Establishes new identifiable and elicitable functionals for complex predictive tasks.
Predicts next item in sequential bundles using Transformers.
problem Predicting next item in sequentially consumed bundles.
method Used custom Transformers, GPT-3, LSTM, reinforcement learning, Markov models.
result Custom Transformer with decoder-only architecture most accurate.
Proposes a tabular transformer model to maintain feature effect intelligibility.
problem Losing marginal feature effects in deep tabular transformer networks.
method Adapts tabular transformer networks to identify marginal feature effects.
result The model accurately identifies marginal feature effects, matching black-box performance while maintaining intelligibility.
SMURF-THP improves Transformer Hawkes process models by providing uncertainty quantification.
problem Uncertainty quantification for Transformer Hawkes process predictions.
method Score matching for learning the score function of event arrival times.
result SMURF-THP outperforms likelihood-based methods in confidence calibration.
Deep learning uses layers of transformations to predict structured data with uncertainty.
problem Predicting structured high-dimensional data efficiently and with uncertainty.
method Applying layers of semi-affine input transformations to find features for probabilistic statistical methods.
result Achieves scalable prediction rules with uncertainty quantification and feature selection.
Paper improves Tm prediction of protein fragments using sparsity and probabilistic models.
problem Improving accuracy of melting temperature prediction for protein fragments.
method Promoting sparsity in pre-trained transformer models and adopting probabilistic frameworks.
result Mean absolute error of 0.23C for predicting melting temperature.
Multivariate time series prediction has applications in a wide variety of domains and is considered to be a very challenging task, especially when the variables have correlations and exhibit complex temporal patterns, such as seasonality and trend. Many existing methods suffer from strong statistical assumptions, numer…
MAT uses attention mechanism for molecule property prediction.
problem Designing a competitive neural network for molecule property prediction.
method Augmented attention mechanism using inter-atomic distances and molecular graph structure.
result MAT achieves state-of-the-art performance on diverse molecular prediction tasks.
FutureQuant Transformer predicts price ranges and volatility for futures trading.
problem Complex futures trading with real-time LOBs and vast data.
method FutureQuant Transformer model using attention mechanisms.
result Significantly improved trading performance with an average gain of 0.1193%.
Paper develops a new model for predicting volatility surface.
problem Predicting volatility in financial markets is challenging due to its non-observable nature and complex dynamics.
method Physics-informed convolutional transformer architecture.
result The new model outperforms other deep-learning architectures in predicting volatility surface.
The paper analyzes the training dynamics of a transformer for next-token prediction.
problem Understanding the non-asymptotic performance of transformers in next-token prediction.
method Characterizes training dataset properties, designs a two-stage training algorithm, and analyzes attention gradient properties.
result Trained transformers converge sub-linearly to max-margin solutions and exhibit linear convergence in cross-entropy loss.
We present online prediction methods for time series that let us explicitly handle nonstationary artifacts (e.g. trend and seasonality) present in most real time series. Specifically, we show that applying appropriate transformations to such time series before prediction can lead to improved theoretical and empirical p…
Transformers become faster by linearizing self-attention.
problem Quadratic complexity of transformers makes them slow for long sequences.
method Expressed self-attention as a linear dot-product and used matrix product associativity to reduce complexity.
result Linear transformers are up to 4000x faster on long sequences.
Transformers predict price movements from limit order books.
problem Predicting price movements from limit order books.
method Causal convolutional network with masked self-attention.
result Significantly outperforms existing architectures on FI-2010 dataset.