New clustering method for POIs from spatio-temporal data with temporal constraints.
problem Lack of temporal constraints in existing clustering methods for POIs.
method POI clustering with temporal constraints (PC-TC).
result PC-TC outperforms existing methods in next place prediction.
Model predicts next destination for users based on past trips and features.
problem Predicting the next destination in multi-destination trips.
method Used Cleora for city graph embedding and EMDE for prediction.
result Achieved 2nd place in Booking Data Challenge.
In-Place TTT enhances LLMs with dynamic parameter updates at inference time.
problem Static training limits LLMs from adapting to new information.
method In-Place TTT updates a subset of model parameters (fast weights) at inference time.
result In-Place TTT enables 4B-parameter models to outperform on tasks with up to 128k contexts.
We are now witnessing the increasing availability of event stream data, i.e., a sequence of events with each event typically being denoted by the time it occurs and its mark information (e.g., event type). A fundamental problem is to model and predict such kind of marked temporal dynamics, i.e., when the next event wil…
Seq2Slate models for ranking by predicting appealing item slates.
problem Ranking items in a slate to present a user appealing set of items.
method Sequence-to-sequence model that predicts the next best item to place on the slate.
result The model captures complex dependencies between items and learns from weak supervision.
Study on learning to predict dynamical systems without assuming their structure.
problem Learning to predict the next state of a dynamical system with unknown evolution function.
method Defined new combinatorial measures to quantify mistake and regret bounds in realizable and agnostic settings.
result In the realizable setting, the number of mistakes can grow arbitrarily with time.
We identify and analyze statistical regularities and irregularities in the recent order flow of different NASDAQ stocks, focusing on the positions where orders are placed in the orderbook. This includes limit orders being placed outside of the spread, inside the spread and (effective) market orders. We find that limit …
L2P predicts heavy-tailed outcomes by placing new instances among known ones.
problem Predicting heavy-tailed outcomes (e.g., best-sellers) with under-prediction by existing methods.
method Learning to Place (L2P) learns pairwise preferences and places new instances to estimate outcomes.
result L2P outperforms existing methods in accuracy and reproducing heavy-tailed distributions.
New model explains music emotion predictions with visualizable mid-level features.
problem Challenging to quantify and predict music emotion.
method VGG-style deep neural network with mid-level perceptual features.
result Small loss in performance justifies high explainability of predictions.
A new model predicts race places using changeover-times and log-normal distributions.
problem Predicting race places in orienteering races.
method Fenton-Wilkinson Order Statistics model based on log-normal leg-times and changeover-times.
result The model accurately predicts race places with smaller root-mean-square-errors.
Deep reinforcement learning has been successfully applied to several visual-input tasks using model-free methods. In this paper, we propose a model-based approach that combines learning a DNN-based transition model with Monte Carlo tree search to solve a block-placing task in Minecraft. Our learned transition model pre…
Proving that next-token prediction makes language models generate coherent long documents.
problem Understanding why language models generate coherent documents despite focusing on next-token prediction.
method Proving the power of next-token prediction in learning longer-range structure using Recurrent Neural Networks (RNN).
result Optimizing next-token prediction in RNNs yields a model that closely approximates the training distribution, even for long-range coherence.
Which song will Smith listen to next? Which restaurant will Alice go to tomorrow? Which product will John click next? These applications have in common the prediction of user trajectories that are in a constant state of flux over a hidden network (e.g. website links, geographic location). What users are doing now may b…
Bayesian framework optimizes 3D view selection for specific tasks.
problem Optimizing 3D view selection for specific tasks in reconstruction.
method Bayesian decision theory, prior and posterior distributions, stochastic surface reconstruction.
result Framework achieves superior performance with fewer views.
Reinforcement learning after next-token prediction aids in learning from diverse sequence lengths.
problem Learning from sequences of varying lengths and complexity.
method Introducing a framework to study reinforcement learning with autoregressive transformers, focusing on next-token prediction and mixture distributions of short and long sequences.
result Reinforcement learning after next-token prediction enables autoregressive transformers to generalize from long sequences, even when they are rare.
Split conformal prediction works well for time series despite temporal dependence.
problem Uncertainty quantification for time series predictions with past data.
method Split conformal prediction method for time series data with predictors having memory.
result Theoretical bounds on coverage probability for split conformal prediction in time series with memory.
New deep ESN architectures improve memory capacity and prediction accuracy.
problem Improving memory capacity and prediction accuracy of ESNs.
method Two new deep ESN architectures: parallel and series. Analysis of memory capacity and prediction accuracy.
result Parallel deep ESNs have equivalent memory capacity to shallow ESNs, while series deep ESNs have smaller memory capacity.
A new approach predicts next observations without explicit decoding for better control.
problem High-dimensional observations and unknown dynamics in real-world control tasks.
method Proposes a novel information-theoretic LCE approach using predictive coding to develop a decoder-free model.
result The model reliably learns a controllable latent space leading to superior performance.
This study tackles mutual fund portfolio prediction, focusing on novel items.
problem Predicting novel items in mutual fund portfolios is challenging and less explored.
method Created a comprehensive benchmark dataset and evaluated various recommender system models.
result Autoencoder-based approaches outperform state-of-the-art models in predicting novel items.
City2City translates place representations across cities using language translation techniques.
problem Lack of inter-city perspective in place representations from mobility data.
method Applied unsupervised machine language translation methods to translate place representations across different cities.
result Successfully translated place representations from one city to another, validated using landuse data.
The study compares weather prediction algorithms and finds Nexting performs well for slowly varying signals.
problem Comparing prediction algorithms for short-term weather forecasting.
method Evaluation of neural networks, regression trees, and Nexting on historical weather data.
result Nexting performed well for slowly varying signals with sufficient training data.
Law explains how LLMs learn to predict next tokens.
problem Understanding how LLMs process input data internally.
method Introduced a precise law governing token embeddings in LLMs.
result Each layer equally contributes to next-token prediction accuracy.
Study predicts configurational energy of high entropy alloys using Bayesian methods.
problem Accurately predict the configurational energy of high entropy alloys with limited data.
method Robust data-driven framework based on Bayesian approaches, including effective pair interaction (EPI) model and ensemble sampling.
result Effective prediction of configurational energy with small data, demonstrating robust performance.
The study explores how Transformers predict the next token in a sequence.
problem Understanding the mechanism behind Transformers' autoregressive learning ability.
method Exploring the approximation ability of Transformers for next-token prediction through specific instances and a causal kernel descent method.
result Transformer models can learn context-dependent functions f for next-token prediction based on past and current observations. This paper examines how different decoding algorithms for LLMs align with various goals.
problem Consistency of decoding algorithms with different goals in LLMs.
method Analysis of greedy, lookahead, random sampling, and temperature-scaled random sampling algorithms.
result Random sampling is consistent with the true probability distribution, but other goals require optimal algorithms for specific probability distributions.
Paper proposes new features and deep learning models for mid-price prediction.
problem Challenging task of predicting mid-price movement based on LOB data.
method Developed new handcrafted features and deep learning models.
result Deep learning models can predict mid-price movement and time to change.
In order to figure out and to forecast the emergence phenomena of social systems, we propose several probabilistic models for the analysis of financial markets, especially around a crisis. We first attempt to visualize the collective behaviour of markets during a financial crisis through cross-correlations between typi…
Predicts user next location using CDR data.
problem Predicting user's next location using CDR data.
method Data processing framework with deep neural network model.
result Error of prediction decreased from 74% to 55%.
Predictive business process monitoring methods exploit logs of completed cases of a process in order to make predictions about running cases thereof. Existing methods in this space are tailor-made for specific prediction tasks. Moreover, their relative accuracy is highly sensitive to the dataset at hand, thus requiring…
Market forecasts converge to true values if some agents are correct.
problem Convergence of market forecasts in dynamic prediction markets.
method Dynamic model of prediction market with agents making forecasts.
result Aggregated market forecasts converge to conditional expectations.
User smeznar achieved 8th place in PGDL by predicting generalization of deep learning models.
problem Understanding and predicting generalization in deep learning models.
method Creating simple metrics and finding their best combination for automatic testing on a dataset.
result Combination of various properties of neural network architectures can be used for generalization prediction.
Paper proposes a GAN-based method for better next event prediction in business processes.
problem Insufficient training data and sub-optimal network configuration limit deep learning approaches to next event prediction.
method Adversarial training framework using Generative Adversarial Networks (GANs) for sequential temporal data.
result The proposed approach achieves at least as good accuracy as non-adversarial methods and outperforms them in accuracy and prediction earliness.
Improves event prediction in complex processes using Petri nets and deep learning.
problem Predicting the next event in complex processes given a state.
method Enhanced Petri net model with time decay functions and deep learning.
result Significant performance improvements over state-of-the-art methods.
This paper explores how Transformers predict next tokens in autoregressive tasks.
problem Understanding the success of Transformers in autoregressive learning.
method Trained a Transformer on a next-token prediction task, focusing on commuting orthogonal matrices.
result Trained Transformers can be seen as implementing gradient descent for a specific objective function.
Pre-training improves model coverage, crucial for downstream performance.
problem Understanding why pre-training enhances model performance.
method Coverage principle, focusing on next-token prediction and model quality.
result Coverage generalizes faster than cross-entropy, improving downstream performance.
Research improves federated text models for next word prediction.
problem Training models on distributed devices efficiently and effectively.
method Employ transfer learning in federated learning for next word prediction.
result Enhancements to current baselines with pretrained embeddings and whole model pretraining.
The paper improves neural network predictions by integrating process knowledge.
problem Improving neural network predictions for process execution data.
method Integrates background process knowledge into neural networks with attention mechanisms.
result Improves prediction accuracy for process execution data.
MPE models traffic trajectory data to predict next locations.
problem Predicting next locations from traffic trajectory data.
method Mobility pattern embedding model MPE.
result MPE significantly outperforms state-of-the-art methods in next location prediction.
Predicts next item in sequential bundles using Transformers.
problem Predicting next item in sequentially consumed bundles.
method Used custom Transformers, GPT-3, LSTM, reinforcement learning, Markov models.
result Custom Transformer with decoder-only architecture most accurate.
Next-gen reservoir computers fail to predict complex processes, highlighting need for better architectures.
problem Predicting complex, non-Markovian processes with recurrent neural networks.
method Lower bound from Fano's inequality and analysis of large probabilistic state machines.
result Next-generation reservoir computers have an error probability at least 60% higher than optimal for highly non-Markovian processes.
A new method improves language model extrapolation without changing training.
problem Improving language model performance on sequences not seen during training.
method Perturbation-based training procedure for language models.
result The method consistently improves out-of-support prediction.
Predicts traffic flow using reinforcement learning and sensor data.
problem Accurately predict expanding and evolving long-term streaming traffic networks.
method Formulates the problem as a continuous reinforcement learning task, where the agent predicts future traffic based on sensor data.
result The approach improves accuracy in predicting traffic flow by updating the agent's state representation over time.
Reduced-gate LSTM improves spatiotemporal prediction with less parameters.
problem Next-frame video prediction in deep learning.
method Predictive coding framework with reduced-gate convolutional LSTM.
result Reduced-gate model achieves equal or better accuracy with fewer parameters.
Machine learning detects regime shifts in online game-experiments with high accuracy.
problem Detecting regime shifts in online social systems.
method Gradient-boosted decision trees with memory-retaining features.
result Significantly outperforms standard early warning indicators.
Gradient descent biases linear models in next-token prediction towards data entropy.
problem Optimization bias in next-token prediction models.
method Analysis of gradient descent on linear models with sparse conditional distributions.
result Gradient descent selects parameters that equate token logits differences to log-odds in the data subspace.
Interdisciplinary comparison of sequence modeling methods for next-element prediction.
problem Comparing sequence modeling methods across different fields.
method Experimental evaluation of four real-life sequence datasets using machine learning, process mining, and grammar inference techniques.
result Machine learning techniques outperform interpretability-focused methods in next-element prediction accuracy.
The paper analyzes the training dynamics of a transformer for next-token prediction.
problem Understanding the non-asymptotic performance of transformers in next-token prediction.
method Characterizes training dataset properties, designs a two-stage training algorithm, and analyzes attention gradient properties.
result Trained transformers converge sub-linearly to max-margin solutions and exhibit linear convergence in cross-entropy loss.
Language models help text classification tasks by predicting next words.
problem Lack of theoretical understanding of why language models perform well on downstream tasks.
method Mathematical study of the connection between next word prediction and text classification, formalizing it and quantifying the benefit.
result Language models that are ε-optimal in cross-entropy learn features that can solve classification tasks with linear approximation.