Sparse attention model reduces long-context inference time with exponential accuracy guarantees.
problem Efficiently processing long-context queries in large language models.
method Formalizes attention as a projection onto key vectors, analyzes entropic relaxation, and introduces Vashista Sparse Attention.
result Sparse attention concentrates on a constant-size active face, leading to exponential decay of inactive tokens' mass and linear scaling of active face error.
Timer-XL predicts multidimensional time series using a unified Transformer approach.
problem Unified time series forecasting across various tasks and contexts.
method Decoder-only Transformers with a universal TimeAttention mechanism and deft position embedding.
result State-of-the-art performance across multiple forecasting benchmarks.
Transformer models waste resources on long-context tasks.
problem Redundant attention computations in Transformer models for long-context tasks.
method Reformulate sequence modeling as supervised learning, analyze attention sparsity, formulate attention optimization as linear coding problem, propose Dynamic Group Attention.
result DGA reduces computational costs while maintaining performance.
The paper tackles long-context linear system identification with improved sample complexity bounds.
problem Identifying dynamical systems with long dependencies over fixed context windows.
method Established sample complexity bounds for systems with linear dependencies over a context window of length p.
result The learning process is not hindered by slow mixing properties in extended context windows.
Scales attention for long contexts in LLMs.
problem Development of attention mechanisms for long context inference.
method Scale-invariant total attention and sparsity conditions, with a position-dependent transformation of logits.
result Scale-invariant attention scheme improves validation loss and long-context retrieval.
Hydra boosts efficiency for long-context reasoning in resource-constrained settings.
problem Quadratic complexity of transformers limits long-context reasoning in resource-constrained systems.
method Hydra uses a modular architecture with adaptive routing between sparse global attention, mixture-of-experts, and dual memories.
result Hydra achieves significant throughput and accuracy improvements for long-context reasoning.
Mamba struggles with long context lengths, but spectrum scaling improves performance.
problem Mamba's performance degrades with increasing context length.
method Spectrum scaling applied to pre-trained Mamba models to improve long-context generalization.
result Spectrum scaling significantly improves performance in long-context settings.
FibQuant improves KV-cache compression for long-context inference.
problem Memory traffic bottleneck in long-context inference due to KV cache growth.
method Introduces FibQuant, a universal vector quantizer that combines Beta-quantile radii, Fibonacci/Roberts-Kronecker directions, and Lloyd-Max refinement.
result FibQuant achieves high compression rates with minimal loss in attention cosine similarity.
New Performer model tackles long-sequence protein modeling.
problem Challenges of training complex Transformer models for long sequences.
method Linearly scalable long-context Transformer architecture, Performer.
result Performer provides strong theoretical guarantees and is effective for protein sequence modeling.
Cisco introduces a new time series model for better forecasting.
problem Improving time series forecasting accuracy.
method Developed a new multiresolution decoder-only model trained on large datasets.
result The new model achieves superior performance on observability datasets.
Analysis of ridge regression under concept shift reveals nontrivial effects on generalization performance.
problem Understanding and mitigating the impact of distribution shift in machine learning models.
method Derivation of exact prediction risk expression in the thermodynamic limit for ridge regression under concept shift.
result Reveals a phase transition and nonmonotonic data dependence of test performance under concept shift.
Enhanced TSFMs improve time series forecasting accuracy and reliability.
problem Variance, bias, and uncertainty in TSFMs' predictions on real data.
method Statistical and ensemble techniques including bagging, stacking, residual modeling, and prediction intervals.
result Hybrid models consistently outperform standalone TSFMs across multiple horizons.
A new framework for efficient sequence maps using Bayesian filtering and covariance.
problem Designing efficient recurrent sequence maps from explicit memory assumptions.
method Design-model framework, exact Bayesian filtering, query-dependent readout, linear-Gaussian instantiation.
result Improved robustness and retrieval performance across various benchmarks.
Existing language models such as n-grams for software code often fail to capture a long context where dependent code elements scatter far apart. In this paper, we propose a novel approach to build a language model for software code to address this particular issue. Our language model, partly inspired by human memory, i…
Jukebox generates high-fidelity songs with singing in raw audio.
problem Generating music with singing in raw audio.
method Multi-scale VQ-VAE for compression, autoregressive Transformers for modeling.
result Generates high-fidelity and diverse songs with coherence up to multiple minutes.
Express improves causal attention guarantees for language models.
problem Improving causal attention guarantees for language models.
method Introducing Express, a tool for converting non-causal attention into causal with matching guarantees.
result Express improves causal attention guarantees to log^(3/2)(n)/s with minimal memory and compression overhead.
ZeroS improves Transformers by adding negative weights, matching or beating softmax attention.
problem Limited performance of linear attention methods, especially in long context sequences.
method Proposes Zero-Sum Linear Attention (ZeroS) that removes the zero-order term and reweights zero-sum softmax residuals.
result ZeroS matches or exceeds standard softmax attention across various benchmarks, theoretically expanding representable functions.
Develops PromptShift-CRC for drift-aware conformal risk control in foundation models under prompt and domain shift.
problem Fixed calibration risk in foundation models due to prompt and domain shift.
method Embeds prompts and responses, measures drift, gives more weight to recent examples, and updates risk online.
result Develops method to control risk up to terms for distribution mismatch and weighted quantile uncertainty.
Unified framework for critical scaling of inverse temperature in self-attention.
problem Conflicting inverse-temperature laws for long-context self-attention.
method Counting gaps and defining an upper-tail accumulation scale.
result Critical inverse-temperature scale determined by gap-counting function.
New insights into attention mechanisms reveal dramatic trade-offs between rank and heads.
problem Dramatic trade-offs between rank and number of heads in attention mechanisms.
method Presented a simple target function and proved theoretical limits.
result Full-rank attention is necessary for long contexts, while low-rank is sufficient for short ones.
FinanceBench benchmarks LLMs on financial QA, revealing limitations.
problem Evaluating LLMs' performance on financial question answering.
method Developed a comprehensive test suite (FinanceBench) with 10,231 questions, tested 16 models, and manually reviewed answers.
result Existing LLMs have significant limitations for financial QA, especially GPT-4-Turbo.
Transformers learn chain-of-thought reasoning for longer problems, proving length generalization.
problem Challenging problems require deeper reasoning, but how do models generalize this to longer tasks?
method Theoretical analysis of transformers on synthetic state-tracking tasks, proving length generalization through attention concentration.
result Transformers can learn chain-of-thought reasoning for longer problems, proving length generalization.
Improves many-shot learning by optimizing and generating influential examples.
problem Limited performance in many-shot in-context learning (ICL).
method Iterative optimization and generation of influential examples.
result Significant improvements across various tasks using BRIDGE.
Language models fail to execute simple steps, showing gating and binding errors.
problem Procedural hallucinations in language models, failing to execute simple steps.
method Analyzed long-context binding tasks, identifying gating and binding errors.
result Procedural errors are due to gating and binding failures, with recency bias contributing to the latter.
A new algorithm reduces memory usage for long token attention in streaming applications.
problem Memory inefficiency in computing attention for long documents.
method One-pass streaming algorithm using sublinear space storage.
result Super-efficient memory usage for long token attention.
Neural language models (LMs) based on recurrent neural networks (RNN) are some of the most successful word and character-level LMs. Why do they work so well, in particular better than linear neural LMs? Possible explanations are that RNNs have an implicitly better regularization or that RNNs have a higher capacity for …
Fuzzy prediction sets generalize binary predictions to include elements at varying confidence levels.
problem Binary prediction sets are limited; fuzzy prediction sets offer richer guarantees.
method Generalize prediction sets to fuzzy sets, showing they are e-values with merging properties.
result Optimal e-values lead to optimal fuzzy prediction sets, including optimal conformal prediction.
This paper re-examines conformal e-prediction and its advantages over conformal prediction.
problem The relationship between conformal prediction and conformal e-prediction.
method Systematic re-examination of conformal prediction and conformal e-prediction from a modern perspective.
result Conformal e-prediction has advantages such as ease of designing conditional predictors and guaranteed validity of cross-predictors.
Self-calibrating conformal prediction improves interval efficiency and offers a practical alternative.
problem Improving the reliability and uncertainty quantification of machine learning predictions.
method Combines Venn-Abers calibration and conformal prediction for binary and regression problems.
result Improves interval efficiency through model calibration and offers practical alternatives.
Study uses deep learning to predict asset prices, finds complex target processes lead to meaningless predictions.
problem Complexity of successful price prediction models hinders understanding.
method Deep learning models for high-frequency price prediction, focusing on volatility and directional prediction.
result Inadequately defined target price process renders predictions meaningless.
The paper emphasizes the importance of joint predictions over marginal predictions for decision-making.
problem The need for accurate joint predictions in decision-making problems.
method The paper analyzes combinatorial decision problems, sequential predictions, and multi-armed bandits, introducing an approximate Thompson sampling algorithm and new regret bounds.
result Accurate joint predictions are essential for good performance in decision-making problems.
Behavior modification improves prediction accuracy by nudging user behavior.
problem Improving prediction accuracy using behavior modification techniques.
method Combining prediction and behavior modification with reinforcement learning algorithms.
result Behavior modification can make predictions more certain but may not generalize.
Conformal predictive systems are a recent modification of conformal predictors that output, in regression problems, probability distributions for labels of test observations rather than set predictions. The extra information provided by conformal predictive systems may be useful, e.g., in decision making problems. Conf…
Predictions can shape outcomes, study helps predict these effects.
problem Understanding how predictions influence real-world outcomes.
method Causal identifiability analysis of prediction-covariate-outcome relationships.
result Standard supervised learning can identify transferable relationships from predictions.
Prediction problems often admit competing models that perform almost equally well. This effect challenges key assumptions in machine learning when competing models assign conflicting predictions. In this paper, we define predictive multiplicity as the ability of a prediction problem to admit competing models with confl…
Proposes feature conformal prediction for broader application in semantic feature spaces.
problem Establishing valid prediction intervals in semantic feature spaces.
method Extends conformal prediction to semantic feature spaces using deep representation learning.
result Feature conformal prediction outperforms regular conformal prediction under mild assumptions.
Acute kidney injury (AKI) commonly occurs in hospitalized patients and can lead to serious medical complications. In order to optimally predict AKI before it develops at any time during a hospital stay, we present a novel framework in which AKI is continually predicted automatically from EHR data over the entire hospit…
AutoCP automates the construction of accurate prediction intervals.
problem Creating valid and accurate prediction intervals for machine learning models.
method AutoML framework that optimizes prediction interval length for better accuracy and less conservatism.
result AutoCP significantly outperforms benchmark algorithms in constructing accurate prediction intervals.
Proposes a method to apply conformal prediction to probabilistic time series forecasting models.
problem Obtaining accurate prediction regions for multi-step time series forecasting with probabilistic models.
method Conformalises conditional normalising flows to generate potentially disjoint prediction regions.
result Improves predictive efficiency in time series forecasting with multimodal distributions.
ICP improves prediction intervals for continuous outcomes at lower computational cost.
problem Systematic bias in point predictions that undermines their use in decision-making.
method Develops Isotonic Conformal Prediction (ICP) framework to decouple calibration from prediction-set construction.
result SICP and TICP procedures match SC-CP coverage at lower computational cost.
Optimizes predictions for specific tasks using parametrized decision analysis.
problem Optimizing predictions for specific decision tasks of interest.
method Designs a class of parametrized actions for Bayesian decision analysis.
result Derives efficient and interpretable solutions for various action parametrizations and loss functions.
New measures for prediction validity and consonant plausibility introduced.
problem Challenges in predicting future observations and quantifying prediction uncertainty.
method Introducing Type-2 validity and using consonant plausibility measures and conformal prediction.
result Achieving both Type-1 and Type-2 validity through consonant plausibility measures and conformal prediction.
RFpredInterval package builds prediction intervals for random forests and boosted forests.
problem Quantifying uncertainty in random forest and boosted forest point predictions.
method 16 methods to build prediction intervals with random forests and boosted forests.
result The proposed method outperforms existing methods in building prediction intervals.
FPPI selectively uses predictions to improve inference efficiency.
problem Improving statistical inference with limited labeled data and heterogeneous prediction quality.
method Filtered Prediction-Powered Inference (FPPI) framework.
result FPPI achieves strictly improved asymptotic efficiency compared to existing methods.
COP improves online conformal prediction by incorporating data patterns, leading to tighter prediction sets.
problem Overly conservative prediction sets in online conformal prediction methods when data distribution shifts.
method Conformal Optimistic Prediction (COP) incorporating estimated cumulative distribution function of non-conformity scores.
result COP produces tighter prediction sets with valid coverage guarantees, outperforming other methods.
Most existing examples of full conformal predictive systems, split-conformal predictive systems, and cross-conformal predictive systems impose severe restrictions on the adaptation of predictive distributions to the test object at hand. In this paper we develop split-conformal and cross-conformal predictive systems tha…
This paper studies trade-offs in private prediction methods.
problem Leakage of training data information in machine learning predictions.
method Private training and private prediction methods with trade-offs.
result Private training methods outperform private prediction methods in various settings.
Conformal prediction helps quantify uncertainty but its use by humans is unclear.
problem Uncertainty quantification in predictions for human decision making.
method Decision theoretic framework for evaluating predictive uncertainty.
result Conformal prediction sets and human decision making goals are in tension.