Research improves federated text models for next word prediction.
problem Training models on distributed devices efficiently and effectively.
method Employ transfer learning in federated learning for next word prediction.
result Enhancements to current baselines with pretrained embeddings and whole model pretraining.
Language models help text classification tasks by predicting next words.
problem Lack of theoretical understanding of why language models perform well on downstream tasks.
method Mathematical study of the connection between next word prediction and text classification, formalizing it and quantifying the benefit.
result Language models that are ε-optimal in cross-entropy learn features that can solve classification tasks with linear approximation.
Neural language models have been widely used in various NLP tasks, including machine translation, next word prediction and conversational agents. However, it is challenging to deploy these models on mobile devices due to their slow prediction speed, where the bottleneck is to compute top candidates in the softmax layer…
Paper trains language models without memorizing user data.
problem Training production-scale language models with user privacy.
method Federated Learning with Differentially Private Federated Averaging (DP-FedAvg).
result Demonstrated training of a production NWP model with user-level differential privacy.
We present a recurrent neural network memory that uses sparse coding to create a combinatoric encoding of sequential inputs. Using several examples, we show that the network can associate distant causes and effects in a discrete stochastic process, predict partially-observable higher-order sequences, and enable a DQN a…
Multi-task learning (MTL) has recently contributed to learning better representations in service of various NLP tasks. MTL aims at improving the performance of a primary task, by jointly training on a secondary task. This paper introduces automated tasks, which exploit the sequential nature of the input data, as second…
Neural Machine Translation (NMT) generates target words sequentially in the way of predicting the next word conditioned on the context words. At training time, it predicts with the ground truth words as context while at inference it has to generate the entire sequence from scratch. This discrepancy of the fed context l…
Local adaptation improves federated learning models.
problem Improving accuracy of federated learning models on non-iid data.
method Local adaptation techniques (fine-tuning, multi-task learning, knowledge distillation).
result Participants benefit from local adaptation, improving federated model accuracy.
Proving that next-token prediction makes language models generate coherent long documents.
problem Understanding why language models generate coherent documents despite focusing on next-token prediction.
method Proving the power of next-token prediction in learning longer-range structure using Recurrent Neural Networks (RNN).
result Optimizing next-token prediction in RNNs yields a model that closely approximates the training distribution, even for long-range coherence.
LLMs encode latent topic distributions, suggesting Bayesian inference.
problem Capturing topic structure from large language models.
method Connecting LLM optimization to implicit Bayesian inference and de Finetti's theorem.
result LLMs recover latent topic distributions, matching LDA-generated topics.
Optimizes communication in federated learning using rate-distortion theory.
problem Reduces communication cost in federated learning while maintaining model accuracy.
method Applies rate-distortion theory to model updates, proposing distortion as a proxy for accuracy.
result Near-optimal communication reduction, outperforming other methods on a FL benchmark.
Deep models learn to parse complex language structures from local data patterns.
problem Understanding how deep models parse and represent language structures.
method Introduced tunable probabilistic context-free grammars and a learning algorithm inspired by deep networks.
result Data correlations across scales enable hierarchical language representations.
In sequence learning tasks such as language modelling, Recurrent Neural Networks must learn relationships between input features separated by time. State of the art models such as LSTM and Transformer are trained by backpropagation of losses into prior hidden states and inputs held in memory. This allows gradients to f…
Sequence-to-Sequence (seq2seq) modeling has rapidly become an important general-purpose NLP tool that has proven effective for many text-generation and sequence-labeling tasks. Seq2seq builds on deep neural language modeling and inherits its remarkable accuracy in estimating local, next-word distributions. In this work…
Recent progress on automatic generation of image captions has shown that it is possible to describe the most salient information conveyed by images with accurate and meaningful sentences. In this paper, we propose an image caption system that exploits the parallel structures between images and sentences. In our model, …
A new algorithm optimizes softmax units in large language models.
problem Efficiently computing gradients for large-scale language models.
method Zero-th Order method for approximating gradients.
result The algorithm converges and efficiently computes gradients.
Enhances image captioning with novel context combination methods.
problem Improving machine learning for image captioning with structured learning and meaningful interpretation.
method Combines Feature Distribution Composition (FDC), Multiple Role Representation Crossover (MRRC) attention layers, and language decoder.
result Significantly improved image captioning performance (35.3%) and established new standards.
Federated Learning prioritizes client data contributions for better model quality.
problem Privacy concerns and reluctance to share private data in machine learning.
method Prioritizes client data contributions in Federated Learning by assigning scores based on defined criteria.
result The proposed approach yields a higher quality global model compared to standard Federated Learning.
Fuzzy prediction sets generalize binary predictions to include elements at varying confidence levels.
problem Binary prediction sets are limited; fuzzy prediction sets offer richer guarantees.
method Generalize prediction sets to fuzzy sets, showing they are e-values with merging properties.
result Optimal e-values lead to optimal fuzzy prediction sets, including optimal conformal prediction.
Self-calibrating conformal prediction improves interval efficiency and offers a practical alternative.
problem Improving the reliability and uncertainty quantification of machine learning predictions.
method Combines Venn-Abers calibration and conformal prediction for binary and regression problems.
result Improves interval efficiency through model calibration and offers practical alternatives.
Study uses deep learning to predict asset prices, finds complex target processes lead to meaningless predictions.
problem Complexity of successful price prediction models hinders understanding.
method Deep learning models for high-frequency price prediction, focusing on volatility and directional prediction.
result Inadequately defined target price process renders predictions meaningless.
The paper emphasizes the importance of joint predictions over marginal predictions for decision-making.
problem The need for accurate joint predictions in decision-making problems.
method The paper analyzes combinatorial decision problems, sequential predictions, and multi-armed bandits, introducing an approximate Thompson sampling algorithm and new regret bounds.
result Accurate joint predictions are essential for good performance in decision-making problems.
Behavior modification improves prediction accuracy by nudging user behavior.
problem Improving prediction accuracy using behavior modification techniques.
method Combining prediction and behavior modification with reinforcement learning algorithms.
result Behavior modification can make predictions more certain but may not generalize.
Conformal predictive systems are a recent modification of conformal predictors that output, in regression problems, probability distributions for labels of test observations rather than set predictions. The extra information provided by conformal predictive systems may be useful, e.g., in decision making problems. Conf…
Predictions can shape outcomes, study helps predict these effects.
problem Understanding how predictions influence real-world outcomes.
method Causal identifiability analysis of prediction-covariate-outcome relationships.
result Standard supervised learning can identify transferable relationships from predictions.
Prediction problems often admit competing models that perform almost equally well. This effect challenges key assumptions in machine learning when competing models assign conflicting predictions. In this paper, we define predictive multiplicity as the ability of a prediction problem to admit competing models with confl…
Proposes feature conformal prediction for broader application in semantic feature spaces.
problem Establishing valid prediction intervals in semantic feature spaces.
method Extends conformal prediction to semantic feature spaces using deep representation learning.
result Feature conformal prediction outperforms regular conformal prediction under mild assumptions.
Acute kidney injury (AKI) commonly occurs in hospitalized patients and can lead to serious medical complications. In order to optimally predict AKI before it develops at any time during a hospital stay, we present a novel framework in which AKI is continually predicted automatically from EHR data over the entire hospit…
AutoCP automates the construction of accurate prediction intervals.
problem Creating valid and accurate prediction intervals for machine learning models.
method AutoML framework that optimizes prediction interval length for better accuracy and less conservatism.
result AutoCP significantly outperforms benchmark algorithms in constructing accurate prediction intervals.
Proposes a method to apply conformal prediction to probabilistic time series forecasting models.
problem Obtaining accurate prediction regions for multi-step time series forecasting with probabilistic models.
method Conformalises conditional normalising flows to generate potentially disjoint prediction regions.
result Improves predictive efficiency in time series forecasting with multimodal distributions.
ICP improves prediction intervals for continuous outcomes at lower computational cost.
problem Systematic bias in point predictions that undermines their use in decision-making.
method Develops Isotonic Conformal Prediction (ICP) framework to decouple calibration from prediction-set construction.
result SICP and TICP procedures match SC-CP coverage at lower computational cost.
Optimizes predictions for specific tasks using parametrized decision analysis.
problem Optimizing predictions for specific decision tasks of interest.
method Designs a class of parametrized actions for Bayesian decision analysis.
result Derives efficient and interpretable solutions for various action parametrizations and loss functions.
This paper discusses a counterpart of conformal prediction for e-values, conformal e-prediction. Conformal e-prediction is conceptually simpler and had been developed in the 1990s as a precursor of conformal prediction. When conformal prediction emerged as result of replacing e-values by p-values, it seemed to have imp…
RFpredInterval package builds prediction intervals for random forests and boosted forests.
problem Quantifying uncertainty in random forest and boosted forest point predictions.
method 16 methods to build prediction intervals with random forests and boosted forests.
result The proposed method outperforms existing methods in building prediction intervals.
FPPI selectively uses predictions to improve inference efficiency.
problem Improving statistical inference with limited labeled data and heterogeneous prediction quality.
method Filtered Prediction-Powered Inference (FPPI) framework.
result FPPI achieves strictly improved asymptotic efficiency compared to existing methods.
COP improves online conformal prediction by incorporating data patterns, leading to tighter prediction sets.
problem Overly conservative prediction sets in online conformal prediction methods when data distribution shifts.
method Conformal Optimistic Prediction (COP) incorporating estimated cumulative distribution function of non-conformity scores.
result COP produces tighter prediction sets with valid coverage guarantees, outperforming other methods.
Most existing examples of full conformal predictive systems, split-conformal predictive systems, and cross-conformal predictive systems impose severe restrictions on the adaptation of predictive distributions to the test object at hand. In this paper we develop split-conformal and cross-conformal predictive systems tha…
This paper studies trade-offs in private prediction methods.
problem Leakage of training data information in machine learning predictions.
method Private training and private prediction methods with trade-offs.
result Private training methods outperform private prediction methods in various settings.
Conformal prediction helps quantify uncertainty but its use by humans is unclear.
problem Uncertainty quantification in predictions for human decision making.
method Decision theoretic framework for evaluating predictive uncertainty.
result Conformal prediction sets and human decision making goals are in tension.
New tree splitting criteria improve probabilistic predictions.
problem Improving tree-based nonparametric predictive distributions.
method Using proper scoring rules for tree splitting criteria.
result Trees with new splitting criteria produce better predictive distributions.
Prediction is arguably one of the most basic functions of an intelligent system. In general, the problem of predicting events in the future or between two waypoints is exceedingly difficult. However, most phenomena naturally pass through relatively predictable bottlenecks---while we cannot predict the precise trajector…
New method improves predictive systems with better theoretical guarantees.
problem Constructing predictive systems with out-of-sample calibration guarantees.
method Residual Distribution Predictive Systems (RDPs) that nest conformal predictive systems and offer flexibility.
result Empirically, RDPs perform competitively with conformal predictive systems and can be implemented with various regression methods.
Paper introduces PCP for efficient, reliable predictive inference.
problem Developing reliable predictive inference methods for target variables.
method Probabilistic conformal prediction using conditional random samples.
result PCP provides sharper predictive sets compared to existing methods.
A framework for private prediction sets using conformal prediction and differential privacy.
problem Jointly addressing reliability and privacy in machine learning predictions.
method Split conformal prediction with privatized quantile subroutine.
result Private prediction sets can be generated from privately-trained models.
Prediction of future observations is an important and challenging problem. The two mainstream approaches for quantifying prediction uncertainty use prediction regions and predictive distributions, respectively, with the latter believed to be more informative because it can perform other prediction-related tasks. The st…
Variational Prediction simplifies Bayesian inference without test time costs.
problem Bayesian inference's computational costs and posterior predictive distribution marginalization.
method Variational Prediction learns a variational approximation to the posterior predictive distribution using a variational bound.
result Directly learns a variational approximation to the posterior predictive distribution without test time marginalization costs.
Paper uses GNN and conformal prediction for accurate edge weight prediction.
problem Predicting edge weights on graphs for various applications.
method Graph Neural Network (GNN) with conformal prediction and error reweighting.
result Our method provides better coverage and efficiency than baselines.
Conformal Prediction is a machine learning methodology that produces valid prediction regions under mild conditions. In this paper, we explore the application of making predictions over multiple data sources of different sizes without disclosing data between the sources. We propose that each data source applies a trans…