A new model for context-aware recommendations using LSTM and latent context.
problem Challenges in incorporating context into recommendation models, especially sparsity and dimensionality issues.
method Sequential latent context modeling using LSTM, reducing context dimensions to a compressed latent space.
result The proposed SLCM outperforms state-of-the-art CARS models in empirical analysis.
HCFContext predicts mobile context using collaborative filtering and homomorphic encryption.
problem Accurate mobile context determination for enterprise policies.
method Proposes HPContext and HCFContext models using sequential history and collaborative filtering, with privacy-preserving homomorphic encryption.
result HCFContext enhances context prediction by leveraging related users' observations.
Enhances activity recognition in wearable computing with context awareness and uncertainty quantification.
problem Context-dependent activity recognition and unknown contexts in wearable computing.
method Developed the α-{eta} network coupled with uncertainty quantification (UQ) based on maximum entropy.
result Improved accuracy and F-score by 10% through high-level context identification.
Introduces CStrees for modeling context-specific causal models from observational and interventional data.
problem Modeling context-specific causal relationships from mixed data types.
method Introduces CStrees with a novel factorization criterion and graphical characterization for context-specific conditional independence models.
result Derives a graphical characterization of model equivalence for observational CStrees and extends it to CStree models under context-specific interventions.
MLPs can approximate any function in context, challenging the importance of in-context universality.
problem Understanding why transformers are more effective than classical models.
method Proved MLPs with trainable activation functions are universal in context.
result Transformer success is likely due to factors other than in-context universality.
Proposes BehavDT model for context-aware user behavior prediction.
problem Building a context-aware predictive model based on diverse user behavioral activities.
method Introduces BehavDT, a behavioral decision tree that considers user behavior-oriented generalization.
result BehavDT model outperforms traditional machine learning approaches in predicting user diverse behaviors considering multi-dimensional contexts.
ContextFlow++ improves generative models by conditioning on mixed-variable contexts.
problem Lack of effective methods for context conditioning in flow-based generative models.
method Proposes ContextFlow++ with additive conditioning and mixed-variable architecture.
result ContextFlow++ achieves higher performance metrics and faster training.
A new bandit model with stochastic context distributions and UCB algorithm.
problem Learning optimal actions in a stochastic environment with hidden contexts.
method Stochastic contextual bandit model and UCB algorithm.
result Order-optimal high-probability bound on cumulative regret for linear and kernelized reward functions.
This study examines how sequential correlations affect in-context learning in sequence models.
problem Understanding how in-context learning works with sequentially correlated data.
method Extended linear regression model to sequentially correlated data, tested on transformer architectures.
result Sequential correlations alter the effective context length and attention architecture effectiveness.
Transformers can scale both context and task, but MLPs can only scale task.
problem Understanding and scaling In-Context Learning in transformers.
method Simplified transformer architecture, feature map, and MLP combination.
result Simplified transformer can perform ICL and context-scaling but not task-scaling.
New insights into how depth and width affect in-context learning in deep models.
problem Understanding how various resources impact in-context learning in deep models.
method Analyzed linear regression in a deep linear self-attention model, varying resources like depth, width, context length, and training steps.
result Increasing depth improves in-context learning even at infinite context length, contrary to previous findings.
Paper argues context equals environment, improving AI generalization.
problem AI models struggle to generalize in new environments.
method In-Context Risk Minimization (ICRM) algorithm.
result ICRM leads to significant out-of-distribution performance improvements.
This work introduces a method to decompose uncertainty in in-context learning for large language models.
problem Understanding the sources of uncertainty in in-context learning for large language models.
method Variational uncertainty decomposition framework without sampling from latent parameter posterior.
result Quantitative and qualitative validation of decomposed epistemic and aleatoric uncertainties.
Bayesian framework uses context trees for efficient time series modeling.
problem Building flexible and interpretable mixture models for real-valued time series.
method Hierarchical Bayesian modelling with context trees and autoregressive models.
result The methods outperform state-of-the-art techniques on simulated and real-world experiments.
AppsPred predicts smartphone app usage based on context.
problem Predicting personalized usage behavior of smartphone apps based on contexts.
method Random Forest machine learning technique considering multi-dimensional contexts.
result AppsPred significantly outperforms other machine learning approaches in predicting smartphone apps.
Transformers can be hijacked by context, but deeper models are more robust.
problem Robustness of Transformers against context hijacking for linear classification.
method Developed a theoretical analysis on the robustness of linear transformers, considering model depth, training context lengths, and number of hijacking context tokens.
result Deeper transformers are more robust to context hijacking.
Fine-tuning harms in-context learning, but restricting updates to the value matrix improves zero-shot performance.
problem Fine-tuning harms in-context learning, reducing zero-shot performance on unseen tasks.
method Theoretical analysis of linear attention models, identifying conditions for degraded few-shot performance.
result Restricting updates to the value matrix improves zero-shot performance while preserving in-context learning.
Importance of visual context in scene understanding tasks is well recognized in the computer vision community. However, to what extent the computer vision models for image classification and semantic segmentation are dependent on the context to make their predictions is unclear. A model overly relying on context will f…
Optimally explores dynamical systems with varying properties using context inference.
problem Learning dynamics models for systems with varying properties.
method Formulates dynamics models as stochastic processes conditioned on a latent context variable inferred from system transitions. Uses probabilistic formulation to compute optimal action sequences for exploration.
result Demonstrates effectiveness of the method on non-linear toy-problems and reinforcement learning environments.
Contextualized ML learns context-dependent effects using deep learning.
problem Learning heterogeneous and context-dependent effects in data.
method Applying deep learning to the meta-relationship between contextual information and context-specific parametric models.
result Unified framework for cluster analysis and cohort modeling.
We present a new recurrent neural network topology to enhance state-of-the-art machine learning systems by incorporating a broader context. Our approach overcomes recent limitations with extended narratives through a multi-layered computational approach to generate an abstract context representation. Therefore, the dev…
A method for learning a context latent vector to improve generalization in model-based RL.
problem Learning a global dynamics model that can generalize across different dynamics.
method Decomposes learning a global dynamics model into two stages: learning a context latent vector and predicting next states.
result Achieves superior generalization across various simulated robotics and control tasks.
Transformers forecast time series in-context, improving efficiency and performance.
problem Overfitting and limited performance in time series forecasting.
method Reformulate time series forecasting as input tokens, aligning with in-context learning mechanisms.
result Consistently better performance across various settings (full-data, few-shot, zero-shot).
LLMs learn probability density functions in-context, showing distinct learning trajectories.
problem Density estimation of time series data in LLMs.
method Intensive Principal Component Analysis (InPCA) to visualize and analyze LLMs' learning dynamics.
result LLMs follow similar learning trajectories in a low-dimensional InPCA space, distinct from traditional methods.
We present a Bayesian nonparametric framework for multilevel clustering which utilizes group-level context information to simultaneously discover low-dimensional structures of the group contents and partitions groups into clusters. Using the Dirichlet process as the building block, our model constructs a product base-m…
The paper tackles long-context linear system identification with improved sample complexity bounds.
problem Identifying dynamical systems with long dependencies over fixed context windows.
method Established sample complexity bounds for systems with linear dependencies over a context window of length p.
result The learning process is not hindered by slow mixing properties in extended context windows.
Unified view of context-adaptive inference across statistics, meta-learning, and foundation models.
problem Adapting predictive systems to specific situations.
method Unified framework for context-adaptive inference, bridging statistics, meta-learning, and foundation models.
result Explicit and implicit approaches are mathematically equivalent under certain conditions.
New method discovers context effects in choice data.
problem Identifying context effects from choice data is challenging.
method Automatic discovery of context effects from observed choices.
result Automatic discovery of context effects from observed choices.
TL-ANDI distills context from source data to improve transfer learning for TFMs.
problem Limited transfer learning due to context-size constraints and distribution shifts.
method TL-ANDI uses posterior-aware distillation to construct a compact source context and locally distills labels.
result Improves transfer performance by addressing context-size and distribution shifts.
Unified view of context in machine learning models.
problem Exploring context in machine learning models.
method Decompose observation into context-free and context-sensitive parts.
result Significant performance boost in sentence embedding models.
Paper studies linear CMDPs, improving sample complexity for both models.
problem Improving sample complexity for linear CMDPs.
method Proposes novel model-based algorithms for two linear function approximation models.
result Guaranteed ε-suboptimality gap with desired polynomial sample complexity.
Paper proposes linear transformers for efficient in-context learning without context length limitations.
problem Quadratic complexity of softmax transformers limits data processing speed.
method Investigates linear transformers under domain generalization, showing they learn mappings from context distributions to response functions.
result Linear transformers achieve in-context learning with a linear complexity in context length, offering a dimension-independent convergence rate.
Proposes E-MIIM for better mobile phone interruptions management.
problem Improper phone interruptions in daily life activities.
method Ensemble learning based on context-aware mobile telephony model.
result E-MIIM outperforms existing MIIM models in prediction accuracy.
Unified Bayesian model explains in-context learning and activation steering in LLMs.
problem Understanding and controlling the behavior of large language models (LLMs) through prompts and activations.
method Developed a Bayesian model to explain and predict the effects of in-context learning and activation steering.
result Unified model predicts distinct phases and sudden shifts in LLM behavior, explaining prior empirical phenomena.
Enhances neural processes to learn from multiple related datasets.
problem Improving predictions from datasets with shared similarities.
method Developed the in-context in-context learning pseudo-token TNP (ICICL-TNP) to condition on both sets of datapoints and sets of datasets.
result Demonstrated the importance and effectiveness of in-context in-context learning.
MCPCA analyzes shared factors across multiple data contexts.
problem No tools to recover shared factors across multiple contexts.
method Developed a theoretical and algorithmic framework (MCPCA).
result Reveals shared axes of variation across subsets of contexts.
This work introduces RISE to explain LLMs more reliably by distinguishing essential context.
problem Identifying which context elements influence LLM outputs reliably.
method RISE (Redundancy-Insensitive Scoring of Explanation) method.
result RISE provides more robust explanations than traditional methods.
We consider a novel formulation of the multi-armed bandit model, which we call the contextual bandit with restricted context, where only a limited number of features can be accessed by the learner at every iteration. This novel formulation is motivated by different online problems arising in clinical trials, recommende…
The paper shows context vectors are half the dimensions of word vectors.
problem Understanding the relationship between word and context embeddings.
method Starting from probabilistic assumptions, the paper shows context vectors are reflections of word vectors in half the dimensions.
result Context vectors are reflections of word vectors in approximately half the dimensions.
Transformers learn to perform logistic regression in-context.
problem Understanding how transformers learn to perform specific tasks in-context.
method Constructed multi-layer transformers that perform in-context logistic regression through normalized gradient descent.
result Transformers can be trained to perform in-context logistic regression effectively.
Interactions such as double negation in sentences and scene interactions in images are common forms of complex dependencies captured by state-of-the-art machine learning models. We propose Mahé, a novel approach to provide Model-agnostic hierarchical éxplanations of how powerful machine learning models, such as deep ne…
Transformers show better in-context learning resilience under distribution shifts than simple MLPs.
problem Understanding in-context learning under varying distribution shifts.
method Comparing transformers and set-based MLPs on linear regression tasks.
result Transformers better emulate OLS performance and exhibit better resilience to mild distribution shifts.
GPLVMF improves CARS performance by addressing overfitting and context importance.
problem Overfitting and lack of automatic context importance determination in GP-based CARS.
method GPLVMF applies a non-zero mean function and real-valued latent space to improve GP model performance.
result Significant improvement in performance on real datasets and automatic context importance determination.
Bayesian context trees capture complex dependencies in categorical sequences.
problem Complex, long-range dependencies in categorical sequences are not well captured by simple models.
method Parsimonious Bayesian context trees with model-based agglomerative clustering for efficient inference.
result The proposed framework outperforms existing models on real-world data.
Benchmark assesses forecasting models' ability to use textual context.
problem Forecasting models struggle with integrating textual context.
method Introduces a benchmark with numerical and textual data, evaluates various models.
result LLM prompting method outperforms other models.
CAZSL learns to generalize physical interactions from context.
problem Designing models that can generalize over unknown objects during manipulation.
method Context-aware zero-shot learning using Siamese network architecture, embedding space masking, and regularization.
result CAZSL models can generalize to different parameters or features of interacting objects.
New model uses attention for in-context learning of categorical data.
problem Learning from categorical data in context.
method Attention-based network with self-attention and cross-attention layers, using functional gradient descent.
result Model can perform multi-step inference for categorical observations.
The paper introduces contexture theory to characterize representation learning from contexts.
problem Lack of systematic characterization of representation learning methods.
method Characterizes representation learning as learning from the association between input and context variable.
result Contexture theory shows that representations can be approximated by top singular functions of the context.