Active testing for large language models is made more efficient and accurate.
problem Efficient evaluation of large language models with limited labels.
method Cost-saving measures and in-context learning for constructing a surrogate model.
result Significantly more accurate evaluations of LLM performance compared to random data acquisition.
Enhances drug discovery models by understanding human language.
problem Low predictive quality of activity prediction models in drug discovery.
method Proposes a novel architecture with separate chemical and natural language input modules and a contrastive pre-training objective.
result Improves predictive performance on few-shot and zero-shot learning benchmarks.
This paper studies activation sparsity in large language models, finding key trends and implications.
problem Activation sparsity in large language models (LLMs) can be improved for efficiency and interpretability.
method Proposes PPL-p% sparsity, analyzes trends with training data, width-depth ratio, and parameter scale. result ReLU is more efficient for sparsity than SiLU, and deeper architectures can improve sparsity.
ContextBench benchmarks methods for generating linguistically fluent inputs that activate specific latent features in language models.
problem Identifying inputs that trigger specific behaviours or latent features in language models.
method Context modification and benchmarking methods like Evolutionary Prompt Optimisation (EPO) with LLM-assistance and diffusion model inpainting.
result Enhanced methods achieve state-of-the-art performance in balancing elicitation effectiveness and fluency.
Spectral measurements reveal hidden representation geometry in language model training.
problem Hidden internal representation in language model training is hard to examine.
method Empirical protocol using activation covariance and per-sample gradient SVD spectra.
result Batch size affects representation geometry, and activation spectra predict token efficiency.
Token-adaptive FFN design improves LLM expressivity.
problem Fixed activation functions limit FFN expressivity in LLMs.
method Mixture of Activations (MoA) and learnable activations (LA).
result MoA achieves lower terminal loss and better scaling than baselines.
Efficiently identifies good policies by choosing contexts for human feedback.
problem Efficiently acquiring human feedback for preference alignment in large language models.
method Formalizes active exploration as a dueling bandit problem and proposes an active exploration algorithm with a polynomial worst-case regret bound.
result Proposed method outperforms baselines with limited human preferences on various language models and datasets.
ZDP detects drift in large language models without labels, proving key theorems and metrics.
problem Detecting drift in large language models without task labels or output evaluations.
method Zero-Direction Probing (ZDP) framework based on null directions of transformer activations, proving theoretical guarantees.
result Proves the Variance--Leak Theorem, Fisher Null-Conservation, Rank--Leak bound, and logarithmic-regret guarantee.
Study improves estimation of rare language model outputs.
problem Estimating probabilities of rare outputs in language models.
method Importance sampling vs. activation extrapolation for low probability estimation.
result Importance sampling outperforms activation extrapolation.
Bal-PM reduces preference labeling costs for LLMs.
problem Efficiently acquiring human feedback for preference modeling in large language models.
method Bayesian Active Learning with entropy maximization in feature space.
result Bal-PM reduces the number of required preference labels by 33% to 68%.
IMPACT optimizes LLM compression by focusing on activation importance, reducing model size up to 55.4%.
problem Resource constraints in deploying large language models (LLMs).
method IMPACT integrates activation importance into low-rank compression, optimizing for both size and accuracy.
result IMPACT achieves up to 55.4% greater model size reduction while maintaining comparable or better accuracy.
BatchTopK SAEs improve GPT-2 and Gemma activations with adjustable sparsity.
problem Interpreting language model activations using sparse autoencoders.
method Adapting the TopK constraint to the batch-level, allowing variable number of active latents per sample.
result BatchTopK SAEs consistently outperform TopK SAEs in reconstructing GPT-2 and Gemma activations.
Several recent papers investigate Active Learning (AL) for mitigating the data dependence of deep learning for natural language processing. However, the applicability of AL to real-world problems remains an open question. While in supervised learning, practitioners can try many different methods, evaluating each agains…
We stabilize the activations of Recurrent Neural Networks (RNNs) by penalizing the squared distance between successive hidden states' norms. This penalty term is an effective regularizer for RNNs including LSTMs and IRNNs, improving performance on character-level language modeling and phoneme recognition, and outperfor…
Enhanced network threat detection using KG, LLM, and imbalanced learning.
problem Challenges in network threat detection due to complex attack patterns and limited historical data.
method Integrated framework combining Knowledge Graph, Imbalanced Learning, and Large Language Model.
result Improved threat capture rate by 3%-4% and increased interpretability of risk predictions.
Painless Activation Steering automates post-training for LMs without manual intervention.
problem Manual post-training methods are time-consuming and labor-intensive.
method Painless Activation Steering (PAS) is a fully automated approach that requires no manual intervention.
result PAS reliably improves performance for behavior tasks but not for intelligence-oriented tasks.
Robots learn actions and language through curiosity-driven self-exploration.
problem Efficient development of actions and language in infants and robots.
method Curiosity-driven self-exploration using Q-learning to amortize active inference.
result Curiosity-driven exploration enables faster learning and compositional generalization.
FAQ efficiently evaluates LLMs with statistical guarantees using adaptive query selection.
problem Efficiently evaluating many LLMs on a large suite of benchmarks is expensive.
method FAQ uses Bayesian factor models, adaptive sampling, and proactive active inference to select queries.
result FAQ delivers up to 5x effective sample size gains over baselines, matching CI width with fewer queries.
Paper tackles active learning under human label variation, proposing a new framework.
problem Active learning assumes a single ground truth, ignoring human label variation.
method Survey and propose a conceptual framework for HLV-aware active learning.
result Lay a conceptual foundation for HLV-aware active learning.
Paper proposes active learning for mining conflict dynamics from textual data.
problem Lack of detailed information on conflict dynamics in existing datasets.
method Active learning with a large, encoder-only language model.
result Performance similar to human coding with reduced human annotation.
Dropout is commonly used to help reduce overfitting in deep neural networks. Sparsity is a potentially important property of neural networks, but is not explicitly controlled by Dropout-based regularization. In this work, we propose Sparseout a simple and efficient variant of Dropout that can be used to control the spa…
Recurrent neural networks trained on regular languages exhibit stable states that can recover from noise.
problem Stability of internal states in recurrent neural networks trained on regular languages.
method Empirical study with analysis of network activation and transitions between states.
result Recurrent neural networks trained on regular languages can recover from random perturbations and maintain stable states.
Plex improves model reliability across vision and language tasks.
problem Improving model reliability in diverse decision-making tasks involving uncertainty and adaptation.
method Developed ViT-Plex and T5-Plex pretrained model extensions to evaluate and improve reliability across 40 datasets.
result Plex greatly improves state-of-the-art across reliability tasks, simplifying evaluation and performance.
Unified SVD compression fails in practical tasks, highlighting the importance of per layer activation reconstruction.
problem The failure of a unified SVD compression method in practical tasks like perplexity and accuracy.
method Unified optimization problem for SVD based compression methods, focusing on cross-layer coupling.
result Downstream metrics like perplexity and accuracy degrade severely compared to standard per layer SVD LLM.
New method corrects quantization errors in LLMs using low-rank matrices.
problem Correcting quantization errors in large language models.
method Introducing low-rank weight matrices to correct quantized activations in LLMs.
result Reduces accuracy gap with original model by more than 50% using low-rank matrices.
Paper proposes efficient RLHF methods for LLMs using active queries.
problem Efficiently aligning LLMs with human preferences using RLHF.
method Formalizes RLHF as a dueling bandit problem, introduces APPO and ADPO algorithms.
result ADPO achieves similar performance to state-of-the-art methods with fewer queries.
The key to a Transformer model is the self-attention mechanism, which allows the model to analyze an entire sequence in a computationally efficient manner. Recent work has suggested the possibility that general attention mechanisms used by RNNs could be replaced by active-memory mechanisms. In this work, we evaluate wh…
Voice-controlled personal and home assistants (such as the Amazon Echo and Apple Siri) are becoming increasingly popular for a variety of applications. However, the benefits of these technologies are not readily accessible to Deaf or Hard-ofHearing (DHH) users. The objective of this study is to develop and evaluate a s…
Softmax is an output activation function for modeling categorical probability distributions in many applications of deep learning. However, a recent study revealed that softmax can be a bottleneck of representational capacity of neural networks in language modeling (the softmax bottleneck). In this paper, we propose an…
This paper addresses anisotropy in Transformer models, providing geometric insights and empirical support.
problem Anisotropy phenomenon in Transformer models, challenging their geometric interpretation.
method Derive geometric arguments and use concept-based mechanistic interpretability during training.
result Activation-derived directions capture large gradient energy and a larger share of gradient anisotropy than normal controls.
Active-GRPO improves molecular optimization by actively deciding when to imitate or self-improve.
problem Training robust and efficient molecular optimization models with large language models.
method Active-GRPO combines imitation and reinforcement learning, upgrading references and policies dynamically.
result Improves molecular optimization performance, achieving statistically significant gains.
We use GG distributions to optimize LLMs, reducing size and training time.
problem Lack of understanding in the statistical structure of LLMs.
method GG-based initialization, ACT, GCT.
result Smaller, faster models with minimal communication overhead.
Study uses simulation-based inference to decode brain activity from synthetic stimuli.
problem Reversing the process of brain activity emulation to recover stimuli or their properties.
method Pairing brain emulator with LLMs to learn a probabilistic mapping from brain maps to stimulus parameters.
result LLMs can serve as controllable stimulus generators and parameters can be recovered from brain maps.
We propose a simple extension to the ReLU-family of activation functions that allows them to shift the mean activation across a layer towards zero. Combined with proper weight initialization, this alleviates the need for normalization layers. We explore the training of deep vanilla recurrent neural networks (RNNs) with…
Hidden Markov Models detect hand gestures from wearable sEMG signals.
problem Detecting activity regions in continuous sEMG signals.
method Hidden Markov Models applied to sEMG signals for gesture recognition.
result Average accuracy of 96.25% for activity onsets and 87.5% for activity terminations.
ELS framework improves safety alignment by dynamically steering LLMs towards helpful responses.
problem Over-Refusal in Aligned Large Language Models
method Fine-tuning free framework using an Energy-Based Model (EBM) to dynamically steer LLMs during inference.
result Extensive experiments show a significant reduction in false refusals (from 57.3% to 82.6%) while maintaining safety performance.
GAPA method provides efficient uncertainty quantification for pretrained networks.
problem Reliable uncertainty estimates for pretrained models are challenging.
method Post-hoc Gaussian Process Activations (GAPA) method that shifts Bayesian modeling from weights to activations.
result GAPA method provides efficient uncertainty quantification without altering the backbone's predictions.
System learns to combine multiple model components for personalized text generation.
problem Adapting and biasing language models for personal preferences.
method Combines model-defined components, learns activation and probability combination from unlabeled text.
result Directly generates text with personalized components from unlabeled data.
We consider stochastic point processes generating time series exhibiting power laws of spectrum and distribution density (Phys. Rev. E 71, 051105 (2005)) and apply them for modeling the trading activity in the financial markets and for the frequencies of word occurrences in the language.
Neural networks trained with backpropagation often struggle to identify classes that have been observed a small number of times. In applications where most class labels are rare, such as language modelling, this can become a performance bottleneck. One potential remedy is to augment the network with a fast-learning non…
DIAL learns embeddings to maximize recall and accuracy for entity resolution.
problem Low resource settings for entity resolution with large Cartesian product search space.
method DIAL uses an Index-By-Committee framework with pre-trained transformer language models to jointly learn embeddings for recall and accuracy.
result DIAL achieves high precision, recall, and efficiency on benchmark datasets.
Large language models struggle with causal relationships, leading to biases and hallucinations.
problem LLMs struggle with true causal relationships, leading to biases and hallucinations.
method Embed causality into LLMs training process at every stage.
result LLMs need to be trained to understand and apply causal knowledge, not just recite it.
In this paper, we address customer review understanding problems by using supervised machine learning approaches, in order to achieve a fully automatic review aspects categorisation and sentiment analysis. In general, such supervised learning algorithms require domain-specific expert knowledge for generating high quali…
Concerns about interpretability, computational resources, and principled inductive priors have motivated efforts to engineer sparse neural models for NLP tasks. If sparsity is important for NLP, might well-trained neural models naturally become roughly sparse? Using the Taxi-Euclidean norm to measure sparsity, we find …
This paper presents a self-supervised method for visual detection of the active speaker in a multi-person spoken interaction scenario. Active speaker detection is a fundamental prerequisite for any artificial cognitive system attempting to acquire language in social settings. The proposed method is intended to compleme…
In de novo drug design, computational strategies are used to generate novel molecules with good affinity to the desired biological target. In this work, we show that recurrent neural networks can be trained as generative models for molecular structures, similar to statistical language models in natural language process…
The lack of adequate training data is one of the major hurdles in WiFi-based activity recognition systems. In this paper, we propose Wi-Fringe, which is a WiFi CSI-based device-free human gesture recognition system that recognizes named gestures, i.e., activities and gestures that have a semantically meaningful name in…
AlgebraNets use alternative algebras for neural networks, improving performance on image and language tasks.
problem Improving neural network performance on large-scale image and language tasks.
method Considered alternative algebras (C, H, M2(R), M2(C), M3(R), M4(R)) for activations and weights, and studied their performance on ImageNet and enwiki8 datasets.
result Alternative algebras deliver better parameter and computational efficiency compared with real numbers, especially in sparse and auto-regressive inference scenarios.