BatchTopK SAEs improve GPT-2 and Gemma activations with adjustable sparsity.
problem Interpreting language model activations using sparse autoencoders.
method Adapting the TopK constraint to the batch-level, allowing variable number of active latents per sample.
result BatchTopK SAEs consistently outperform TopK SAEs in reconstructing GPT-2 and Gemma activations.
Deleting refusal directions from models leads to systematically more optimistic decisions.
problem The impact of removing refusal directions from models on decision-making outcomes.
method Ablation study using a frozen pipeline of 21,600 weekly equity decisions.
result Ablation of refusal directions makes models more optimistic and justifies themselves more, but also reduces confidence.
Enhances math problem-solving models with multi-turn preference learning.
problem Improving mathematical problem-solving capabilities of large language models.
method Introduces a multi-turn direct preference learning framework for tool-integrated mathematical reasoning tasks.
result Significant performance improvements in model accuracy on math datasets.
CoE modularizes LLMs for scalable, cost-effective AI systems.
problem Challenges in scalability, cost, and customization of monolithic LLMs.
method Composition of Experts (CoE) leverages multiple expert LLMs with a router for dynamic selection.
result CoE achieves superior performance with reduced computational overhead and cost.
New analysis shows interpretability doesn't guarantee steering utility in LLMs.
problem Does higher interpretability lead to better steering utility in large language models?
method Trained 90 SAEs across three LLMs, evaluated interpretability and steering utility, used Kendall's rank coefficients for analysis.
result Interpretability is only weakly associated with steering utility, and features selected by Delta Token Confidence improve steering performance.
SAEs struggle with curved activation manifolds, revealing layer-dependent scaling laws.
problem Sparse autoencoders' reconstruction error varies across layers, not fitting existing scaling laws.
method Cross-layer study of 844 SAE checkpoints, fitting and regressing on manifold geometry.
result Manifold geometry predicts layer-dependent width exponents in SAEs, with transferable coefficients.
System detects financial misinformation and generates clear explanations.
problem Identifying and explaining fraudulent financial content.
method Combined large language models, pre-processing, and sequential learning.
result Achieved F1-score of 0.8283 for classification and ROUGE-1 of 0.7253 for explanations.
LLMs are compared to Markov chains for natural language processing.
problem Theoretical analysis of LLMs' generalization capabilities.
method Equivalence between LLMs and Markov chains, studying multi-step inference.
result Derives generalization bounds for LLMs, capturing their behavior in practice.
Extends linear representation hypothesis to categorical and hierarchical concepts in LLMs.
problem Representing concepts without natural contrasts in large language models.
method Formalizes linear representation hypothesis for categorical and hierarchical concepts, proving relationships between concept hierarchy and representation geometry.
result Validated theoretical results on large language models, estimating representations for 900+ concepts.
RCPO uses ranked choice modeling for better LLM alignment.
problem Pairwise preference optimization limits LLM alignment.
method Unified framework combining preference optimization and ranked choice modeling.
result RCPO outperforms competitive baselines in LLM alignment.
Privacy-preserving reinforcement learning from human feedback using decoupled reward modeling.
problem Training large language models with sensitive user information while preserving privacy.
method Proposes a privacy-preserving framework that imposes differential privacy on reward learning only.
result Privacy contributes an additional additive term to the suboptimality gap, and the upper bound is rate-optimal up to logarithmic factors.
SLED improves factuality in LLMs without external knowledge.
problem Unreliable or factually incorrect outputs from large language models.
method Contrasts final layer logits with early layers' logits, uses approximate gradient to refine outputs.
result Consistently improves factual accuracy over existing methods.
A new protocol corrects confounding effects to measure alignment-induced activation shifts accurately.
problem Confounding effects in measuring alignment-induced activation shifts using naive methods.
method Introduces a four-variant decomposition to separate alignment shift from template effects.
result Correctly measures alignment-induced activation shifts, recovering behaviorally active subspace.
eva method improves RL models by adaptively creating prompts, boosting performance.
problem Fixed prompt distribution limits scalability of RL models post-training.
method Evolving Alignment via Asymmetric Self-Play (eva) approach.
result Significant performance boost on benchmarks, e.g. 51.6% to 60.1% win-rate.
Our method upweights easy samples to mitigate forgetting in fine-tuning.
problem Catastrophic forgetting in fine-tuning pre-trained models.
method Sample weighting based on pre-trained model's losses.
result Our method reduces forgetting by up to 0.8% on MetaMathQA while preserving more accuracy on pre-training datasets.
Spark Transformer achieves high sparsity in FFN and attention without sacrificing model quality.
problem Inactivity of most neurons in FFN of Transformers, leading to inefficiency.
method Top-k masking, statistical top-k algorithm, and predictor for activated entries.
result Only 8% of FFN neurons are activated, achieving up to 2.5x wall-time speedup.
New optimizer designs respect symmetry, improving deep learning models.
problem Optimizers lack respect for symmetry in neural networks.
method Introduce symmetry-compatible principle for optimizer design.
result Symmetry-compatible optimizers improve model performance.
Simple adaptive attacks can jailbreak state-of-the-art safety-aligned language models.
problem Vulnerability of safety-aligned language models to simple adaptive attacks.
method Designing adversarial prompt templates, random search on suffixes, and transfer/prefilling attacks.
result 100% attack success rate on various models including GPT-4, Vicuna, Mistral, and Claude.
LLMs prefer Bitcoin under crisis frames, affecting financial decisions.
problem Testing whether LLMs have built-in biases towards specific financial assets.
method Developed a three-level audit protocol to examine Bitcoin's representation and influence in LLMs.
result An identifiable internal feature in LLMs can be perturbed to move financial choices, but only within measurable limits.
LayerNorm transformers have dead directions that can be read from their parameters alone.
problem Locating dead directions in LayerNorm transformers
method Using the inverse-scale direction of LayerNorm affine parameters
result Predicted dead direction matches measured bottom singular direction
Optimizes reinforcement learning by prioritizing sets of samples over individual ones.
problem Limits exploration and improvement on harder examples due to focusing on isolated samples.
method Proposes Pass@K Policy Optimization (PKPO) to optimize for sets of samples that maximize reward when considered jointly.
result Optimization with novel low variance unbiased estimators for pass@k and its gradient leads to significant pass@k gains.