Interpretability · NeurIPS 2024 · SoLaR Workshop

SCAR

SCAR: Sparse Conditioned Autoencoders for Concept Detection and Steering in LLMs

Detecting concepts and steering generation while keeping the language model’s weights fixed.

Read the paper Code Model weights

Authors

R. Härle, F. Friedrich, M. Brack, B. Deiseroth, P. Schramowski, K. Kersting

THE IDEA, AT A GLANCE

CONCEPTUAL ILLUSTRATION

Freeze backbone weights

Train the added module with the backbone weights frozen.

Condition on a concept

Use supervision to learn a concept-specific handle.

Scale & reconstruct

Return the adjusted reconstruction to the model.

Keep the LLM weights fixed, learn a conditioned feature, then feed its adjusted reconstruction back into the model.

Motivation

Language models can generate fluent text that does not match the behavior a user wants. We need ways to detect and adjust concepts such as toxicity or writing style without retraining the whole model.

Method

SCAR trains a sparse autoencoder on a transformer block’s feed-forward activations while keeping the LLM’s weights frozen. Labels condition a designated latent feature on the target concept. At inference, its decoded reconstruction replaces the original feed-forward signal; scaling the feature steers generation.

Results

Conditioned features detected toxicity, safety-related content, and Shakespearean style more clearly than an unconditioned SAE. In the toxicity-steering experiments, changing the feature changed output toxicity while the evaluated language-model benchmark scores remained largely unchanged.