Sparse Autoencoders & Dictionary Learning

If superposition packs n features into m < n dimensions, the natural decoder is an overcomplete dictionary: an autoencoder that expands activations into a much larger space under a sparsity penalty, then reconstructs.

A sparse autoencoder (SAE) has:

• Encoder — maps the m-dimensional activation to a k-dimensional code, k >> m, sparse (few nonzero)
• Sparsity penalty — forces the code to activate few features per input, so each latent must be selective
• Decoder — reconstructs the original activation; reconstruction error is the fidelity budget

The bet: with sparsity as the bottleneck, the learned latents become monosemantic — one clean concept per direction. Anthropic's 'Towards Monosemanticity' showed it works; 'Scaling Monosemanticity' showed it works on frontier models.