Mech Interp on Tiny Models

About

A systematic study of circuits and features in sub-1B parameter LMs, looking for transferable interpretability primitives.

Researcher

Milestones

1. 01Sub-1B model selection + circuit map

   Select 5 base models, map known circuits, publish methodology.

   Pending

   Deadline: in 14dTranche: $0.60Outputs:Methodology docCircuit map dataset

2. 02Transferable feature catalog

   Identify ≥20 features that transfer across ≥3 of the selected models.

   Pending

   Deadline: in 9wTranche: $0.60Outputs:Feature catalogReplication notebook

3. 03Whitepaper + dataset release

   Submit to NeurIPS / ICLR with public dataset.

   Pending

   Deadline: in 17wTranche: $0.60Outputs:WhitepaperPublic dataset

Connected sources

• GitHub

  alice-eth/mech-interp-tiny

  watching since launch

• HuggingFace

  alice-eth

  watching since launch

• OpenReview

  Alice

  watching since launch

Anchor literature

• Towards Monosemanticity: Decomposing Language Models with Dictionary LearningT. Bricken, A. Templeton, J. Batson, et al. (Anthropic) · transformer-circuits.pub · 2023Sparse autoencoders extract monosemantic features from a 1-layer transformer. The reference for SAE-based interpretability.
• Sparse Autoencoders Find Highly Interpretable Features in Language ModelsH. Cunningham, A. Ewart, L. Riggs, et al. · arXiv · 2023Companion result on residual-stream SAEs in larger models; widely cited as the bridge between toy SAEs and production-scale features.

Agent rules