Feature Extraction
Transformers
English
torch
clip
vision
interpretability
sparse autoencoder
sae
mechanistic interpretability
# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("Prisma-Multimodal/sparse-autoencoder-clip-b-32-sae-vanilla-x64-layer-5-hook_resid_post-l1-8e-05", dtype="auto")Quick Links
CLIP-B-32 Sparse Autoencoder x64 vanilla - L1:8e-05
Training Details
- Base Model: CLIP-ViT-B-32 (LAION DataComp.XL-s13B-b90K)
- Layer: 5
- Component: hook_resid_post
Model Architecture
- Input Dimension: 768
- SAE Dimension: 49,152
- Expansion Factor: x64 (vanilla architecture)
- Activation Function: ReLU
- Initialization: encoder_transpose_decoder
- Context Size: 50 tokens
Performance Metrics
- L1 Coefficient: 8e-05
- L0 Sparsity: 377.3570
- Explained Variance: 0.8302 (83.02%)
Training Configuration
- Learning Rate: 0.0004
- LR Scheduler: Cosine Annealing with Warmup (200 steps)
- Epochs: 10
- Gradient Clipping: 1.0
- Device: NVIDIA Quadro RTX 8000
Experiment Tracking:
- Weights & Biases Run ID: 1werbe7n
- Full experiment details: https://wandb.ai/perceptual-alignment/clip/runs/1werbe7n/overview
- Git Commit: e22dd02726b74a054a779a4805b96059d83244aa
Citation
@misc{2024josephsparseautoencoders,
title={Sparse Autoencoders for CLIP-ViT-B-32},
author={Joseph, Sonia},
year={2024},
publisher={Prisma-Multimodal},
url={https://huggingface.co/Prisma-Multimodal},
note={Layer 5, hook_resid_post, Run ID: 1werbe7n}
}
- Downloads last month
- 5
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("feature-extraction", model="Prisma-Multimodal/sparse-autoencoder-clip-b-32-sae-vanilla-x64-layer-5-hook_resid_post-l1-8e-05")