---
license: mit
language:
- en
tags:
- mechanistic-interpretability
- sparse-autoencoder
- cross-layer-transcoder
- qwen2.5-vl
- vision-language-model
- circuit-tracer
library_name: safetensors
pipeline_tag: feature-extraction
---

# Qwen2.5-VL-7B CLTs (circuit-tracer format)

This is the circuit-tracer compatible version of Cross-Layer Transcoders (CLTs) trained on Qwen2.5-VL-7B-Instruct.

## Usage with circuit-tracer

```python
from circuit_tracer import ReplacementModel

model = ReplacementModel.from_pretrained(
    model_name="Qwen/Qwen2.5-VL-7B-Instruct",
    transcoder_set="KokosDev/qwen2p5vl-7b-clt",
)
```

Or use the convenience shortcut:

```python
from circuit_tracer.vlm import VLModelWrapper

model = VLModelWrapper.from_pretrained(
    'Qwen/Qwen2.5-VL-7B-Instruct',
    transcoder_set='qwen',  # Shortcut for this repo
    dtype=torch.bfloat16
)
```

## Model Details

- **Architecture**: Cross-Layer Transcoders (CLTs)
- **Base Model**: Qwen/Qwen2.5-VL-7B-Instruct
- **Hidden Dimension**: 3584
- **Feature Dimension**: 8192
- **Layers**: 27 (layers 0-26)
- **Sparsity**: ~12% L0
- **Training Steps**: 5000

## Format

- `layer_*.safetensors`: Transcoder weights for each layer
- `config.yaml`: Configuration for circuit-tracer
- Uses safetensors format for fast loading

## Training Details

- **Optimizer**: AdamW
- **Learning Rate**: 3e-4
- **Scheduler**: Cosine
- **Target L0**: 0.12
- **Validation Loss**: 10.3 - 19.1

## Citation

If you use these transcoders in your research, please cite:

```bibtex
@misc{qwen2p5vl7b-clt,
  author = {KokosDev},
  title = {Qwen2.5-VL-7B Cross-Layer Transcoders},
  year = {2025},
  publisher = {HuggingFace},
  url = {https://huggingface.co/KokosDev/qwen2p5vl-7b-clt}
}
```