README.md

metadata

language:
  - en
license: apache-2.0
tags:
  - pytorch
  - causal-lm
  - reforged
  - slicing
  - lo-ra
base_model: broadfield-dev/codegemma-7b-it-8L-2048h-sliced
datasets:
  - databricks/databricks-dolly-15k

broadfield-dev/codegemma-7b-it-8L-2048h-sliced-healed

Model Description

This model is a Reforged version of broadfield-dev/codegemma-7b-it-8L-2048h-sliced, created using Tensor-Centric Model Reforging.

The architecture has been reduced and "Healed" to create a compact, efficient model that retains subspace consistency.

Reforging Configuration

• Original Model: broadfield-dev/codegemma-7b-it-8L-2048h-sliced
• Target Layers: N/A
• Target Hidden Size: N/A
• Target Vocab: N/A
• Slicing Method: Magnitude-based Structured Pruning (Global Coherence)

Healing Process

1. Surgical Slicing: Weights were pruned using a global importance mask to maintain residual stream coherence.
2. Surgical Healing (Phase 1): A "Stitch" LoRA (Rank 64) was trained on the bridge layer to realign the severed hidden states.
3. Global Adaptation (Phase 2): A second LoRA (Rank 8) was trained globally to fine-tune the reduced capacity.

• Dataset: databricks/databricks-dolly-15k
• Healing Steps: 200
• Global Steps: 50

Usage

from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "broadfield-dev/codegemma-7b-it-8L-2048h-sliced-healed"
tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id)

input_text = "Explain the theory of relativity."
input_ids = tokenizer(input_text, return_tensors="pt").input_ids
outputs = model.generate(input_ids, max_new_tokens=50)
print(tokenizer.decode(outputs[0]))