README.md

metadata

license: apache-2.0
library_name: transformers
pipeline_tag: text-generation
base_model: mistralai/Mistral-7B-v0.1
tags:
  - dystrio
  - sculpt
  - dense
  - runtime-agnostic
  - no-custom-kernels
  - hf-drop-in
model-index:
  - name: Dystrio Sculpt (Mistral-7B Conservative)
    results:
      - task:
          type: text-generation
        dataset:
          name: WikiText-103 (validation)
          type: wikitext
        metrics:
          - name: perplexity
            type: perplexity
            value: 11.0557

What is Dystrio Sculpt?

Dystrio Sculpt produces dense compiled variants of existing models that:

• reduce memory footprint
• improve prefill throughput
• remain runtime-agnostic
• require no custom kernels
• load with standard HuggingFace Transformers

Key Results

Compared to mistralai/Mistral-7B-v0.1 baseline on an A100 80GB:

• Weights memory: -11% (Conservative) / -23% (Balanced)
• RAG latency (TTFT p95): -7% / -14%
• Decode throughput: ~flat
• No runtime changes: no custom kernels, no new ops, standard transformers loading

Notes: TTFT includes prefill + first decode step. “Weights memory” is computed from parameter sizes (GiB) and is workload-independent.

Benchmark Results

Model	PPL	PPL Ratio	RAG TTFT p95 (ms)	Chat Decode p95 (ms/tok)	Prefill TPS	Decode TPS	Weights (GiB)	Post-load (GiB)	End-of-bench (GiB)	Peak (GiB)
mistral-7b-v0.1 (baseline)	11.0557	1.0	158.357	33.096	7661.1	30.9	13.488777	13.488778	13.5	14.15
sculpt-conservative	12.4484	1.126	147.31	34.169	8296.3	30.2	11.988777	11.996713	12.0	12.63
sculpt-balanced	19.5153	1.7652	135.959	33.302	9175.1	30.7	10.395027	10.402963	10.4	11.02

Benchmark Environment

• GPU: NVIDIA A100-SXM4-80GB
• dtype: bf16
• Torch: 2.10.0+cu128
• Transformers: 5.2.0
• Deterministic: False
• Seed: 0
• Single-GPU, Hugging Face Transformers, no custom kernels.

Metric Definitions

• TTFT incl. prefill: Wall time from prompt submission to first generated token (prefill forward + first decode step). Per-prompt request-level measurement.
• First decode step: Wall time of the first decode forward call only (post-prefill). Per-prompt request-level measurement.
• Prefill/Decode TPS: Throughput from batched microbenchmark iterations (not request-level; used for throughput comparison only).
• Weights (GiB): Model parameter memory only (sum of numel * element_size for all parameters). Deterministic and runtime-independent.
• Post-load (GiB): torch.cuda.memory_allocated() immediately after model.eval() + torch.cuda.empty_cache(). Captures weights + framework overhead before any inference.
• End-of-bench (GiB): torch.cuda.memory_allocated() at end of benchmark workload. Includes KV-cache and activations still held.
• Peak (GiB): torch.cuda.max_memory_allocated() during benchmark. High-water mark for planning GPU headroom.