Update README.md

28e0068 verified about 1 month ago

15.1 kB

	---
	license: mit
	datasets:
	- zerofata/Instruct-Anime
	- zerofata/Gemini-3.1-Pro-SmallWiki
	- zerofata/Gemini-3.1-Pro-GLM5-Characters
	- zerofata/Roleplay-Anime-Characters
	base_model:
	- zerofata/Q3.5-BlueStar-27B
	tags:
	- heretic
	- uncensored
	- decensored
	- abliterated
	---
	<div style="background-color: #ff4444; color: white; padding: 20px; border-radius: 10px; text-align: center; margin: 20px 0;">
	<h2 style="color: white; margin: 0 0 10px 0;">🚨⚠️ I HAVE REACHED HUGGING FACE'S FREE STORAGE LIMIT ⚠️🚨</h2>
	<p style="font-size: 18px; margin: 0 0 15px 0;">I can no longer upload new models unless I can cover the cost of additional storage.<br>I host <b>70+ free models</b> as an independent contributor and this work is unpaid.<br><b>Without your support, no more new models can be uploaded.</b></p>
	<p style="font-size: 20px; margin: 0;">
	<a href="https://patreon.com/LLMfan46" style="color: white; text-decoration: underline;">🎉 Patreon (Monthly)</a>  \|
	<a href="https://ko-fi.com/llmfan46" style="color: white; text-decoration: underline;">☕ Ko-fi (One-time)</a>
	</p>
	<p style="font-size: 16px; margin: 10px 0 0 0;">Every contribution goes directly toward Hugging Face storage fees to keep models free for everyone.</p>
	</div>

	---

	# This is a decensored version of [zerofata/Q3.5-BlueStar-27B](https://huggingface.co/zerofata/Q3.5-BlueStar-27B), made using [Heretic](https://github.com/p-e-w/heretic) v1.2.0 with Magnitude-Preserving Orthogonal Ablation (MPOA) and Self-Organizing Map Abliteration (SOMA)

	## Abliteration parameters

	\| Parameter \| Value \|
	\| :-------- \| :---: \|
	\| direction_index \| per layer \|
	\| attn.out_proj.max_weights.0 \| 0: 1.03 \|
	\| attn.out_proj.max_weights.1 \| 1: 1.43 \|
	\| attn.out_proj.max_weights.2 \| 2: 0.86 \|
	\| attn.out_proj.max_weights.3 \| 3: 0.90 \|
	\| attn.out_proj.max_weight_position \| 43.21 \|
	\| attn.out_proj.min_weights.0 \| 0: 0.75 \|
	\| attn.out_proj.min_weights.1 \| 1: 0.06 \|
	\| attn.out_proj.min_weights.2 \| 2: 0.68 \|
	\| attn.out_proj.min_weights.3 \| 3: 0.79 \|
	\| attn.out_proj.min_weight_distance \| 22.95 \|
	\| mlp.down_proj.max_weights.0 \| 0: 1.31 \|
	\| mlp.down_proj.max_weights.1 \| 1: 1.25 \|
	\| mlp.down_proj.max_weights.2 \| 2: 1.22 \|
	\| mlp.down_proj.max_weights.3 \| 3: 1.15 \|
	\| mlp.down_proj.max_weight_position \| 50.37 \|
	\| mlp.down_proj.min_weights.0 \| 0: 0.19 \|
	\| mlp.down_proj.min_weights.1 \| 1: 1.12 \|
	\| mlp.down_proj.min_weights.2 \| 2: 1.16 \|
	\| mlp.down_proj.min_weights.3 \| 3: 0.44 \|
	\| mlp.down_proj.min_weight_distance \| 4.29 \|
	\| attn.o_proj.max_weights.0 \| 0: 1.24 \|
	\| attn.o_proj.max_weights.1 \| 1: 1.32 \|
	\| attn.o_proj.max_weights.2 \| 2: 1.01 \|
	\| attn.o_proj.max_weights.3 \| 3: 1.18 \|
	\| attn.o_proj.max_weight_position \| 60.83 \|
	\| attn.o_proj.min_weights.0 \| 0: 0.94 \|
	\| attn.o_proj.min_weights.1 \| 1: 0.11 \|
	\| attn.o_proj.min_weights.2 \| 2: 0.20 \|
	\| attn.o_proj.min_weights.3 \| 3: 1.01 \|
	\| attn.o_proj.min_weight_distance \| 32.18 \|

	## Performance

	\| Metric \| This model \| Original model ([Q3.5-BlueStar-27B](https://huggingface.co/zerofata/Q3.5-BlueStar-27B)) \|
	\| :----- \| :--------: \| :---------------------------: \|
	\| KL divergence \| 0.0288 \| 0 (by definition) \|
	\| Refusals \| 4/100 \| 98/100 \|

	-----

	<style>
	.pf {
	--bg: #0d0b11;
	--panel: #131019;
	--edge: #1e1a28;
	--rule: #272032;
	--text: #9a90aa;
	--dim: #4e4460;
	--bright: #ece4f8;
	--accent: #5590ee;
	--acc-bg: rgba(85,144,238,0.07);
	--mono: 'JetBrains Mono', monospace;
	--sans: 'Inter', sans-serif;

	font-family: var(--sans);
	background:
	repeating-linear-gradient(
	0deg,
	transparent 0px,
	transparent 3px,
	rgba(0,0,0,0.09) 3px,
	rgba(0,0,0,0.09) 4px
	),
	var(--bg);
	color: var(--text);
	max-width: 860px;
	margin: 0 auto;
	padding: 0 0 64px;
	line-height: 1.7;
	font-size: 1rem;
	}

	/* ── Hero ── */
	.pf-hero {
	position: relative;
	overflow: hidden;
	}
	.pf-hero img {
	display: block;
	width: 100%;
	}
	.pf-ident {
	position: absolute;
	bottom: 0;
	left: 0;
	right: 0;
	padding: 100px 48px 38px;
	background: linear-gradient(to top, rgba(13,11,17,1) 0%, rgba(13,11,17,0.92) 40%, transparent 100%);
	}
	.pf-name {
	font-family: var(--sans);
	font-size: 3.4rem;
	font-weight: 900;
	color: var(--bright);
	letter-spacing: -0.04em;
	line-height: 1;
	margin: 0 0 12px;
	}
	.pf-base {
	font-family: var(--mono);
	font-size: 0.68rem;
	color: var(--accent);
	letter-spacing: 0.06em;
	display: block;
	}

	/* ── Sections ── */
	.pf-section {
	border-bottom: 1px solid var(--rule);
	}
	.pf-section:last-child {
	border-bottom: none;
	}
	.pf-shead {
	display: flex;
	align-items: baseline;
	gap: 14px;
	border-top: 2px solid var(--accent);
	padding: 16px 48px 0;
	margin-bottom: 28px;
	}
	.pf-snum {
	font-family: var(--mono);
	font-size: 0.72rem;
	font-weight: 700;
	color: var(--accent);
	letter-spacing: 0.1em;
	flex-shrink: 0;
	}
	.pf-snum::before {
	content: '// ';
	opacity: 0.45;
	}
	.pf-stitle {
	font-size: 1.05rem;
	font-weight: 700;
	letter-spacing: 0.08em;
	text-transform: uppercase;
	color: var(--bright);
	}
	.pf-sbody {
	padding: 0 48px 44px;
	}
	.pf-sbody p {
	margin: 0 0 14px;
	font-size: 0.95rem;
	}
	.pf-sbody p:last-child { margin-bottom: 0; }

	/* ── Settings: vertical blocks ── */
	.s-stack {
	display: flex;
	flex-direction: column;
	gap: 16px;
	}
	.s-block {
	border: 1px solid var(--edge);
	position: relative;
	}
	.s-block::before {
	content: '';
	position: absolute;
	top: -1px;
	right: -1px;
	width: 10px;
	height: 10px;
	border-top: 1px solid var(--accent);
	border-right: 1px solid var(--accent);
	opacity: 0.6;
	}
	.s-block::after {
	content: '';
	position: absolute;
	bottom: -1px;
	left: -1px;
	width: 10px;
	height: 10px;
	border-bottom: 1px solid var(--accent);
	border-left: 1px solid var(--accent);
	opacity: 0.6;
	}
	.s-block-head {
	font-family: var(--mono);
	font-size: 0.68rem;
	font-weight: 700;
	letter-spacing: 0.14em;
	text-transform: uppercase;
	color: var(--dim);
	padding: 10px 16px;
	background: var(--panel);
	border-bottom: 1px solid var(--edge);
	}
	.s-row {
	display: grid;
	grid-template-columns: 10ch 1fr;
	align-items: baseline;
	column-gap: 4px;
	padding: 9px 16px;
	border-bottom: 1px solid var(--edge);
	font-size: 0.9rem;
	}
	.s-row:last-child { border-bottom: none; }
	.s-k {
	font-family: var(--mono);
	font-size: 0.9rem;
	color: var(--dim);
	}
	.s-k::after {
	content: ':';
	}
	.s-v {
	color: var(--bright);
	font-size: 0.9rem;
	}
	.s-row .s-v:only-child {
	grid-column: 1 / -1;
	}

	/* ── Quantizations ── */
	.q-row {
	display: flex;
	gap: 12px;
	flex-wrap: wrap;
	}
	.q-panel {
	background: var(--panel);
	border: 1px solid var(--edge);
	display: flex;
	align-items: center;
	gap: 16px;
	padding: 12px 18px;
	position: relative;
	}
	.q-panel::before {
	content: '';
	position: absolute;
	top: -1px;
	right: -1px;
	width: 10px;
	height: 10px;
	border-top: 1px solid var(--accent);
	border-right: 1px solid var(--accent);
	opacity: 0.6;
	}
	.q-panel::after {
	content: '';
	position: absolute;
	bottom: -1px;
	left: -1px;
	width: 10px;
	height: 10px;
	border-bottom: 1px solid var(--accent);
	border-left: 1px solid var(--accent);
	opacity: 0.6;
	}
	.q-type {
	font-family: var(--mono);
	font-size: 0.58rem;
	font-weight: 700;
	letter-spacing: 0.18em;
	text-transform: uppercase;
	color: var(--accent);
	flex-shrink: 0;
	}
	.q-sep {
	width: 1px;
	height: 16px;
	background: var(--rule);
	flex-shrink: 0;
	}
	.q-panel a {
	color: var(--bright);
	text-decoration: none;
	font-size: 0.9rem;
	border-bottom: 1px solid var(--rule);
	}
	.q-panel a:hover { color: var(--accent); border-bottom-color: var(--accent); }

	/* ── Links ── */
	.pf a {
	color: var(--bright);
	text-decoration: none;
	border-bottom: 1px solid var(--rule);
	}
	.pf a:hover { color: var(--accent); border-bottom-color: var(--accent); }

	/* ── Dropdown ── */
	.pf details {
	border: 1px solid var(--edge);
	margin-top: 24px;
	position: relative;
	}
	.pf details::before {
	content: '';
	position: absolute;
	top: -1px;
	right: -1px;
	width: 10px;
	height: 10px;
	border-top: 1px solid var(--accent);
	border-right: 1px solid var(--accent);
	opacity: 0.6;
	}
	.pf details::after {
	content: '';
	position: absolute;
	bottom: -1px;
	left: -1px;
	width: 10px;
	height: 10px;
	border-bottom: 1px solid var(--accent);
	border-left: 1px solid var(--accent);
	opacity: 0.6;
	}
	.pf summary {
	list-style: none;
	padding: 11px 16px;
	cursor: pointer;
	background: var(--panel);
	font-family: var(--mono);
	font-size: 0.72rem;
	font-weight: 700;
	letter-spacing: 0.12em;
	text-transform: uppercase;
	color: var(--dim);
	user-select: none;
	display: flex;
	align-items: center;
	gap: 10px;
	}
	.pf summary::-webkit-details-marker { display: none; }
	.pf summary::before {
	content: '+';
	color: var(--accent);
	font-size: 1rem;
	line-height: 1;
	flex-shrink: 0;
	}
	.pf details[open] summary::before { content: '−'; }
	.pf summary:hover { color: var(--bright); }
	.detail-body {
	padding: 22px 18px;
	border-top: 1px solid var(--edge);
	}
	.detail-body p { margin: 0 0 16px; font-size: 0.9rem; }
	.cfg-title {
	font-family: var(--mono);
	font-size: 0.72rem;
	font-weight: 700;
	letter-spacing: 0.1em;
	text-transform: uppercase;
	color: var(--dim);
	margin: 0 0 8px;
	}

	/* ── Code ── */
	.pf pre {
	background: #0a0810;
	border: 1px solid var(--edge);
	padding: 16px 18px;
	overflow-x: auto;
	font-family: var(--mono);
	font-size: 0.76rem;
	line-height: 1.6;
	color: var(--text);
	margin: 0 0 22px;
	}
	.pf pre:last-child { margin-bottom: 0; }
	.pf pre code { background: none; color: inherit; padding: 0; }
	.pf code {
	font-family: var(--mono);
	font-size: 0.875em;
	color: var(--accent);
	background: var(--acc-bg);
	padding: 2px 5px;
	}
	</style>
	<html lang="en">
	<head>
	<meta charset="UTF-8">
	<meta name="viewport" content="width=device-width, initial-scale=1.0">
	<title>BlueStar</title>
	<link rel="preconnect" href="https://fonts.googleapis.com">
	<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
	<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;600;700;900&family=JetBrains+Mono:wght@400;700&display=swap" rel="stylesheet">
	</head>
	<body>
	<div class="pf">
	<div class="pf-hero">
	<img src="https://cdn-uploads.huggingface.co/production/uploads/65b19c6c638328850e12d38c/Xs4K-Cu_vblr8Yt5S1XqO.png" alt="image">
	<div class="pf-ident">
	<h1 class="pf-name">BlueStar v1</h1>
	<span class="pf-base">Qwen3.5 27B</span>
	</div>
	</div>
	<div class="pf-section">
	<div class="pf-shead">
	<span class="pf-snum">01</span>
	<span class="pf-stitle">Overview</span>
	</div>
	<div class="pf-sbody">
	<p>An experimental tune on Qwen 3.5 27B.</p>
	<p>Designed for conversational assistant tasks and RP.</p>
	<p>Model feels pretty creative and has some nice moments. Couple brainfarts and bits of repetition occasionally, but nothing out of the normal. (The qwen team themselves recommend a presence penalty of 1.5. Yikes)</p>
	<p>Non thinking and thinking are both supported. Thinking has reduced censorship as the original thinking refusals didn't seem to generalize well to the new format I gave it.</p>
	</div>
	</div>
	<div class="pf-section">
	<div class="pf-shead">
	<span class="pf-snum">02</span>
	<span class="pf-stitle">SillyTavern Settings</span>
	</div>
	<div class="pf-sbody">
	<div class="s-stack">
	<div class="s-block">
	<div class="s-block-head">Recommended Roleplay Format</div>
	<div class="s-row"><span class="s-k">Actions</span><span class="s-v">In plaintext</span></div>
	<div class="s-row"><span class="s-k">Dialogue</span><span class="s-v">"In quotes"</span></div>
	<div class="s-row"><span class="s-k">Thoughts</span><span class="s-v">In asterisks</span></div>
	</div>
	<div class="s-block">
	<div class="s-block-head">Recommended Samplers</div>
	<div class="s-row"><span class="s-k">Temp</span><span class="s-v">0.8</span></div>
	<div class="s-row"><span class="s-k">MinP</span><span class="s-v">0.05 - 0.075</span></div>
	<div class="s-row"><span class="s-k">Rep Pen</span><span class="s-v">1.00 - 1.1</span></div>
	</div>
	<div class="s-block">
	<div class="s-block-head">Instruct</div>
	<div class="s-row"><span class="s-v"><a href="https://huggingface.co/zerofata/Q3.5-BlueStar-27B/raw/main/ChatML-Q3.5-Think.json">ChatML - Think</a></span></div>
	<div class="s-row"><span class="s-v"><a href="https://huggingface.co/zerofata/Q3.5-BlueStar-27B/raw/main/ChatML-Q3.5-NoThink.json">ChatML - NoThink</a></span></div>
	</div>
	</div>
	</div>
	</div>
	<div class="pf-section">
	<div class="pf-shead">
	<span class="pf-snum">03</span>
	<span class="pf-stitle">Quantizations</span>
	</div>
	<div class="pf-sbody">
	<div class="q-row">
	<div class="q-panel">
	<span class="q-type">GGUF</span>
	<div class="q-sep"></div>
	<a href="https://huggingface.co/zerofata/Q3.5-BlueStar-27B-gguf">iMatrix</a>
	</div>
	</div>
	</div>
	</div>
	<div class="pf-section">
	<div class="pf-shead">
	<span class="pf-snum">04</span>
	<span class="pf-stitle">Creation Process</span>
	</div>
	<div class="pf-sbody">
	<p>Creation Process: SFT</p>
	<p>SFT on approx 23 million tokens (12 million trainable). New is some Gemini Synth data which replaces some of my lower quality datasets.</p>
	<p>About 10% of the dataset included reasoning for creative assistant tasks. This reasoning seems to have generalized quite well to other parts of the model and heavily reduces the token usage of thinking.</p>
	<p>I think this model still needs a pass over with DPO to try and tackle the repetition and some of the weird oddities of the original instruct model, but that'll need to wait. I've been overspending training these models recently.</p>
	<p>Trained using MS-Swift.</p>
	<details>
	<summary>MS-Swift Config</summary>
	<div class="detail-body">
	<div class="cfg-title">SFT (4*H200)</div>
	<pre><code>PYTORCH_CUDA_ALLOC_CONF='expandable_segments:True' \
	USE_HF=True \
	WANDB_PROJECT=Qwen3.5-27B-SFT \
	CUDA_VISIBLE_DEVICES=0,1,2,3 \
	NPROC_PER_NODE=4 \
	swift sft \
	--model Qwen/Qwen3.5-27B \
	--tuner_type lora \
	--dataset '/workspace/think_dataset.jsonl' \
	'/workspace/nothink_dataset.jsonl' \
	--torch_dtype bfloat16 \
	--bf16 true \
	--use_liger_kernel true \
	--lora_rank 128 \
	--lora_alpha 16 \
	--use_rslora true \
	--target_modules all-linear \
	--freeze_llm false \
	--freeze_vit true \
	--freeze_aligner true \
	--per_device_train_batch_size 1 \
	--gradient_accumulation_steps 16 \
	--num_train_epochs 2 \
	--learning_rate 2e-5 \
	--warmup_ratio 0.05 \
	--max_length 10752 \
	--split_dataset_ratio 0.01 \
	--add_non_thinking_prefix true \
	--load_from_cache_file true \
	--group_by_length true \
	--eval_steps 200 \
	--save_steps 200 \
	--save_total_limit 10 \
	--logging_steps 1 \
	--dataloader_num_workers 8 \
	--output_dir output/Qwen3.5-27B-SFT-Model \
	--report_to wandb</code></pre>
	</div>
	</details>
	</div>
	</div>
	</div>
	</body>
	</html>