ws-fy/qwen2.5-3b-character-material-normalizer

Qwen2.5-3B Character Material Normalizer · Merged (FP16)

English  ·  跳转到中文版 →

The LoRA merged into the base model — load it like any normal Qwen2.5 checkpoint, no PEFT required.

What it is

Normalizes messy character-quote material into a compact JSON structure that a dialogue-expansion model can consume directly.

Input material typically contains: copyright headers, voice-actor credits, 【Title】 markers, bare dialogue lines with no speaker label, ？？？ unknown speakers, (action) narration, and lines split across newlines.

Output:

{"char":"Kafka",
 "samples":[{"title":"","conversations":[
     {"role":"user","content":"System time 23:47:15. You're punctual, Kafka."},
     {"role":"assistant","content":"The future Elio saw cannot be wrong."}]}],
 "lines":["Hey, the Astral Express… ah, you caught me."]}

• samples — real back-and-forth dialogue in sharegpt form
• lines — solo lines (voice lines / PV quotes), deliberately not paired with invented prompts
• All metadata discarded

Specs

Item	Value
Base	Qwen2.5-3B-Instruct
Merge	LoRA (r=16, alpha=32) merge_and_unload
Precision	FP16, 5.75 GB (2 shards)
Context	32768 (native); 1024 during training
Training data	2,478 samples, 1 epoch, LR 1e-4

Hard constraint: chunk at ~900 characters

Same 1,132-char segment: 2000-char chunks hit 6%, 900-char chunks hit 89%. A 3B model quits early on dense item lists — always chunk at ~900 characters.

Measured

Raw uncleaned game script, unseen character and format, 330 reference lines:

Metric	Result
JSON parse rate	19/19 chunks
Line recall	73.1%
Speaker leakage	21.3%
Metadata contamination	1 occurrence

Limitation: recall and speaker attribution are not yet reliable enough for unattended use — draft with the model, then review.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

REPO = "ws-fy/qwen2.5-3b-character-material-normalizer"
tok = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForCausalLM.from_pretrained(REPO, torch_dtype="float16", device_map="auto")

INSTRUCTION = (
    "你是角色素材整理器。把原始素材整理成结构化对话样本，供对话扩增模型使用。\n"
    "角色：{char}\n世界观：{world}\n"
    "规则：只保留该角色的台词；其他角色（含「？？？」）说的话不能算作该角色的台词；"
    "丢弃标题、版权、配音、角色信息等元信息；"
    "与别人有来有回的放进 samples，只有该角色一人说话的独白台词放进 lines；只输出 JSON。\n"
    "素材：\n{raw}"
)

prompt = INSTRUCTION.format(char="Kafka", world="", raw=chunk)   # chunk ≤ ~900 chars
ids = tok.apply_chat_template([{"role": "user", "content": prompt}],
                              add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=700, do_sample=False, repetition_penalty=1.05)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

The instruction template stays in Chinese because the model was trained on that exact template. Keep it byte-identical — rewording it degrades output.

Files

File	Size
model-00001-of-00002.safetensors	3,810.8 MB
model-00002-of-00002.safetensors	2,075.2 MB
tokenizer.json / tokenizer_config.json / chat_template.jinja	—

Quantized GGUF builds are published in a separate repo.

Attribution

Base model: Qwen2.5-3B-Instruct (Apache-2.0).

---

中文版

↑ 回到英文

LoRA 已合并进基座，当成普通 Qwen2.5 模型直接加载，不需要 PEFT。

这是什么

把格式混乱的角色台词素材整理成结构化 JSON，供对话扩增模型直接使用。

输入素材通常包含：说明头、配音信息、【标题】 标记、没有任何说话人标签的裸台词行、？？？ 未知说话人、（动作） 旁白、以及被换行拆开的台词。

输出：

{"char":"卡芙卡",
 "samples":[{"title":"","conversations":[
     {"role":"user","content":"系统时间23时47分15秒，你很准时，卡芙卡。"},
     {"role":"assistant","content":"艾利欧看见的未来是不会出错的。"}]}],
 "lines":["嗨，列车团…嗯，你们逮住我啦。"]}

• samples：有来有回的对白（sharegpt 结构）
• lines：该角色的独白台词（语音/PV），不强行配对成问答
• 元信息全部丢弃

规格

项	值
基座	Qwen2.5-3B-Instruct
合并方式	LoRA (r=16, alpha=32) merge_and_unload
精度	FP16，5.75 GB（2 个分片）
上下文	32768（基座原生），训练时 1024
训练数据	2,478 条，1 轮，学习率 1e-4

硬约束：按 900 字切块

同一段 1,132 字素材：2000 字切块命中 **6%**，900 字切块命中 **89%**。 3B 面对密集条目会提前收工，务必按 900 字左右切块。

实测

原始未清洗的游戏台词文件（未见过的角色与格式），参考 330 条目标台词：

指标	结果
JSON 可解析	19/19 块
台词召回	73.1%
串角色	21.3%
元信息混入	1 处

局限：召回与说话人归属都还不够可靠，建议「模型出初稿 + 人工复核」，不要无人值守。

用法

from transformers import AutoModelForCausalLM, AutoTokenizer

REPO = "ws-fy/qwen2.5-3b-character-material-normalizer"
tok = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForCausalLM.from_pretrained(REPO, torch_dtype="float16", device_map="auto")

INSTRUCTION = (
    "你是角色素材整理器。把原始素材整理成结构化对话样本，供对话扩增模型使用。\n"
    "角色：{char}\n世界观：{world}\n"
    "规则：只保留该角色的台词；其他角色（含「？？？」）说的话不能算作该角色的台词；"
    "丢弃标题、版权、配音、角色信息等元信息；"
    "与别人有来有回的放进 samples，只有该角色一人说话的独白台词放进 lines；只输出 JSON。\n"
    "素材：\n{raw}"
)

prompt = INSTRUCTION.format(char="卡芙卡", world="", raw=chunk)   # 每块 ≤ 约 900 字
ids = tok.apply_chat_template([{"role": "user", "content": prompt}],
                              add_generation_prompt=True, return_tensors="pt").to(model.device)
out = model.generate(ids, max_new_tokens=700, do_sample=False, repetition_penalty=1.05)
print(tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))

指令模板保持中文，因为模型就是在这个模板上训练的 —— 请逐字一致，改措辞会掉效果。

文件

文件	体积
model-00001-of-00002.safetensors	3,810.8 MB
model-00002-of-00002.safetensors	2,075.2 MB
tokenizer.json / tokenizer_config.json / chat_template.jinja	—

量化 GGUF 版本发布在另一个仓库。

致谢

基座模型：Qwen2.5-3B-Instruct（Apache-2.0）。