Running Bosun locally (GGUF)

These are GGUF builds of Bosun-4B (4B) for CPU / Apple-Silicon / edge inference with llama.cpp. The full model and the LoRA source live at Hanno-Labs/bosun-4b.

⚠️ Do NOT use llama.cpp's `--rerank` mode

Bosun is a programmable judge: the instruction is the rubric, supplied per request. The GGUF carries the generative Qwen3 chat template, so llama.cpp's --rerank endpoint silently discards your <Instruct> and returns degenerate scores (~1e-12; opposite rules score identically). It looks like it works — it does not. (Thanks to Frederick Wood for the careful report.)

Use the completion + logits path instead.

The contract

Build the prompt from serving.json and read two logits at the final token:

prompt = prefix
       + "<Instruct>: <your rule>\n<Query>: <query>\n<Document>: <document>"
       + suffix          # suffix already contains the empty "<think>\n\n</think>" block

score  = sigmoid( logit[yes_id] - logit[no_id] )   # at the last position

For Bosun-4B: yes_id = 9693, no_id = 2152, max_len = 3072.

Python (llama-cpp-python)

import json, math
from llama_cpp import Llama

cfg = json.load(open("serving.json"))
llm = Llama("Bosun-4B-Q8_0.gguf", n_ctx=cfg["max_len"], logits_all=True, verbose=False)

def score(instruct, query, document):
    body = f"<Instruct>: {instruct}\n<Query>: {query}\n<Document>: {document}"
    prompt = cfg["prefix"] + body + cfg["suffix"]
    toks = llm.tokenize(prompt.encode(), add_bos=False, special=True)
    llm.reset(); llm.eval(toks)
    lg = llm.scores[len(toks) - 1]
    return 1.0 / (1.0 + math.exp(-(lg[cfg["yes_id"]] - lg[cfg["no_id"]])))

# the document is an ORDERED pair — FINDING A then FINDING B (direction matters)
doc = "FINDING A:\nMercury set up its own bank charter.\n\nFINDING B:\nKlar bought a small bank."
print(score("Connected only if both findings are about the same broad topic.",
            "These two findings share the specified relationship.", doc))

Files & fidelity

Validated per-pair against the published transformers inference (logits_to_keep=1) on a fixture spanning the default rubric, instruction steering, and dedup. Mean / max absolute score difference vs that reference:

file	size	mean abs diff	max abs diff
`Bosun-4B-f16.gguf`	8050 MB	0.0004	0.001
`Bosun-4B-Q8_0.gguf`	4280 MB	0.0011	0.0035
`Bosun-4B-Q4_K_M.gguf`	2497 MB	0.0036	0.0097

Pick by use case: f16 is reference-grade; Q8_0 is recommended (calibrated scores intact at ~half the size). Q4_K_M is smallest; on this model it stays calibration-safe too.

All builds preserve steering (scores flip when the rule flips) and ranking order.