zestehl/supersymmetry-v1.1

SuSy Knowledge Base v1.1

A comprehensive, AI-rendered knowledge base of the Supersymmetry (SuSy) modpack for Minecraft 1.12.2 GregTech CEu. Contains 97,289 natural-language documents across 27 types — spanning materials, recipes, machines, items, fluids, ore deposits, process chains, multiblock structures, quest data, cross-modpack material mappings, Minecraft/Forge/GroovyScript internals, and pack architecture. Designed for RAG pipelines, semantic search, and LLM fine-tuning.

This is the v1.1 release — an expansion from the v1 baseline of 81,448 docs (10 types) to 97,289 docs (27 types) via Tracks A–E, P0.1, P0.2, P2.1, P3.1, and P3.2.

Dataset Structure

Each line in corpus.jsonl is a JSON object:

Field	Type	Description
id	string	Unique document identifier, format {type}:{key} (e.g., material:gregtech.material.copper)
type	string	Document type (see table below)
text	string	Natural-language prose paragraph rendered by DeepSeek V4
source_model	string	Model used for rendering (deepseek-v4-pro or deepseek-v4-flash)
tokens_in	integer	Prompt tokens consumed
tokens_out	integer	Completion tokens produced

Document Types

Type	Count	Model	Description
recipe	41,900	V4-Flash	Machine recipes: tier, EU cost, duration, inputs/outputs
item	26,984	V4-Flash	Item reference cards with recipe linkages, material binding
crafting_recipe	12,738	V4-Flash	Vanilla crafting table recipes (shaped/shapeless)
groovy_recipe	4,106	V4-Flash	GroovyScript-defined custom recipes
material	3,412	V4-Pro	Material deep-dives: composition, forms, state classification
fluid	2,426	V4-Flash	Fluid properties: temperature, density, material source
machine	1,644	V4-Flash	Machine reference cards: tier, recipemaps, controller status
quest_step	1,071	V4-Flash	BetterQuesting quest steps with material links
cross_modpack_material	737	V4-Flash	GregTech materials mapped across GTNH/Nomi/Omni/SuSy
smelting_recipe	635	V4-Flash	Vanilla furnace recipes
process_chain	500	V4-Flash	Multi-step processing chains for hub materials
wiki_concept	215	V4-Flash	Wiki-sourced concept documentation
ore_chain	156	V4-Flash	Ore → metal processing chains
recipemap	155	V4-Flash	Recipemap metadata and machine associations
multiblock_structure	135	V4-Flash	Multiblock structure patterns from Java source
ore_deposit	101	V4-Flash	Ore vein locations, dimensions, Y-ranges
forge_internal	52	V4-Flash	Forge 1.12.x event bus, capabilities, registries
material_pipe	48	V4-Flash	Fluid/item pipe properties per material
material_wire	46	V4-Flash	Wire electrical properties per material
lang_description	44	V4-Flash	Lang entry descriptions (multiblock/cover/enchantment)
groovyscript_internal	42	V4-Flash	GroovyScript DSL architecture, sandbox, lifecycle
minecraft_internal	34	V4-Flash	Minecraft 1.12.2 internals: tick, NBT, chunks, registries
quest_chain	31	V4-Flash	BetterQuesting quest line metadata
architecture	28	V4-Flash	Pack architecture: mod roles, processing domains, progression
material_tool	24	V4-Flash	Tool properties per material
progression_stage	13	V4-Flash	Voltage tier progression guides (ULV → UHV+)
tool_behavior	12	V4-Flash	Tool behavior descriptions from lang entries
	97,289

Statistics

• Total tokens: ~33.6M (23.5M prompt + 10.1M completion)
• Median doc length: 43 words
• Token distribution (word count, approximate): P10=31, P25=36, P50=43, P75=54, P90=64, P95=71, P99=107
• File size: 38.2 MB (JSONL)
• Source models: DeepSeek V4 Flash (93,877 docs), DeepSeek V4 Pro with thinking (3,412 material docs)

Generation Pipeline

Built from entity_registry.db (42 MB SQLite, 30 tables) — the definitive Supersymmetry data registry — plus BetterQuesting quest files, GroovyScript source, Forge 1.12.x documentation, Minecraft Wiki, and decompiled GTCEu/GCYM/SuSy Java source.

Pipeline:

1. Corpus generation — 25 generator functions in generate_corpus.py, each querying entity_registry.db or parsed source data
2. AI rendering — deepseek_inference.py with dynamic concurrency (4→256), zero 429s, ~$7.65 total inference cost
3. Embedding — hf_embedder.py via BGE-base-en-v1.5 (768-dim) through HuggingFace InferenceClient
4. Indexing — index_builder.py into ChromaDB persistent index (cosine distance)
5. RAG — rag_query_v3.py hybrid BM25 + BGE semantic retrieval → DeepSeek V4 Flash generation

All content is strictly Minecraft 1.12.2. No post-Flattening (1.13+) content is included.

Test Results

20 curated queries, 100% answer relevance, 80% top-result type match.

License

CC-BY-4.0 — free for research and commercial use with attribution.

Known Gaps

1. No vanilla Minecraft recipes — Corpus is GregTech/SuSy-focused. Vanilla crafting/furnace recipes absent.
2. Machine descriptions are formulaic — Machine docs are DB dumps, not functional guides.
3. Late-game sparsity — ZPM+ tiers have proportionally fewer docs (structural, not a bug).
4. No cross-encoder reranker — v3 drops v2's reranker for simplicity. Retrieval quality remains excellent.
5. Fluid material association — Some fluids lack material_resource linkage.

Usage

import json

with open("corpus.jsonl") as f:
    for line in f:
        doc = json.loads(line)
        print(f"[{doc['type']}] {doc['id']}: {doc['text'][:200]}...")
        break

Source

• Database: entity_registry.db (not included — SuSy-specific extraction)
• Generator: generate_corpus.py
• Inference: deepseek_inference.py (DeepSeek API)
• RAG: rag_query_v3.py + ChromaDB + HuggingFace BGE-base