Lumimaid-Nexus-2x12B-MoE

This model is a Mixture of Experts (MoE) created by combining NeverSleep/Lumimaid-v0.2-12B and ReadyArt/Dark-Nexus-12B-v2.0 using mergekit-moe. It utilizes the self-attention and layer normalization layers from the base donor model while routing multi-layer perceptron (MLP) passes across two distinct expert models.

Model Details

• Base Model: NeverSleep/Lumimaid-v0.2-12B
• Architecture: Mistral-Nemo (12B base parameter size per expert, 2 routed experts)
• Gating Mode: Hidden-state representation routing (hidden)
• Active Experts per Token: 2
• Data Type: bfloat16

Expert Models

1. Expert 1: NeverSleep/Lumimaid-v0.2-12B
   • Focus: Narrative consistency, multi-sensory immersion, and character-driven dialogue.
2. Expert 2: ReadyArt/Dark-Nexus-12B-v2.0
   • Focus: Dynamic scenario processing and complex character interaction dynamics.

Usage

Prompt Format

This model uses the ChatML/Tekken prompt formatting structure:

<|im_start|>system
You are a helpful narrative assistant.<|im_end|>
<|im_start|>user
[Your prompt here]<|im_end|>
<|im_start|>assistant