Degradation after 10k of context?
@user59365 on the community section for Magidonia-24B says that it suffers writing degradation around the 10k mark of context. Does anyone know if this is also the case for Cydonia?
No it's not the case. Technically it's an universal truth that models are better in short context, but Cydonia behaves really decently at even 24K context (the context I tested it against and which my VRAM allows anyway 😀). I mean, it always depends on what you feed to the model, if you only write one liners, sure it'll degrade quicker because it has nothing to use, but if you actually contribute meaningfully to the chat, it's actually one of the better models when it comes to long form content.
Magidonia and all CoT models are not that good with long context due to the way they are trained, the "thinking" bit is great for first few chat message pairs, but after that, it usually tends to be detrimental.
I can also confirm that when i do temporarily swap to the Cydonia v4.3 model on my >1,200+ message chat at 32k context, it handles it just fine. I normally use a 70b and while i'm not sure if i used the cydonia v4.3 for a while if it would degrade, but here and there on my existing RP it's great.
@veldierin , How what settings are you using? i have been trying to get longer RP's in sillytavern but everytime around the 120-150 messages my model degrades to the point where it is repeating sentences from 10 minutes ago
@veldierin , How what settings are you using? i have been trying to get longer RP's in sillytavern but everytime around the 120-150 messages my model degrades to the point where it is repeating sentences from 10 minutes ago
It's hard to say what's allowing me to do it, cause i used to have the same problem when i used exclusively mistral based 24b models for my rps. When i moved over to the 70b's, i kept my 32k context size, and installed "memory books", "openvault", and "lorebookordering" extensions.
For this cydonia (and other mistrals now) I use mistral-v7-tekken-t5-xml for context/instruct templates, and LeCeption-XML-V2.1-Thinking for system prompt (with reasoning blank/disabled). This allows me to continue when i want a different angle on my roleplay from my 70b's.
I would probably start with those extensions i listed above to see if that helps.
I use koboldcpp for my loader, but nothing special there. kv4 for my iqs3_xxs 70b's and generally a q6/kv8 with my 32k context on the mistral 24bs to run on my 5090.