DeepSeek-V4-Flash-0731
looking into it :)
Also downloading to create quantized GGUFs. @bartowski not sure why you didn't create quants below the original MXFP4, but I published scripts that show some examples of how to convert and quantize: https://huggingface.co/tarruda/DeepSeek-V4-Flash-GGUF/tree/main/scripts
Also this new version comes with dspark draft models built in, but we might need to wait for llama.cpp to fully support it https://github.com/ggml-org/llama.cpp/pull/25173#issuecomment-5117724461
https://huggingface.co/bartowski/DeepSeek-V4-Flash-0731-GGUF
uploaded the mxfp4 version which is full quality
I may look into smaller versions, I was under the impression that quantizing the mxfp4 layers resulted in very aggressive degradation, but I may have been wrong, so will investigate for this model
Thanks!
, I was under the impression that quantizing the mxfp4 layers resulted in very aggressive degradation, but I may have been wrong
Give it a shot at my quants: https://huggingface.co/tarruda/DeepSeek-V4-Flash-GGUF
Even the Q2_K works well, but IQ3_XXS seems to be the sweet spot that works on 128G devices.
I'm undecided, I tried going from the current 4.4 bpw to even just 4.0 bpw and got surprisingly big differences in KLD:
====== Perplexity statistics ======
Mean PPL(Q) : 8.250368 ± 0.111366
Mean PPL(base) : 4.977220 ± 0.058168
Cor(ln(PPL(Q)), ln(PPL(base))): 80.77%
Mean ln(PPL(Q)/PPL(base)) : 0.505386 ± 0.007998
Mean PPL(Q)/PPL(base) : 1.657626 ± 0.013258
Mean PPL(Q)-PPL(base) : 3.273148 ± 0.072952
====== KL divergence statistics ======
Mean KLD: 0.728716 ± 0.005622
Maximum KLD: 18.404858
99.9% KLD: 10.750040
99.0% KLD: 6.275336
95.0% KLD: 2.992667
90.0% KLD: 1.878664
Median KLD: 0.309511
10.0% KLD: 0.000662
5.0% KLD: 0.000078
1.0% KLD: 0.000002
0.1% KLD: -0.000000
Minimum KLD: -0.000004
====== Token probability statistics ======
Mean Δp: -8.865 ± 0.113 %
Maximum Δp: 99.649%
99.9% Δp: 88.941%
99.0% Δp: 51.360%
95.0% Δp: 19.118%
90.0% Δp: 8.059%
75.0% Δp: 0.077%
Median Δp: -0.554%
25.0% Δp: -13.332%
10.0% Δp: -41.793%
5.0% Δp: -64.764%
1.0% Δp: -95.421%
0.1% Δp: -99.829%
Minimum Δp: -100.000%
RMS Δp : 26.492 ± 0.147 %
Same top p: 70.470 ± 0.206 %
like I'm sure it's usable I guess, but it really doesn't feel worth it
@bartowski I have uploaded some quants and even the Q2_K has been working very well here. Could you test PPL/KLD of my IQ3_XXS? To me it feels practically the same as using Deepseek v4 flash official API https://huggingface.co/tarruda/DeepSeek-V4-Flash-0731-GGUF/tree/main/IQ3_XXS
@tarruda hm that performed better than expected:
====== Perplexity statistics ======
Mean PPL(Q) : 5.631696 ± 0.068507
Mean PPL(base) : 4.977220 ± 0.058168
Cor(ln(PPL(Q)), ln(PPL(base))): 92.39%
Mean ln(PPL(Q)/PPL(base)) : 0.123539 ± 0.004676
Mean PPL(Q)/PPL(base) : 1.131494 ± 0.005291
Mean PPL(Q)-PPL(base) : 0.654476 ± 0.026711
====== KL divergence statistics ======
Mean KLD: 0.241077 ± 0.002862
Maximum KLD: 14.848409
99.9% KLD: 7.068842
99.0% KLD: 3.234927
95.0% KLD: 1.140221
90.0% KLD: 0.576226
Median KLD: 0.046611
10.0% KLD: 0.000103
5.0% KLD: 0.000015
1.0% KLD: 0.000000
0.1% KLD: -0.000002
Minimum KLD: -0.000004
====== Token probability statistics ======
Mean Δp: -2.481 ± 0.072 %
Maximum Δp: 99.329%
99.9% Δp: 81.362%
99.0% Δp: 41.644%
95.0% Δp: 13.745%
90.0% Δp: 6.404%
75.0% Δp: 0.470%
Median Δp: -0.017%
25.0% Δp: -2.667%
10.0% Δp: -13.508%
5.0% Δp: -28.190%
1.0% Δp: -75.073%
0.1% Δp: -98.125%
Minimum Δp: -99.978%
RMS Δp : 16.208 ± 0.140 %
Same top p: 84.265 ± 0.164 %
So maybe there's some validity to it, need to keep researching, will explore and consider the options
@bartowski can you share the reference logits (upload temporarily so I can download it) and dataset? I'd like to try playing with more quant formulas to see if it is possible to extract better numbers
I've converted the recently released vision deepseek, ran KLD/PPL and saw some interesting results: The Q3_K perplexity was actually lower than the original MXFP4: https://huggingface.co/tarruda/DeepSeek-V4-Flash-Vision-Exp-GGUF/tree/main/logs