nvidia

/

GLM-5.2-NVFP4

like 254

Follow

NVIDIA 62.4k

B200 , 1P1D deployment. I found 2 problems:

1. TAU2 OVERALL score is only 0.8154
2. Print error in prefill node: CUDA Error: CUDA_ERROR_INVALID_VALUE /workspace/include/flashinfer/trtllm/fmha/fmhaKernels.cuh 328

prefill:
--max-prefill-tokens=202752
--context-length=1048576
--mem-fraction-static 0.80
--chunked-prefill-size 65536
--max-running-requests 48
--page-size 64
--disaggregation-transfer-backend=mooncake
--disaggregation-ib-device=${NCCL_IB_HCA}
--disaggregation-mode=prefill
--enable-hierarchical-cache
--hicache-size=200
--hicache-io-backend=kernel
--schedule-conservativeness=0.4
--disaggregation-bootstrap-port 8998
--reasoning-parser glm45
--tool-call-parser glm47 \

decode:
--max-prefill-tokens=202752
--context-length=1048576
--mem-fraction-static=0.88
--cuda-graph-max-bs 512
--load-balance-method total_tokens
--enable-torch-symm-mem
--enable-mscclpp
--page-size 64
--disaggregation-transfer-backend=mooncake
--disaggregation-ib-device=${NCCL_IB_HCA}
--disaggregation-mode=decode \