FP8 version
#19
by
Daemontatox
- opened
is there a possibility to get a FP8 version ?
@Daemontatox - Please stay tuned this week. We plan to release post quantised FP8 version in coming days. I will ping here as soon as it is available publicly.
@Daemontatox @uzvisa - please find FP8 model here - https://huggingface.co/tiiuae/Falcon-H1R-7B-FP8 . We wanted to make sure that the evals are up to the mark. Also please make sure to use latest vllm nightly as some fixes were required in vllm for FP8 enablements. Please find our FP8 release blog too - https://falcon-lm.github.io/blog/falcon-h1r-7b-fp8/
puneeshkhanna
changed discussion status to
closed