VRAM Requirements for Running the Model
#1
by wilfoderek - opened
How much VRAM is needed to run the model?
Is an H200 sufficient?
Thank you in advance!
Same question here, it seems the model params size is ~200GB, how much VRAM is needed? Thanks
@awni says it only has 37B parameters in memory. I'm not sure how that translates to GB. I'm gonna give it a try.
No that’s not quite right. It only needs to move 37B parameters from RAM to cache. To run this thing you need about 400GB of RAM