quantization

Llama 70B VRAM Requirement: Can You Run It on an RTX 4090?

Don't guess. We tested Llama 3 70B on RTX 4090, 3090, and A100. Here is the exact VRAM breakdown for FP16 vs INT4 and why you might get OOM errors.