Proxmox-VE NVIDIA GPU Passthrough

Proxmox VE (Virtual Environment) allows for GPU passthrough, enabling a virtual machine (VM) to use the full power of a physical GPU. This is useful for high-performance graphics, gaming, rendering, or machine learning.

Prerequisites

Proxmox VE installed: Ensure Proxmox VE is installed and running on your system.
Supported hardware: Your CPU and motherboard must support IOMMU (Intel VT-d or AMD-Vi).
NVIDIA GPU: A GPU supported by your VM’s operating system.

Step-by-Step Guide

1. Enable IOMMU in BIOS

Reboot your machine and enter the BIOS/UEFI settings.
Look for settings related to virtualization, such as “Intel VT-d” or “AMD IOMMU”, and enable them.
Save and exit the BIOS/UEFI.

2. Enable IOMMU in Proxmox

Edit the GRUB configuration file:

  nano /etc/default/grub

Add the following parameters to GRUB_CMDLINE_LINUX_DEFAULT:
- For Intel CPUs:
```
intel_iommu=on
```
- For AMD CPUs:
```
amd_iommu=on
```
Update the GRUB configuration:

update-grub

Reboot the Proxmox server:

reboot

3. Verify IOMMU Groups

After rebooting, verify IOMMU groups:

find /sys/kernel/iommu_groups/ -type l

This command will list the devices and their respective IOMMU groups.

4. Blacklist the NVIDIA Driver

Edit the modprobe blacklist configuration:

nano /etc/modprobe.d/blacklist.conf

Add the following lines:

blacklist nouveau
blacklist nvidia
blacklist nvidiafb

Update the initial RAM filesystem:

update-initramfs -u

Reboot the Proxmox server again:

reboot

5. Install Necessary Packages

Install the vfio modules:

apt-get update
apt-get install pve-headers
apt-get install -y vfio vfio-pci

6. Configure GPU Passthrough

Identify the GPU and its associated audio device using lspci:

lspci -nn

Note the PCI IDs of the GPU and the audio device. They look something like ‘01:00.0’ and ‘01:00.1’.
Edit the VFIO configuration to bind these devices:

nano /etc/modprobe.d/vfio.conf

Add the following lines, replacing the PCI IDs with your own:

options vfio-pci ids=10de:1c82,10de:0fb9

7. Edit Proxmox Configuration Files

Add the following to /etc/modules:

vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd

Edit the Proxmox VM configuration file:

nano /etc/pve/qemu-server/<vmid>.conf

Add the following lines:

hostpci0: 01:00.0,pcie=1,x-vga=1

8. Install NVIDIA Drivers on the VM

Start the VM and install the NVIDIA drivers within the guest operating system.

Troubleshooting Tips

Check dmesg logs: Useful for debugging IOMMU issues.

dmesg | grep -e DMAR -e IOMMU

Ensure no other VM or the host is using the GPU.
Verify IOMMU groups: Some devices might need to be isolated into separate groups.

Conclusion

With these steps, your Proxmox VE setup should be capable of passing through an NVIDIA GPU to a VM, allowing it to utilize the GPU directly. This setup is beneficial for high-performance workloads requiring direct access to GPU resources.