Setting up Proxmox on R720

Jun 5, 2022

3 minute read

PROJECT LAYOUT

Plan is to have server run Proxmox, with TrueNAS
- TrueNAS will get HDD passthrough, hence Flash IT Mode Firmware to H710P
- all proxmox and its
Get HW
- DVD caddy and an old SSD for Proxmox
  - use this, because r720 cannot boot into a PCIe NVMe drive
- NVMe to PCI2 adapter
- Samsung 1tb NVMe
  - for all VM
- GPU Powercable for nVidia card
relatively simple installation of proxmox
- plug in USB with latest proxmox ISO
- boot up and select boot device with use UEFI boot
- install, follow prompts. pretty straightforward prompts
  - I setup mine with static IP
wont be able to boot directly into NVMe PCIe
- possible alternatives - use clover, or change DVD drive to ssd bay
Super Helper post to get started with Proxmox
enabled GPU Passthrough
- changed /etc/default/grup to have the following line
  - GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
    - This was later updated to "quiet intel_iommu=on iommu=pt pci=noaer initcall_backlist=sysfb_init to help resolve pass two differen nvidia cards into two different VM
  - then run update-grub
- edited /etc/modprobe.d/vfio.conf
```
options vfio-pci ids=<add all your device ids here separated by commas
```
- added to /etc/modules
```
vfio
vfio_iommu_type1
vfio_pci
vfio_virqfd
```
  - ran update-initramfs -u -k all
  - then reboot
- when adding a new VM, have the following settings
  - BIOS settings
    - q35 for machine type
    - ovmf for boot type
  - Add PCIE hardware.. pick your GPU from list
- debugging why its not working, look at these files and settings
  - cat /etc/default/grub
  - cat /etc/modules
  - lsmod | grep vfio
  - cat /proc/cmdline
  - dmesg | grep -E 'IOMMU|DMAR'
  - find /sys/kernel/iommu_groups/ -type l
  - lspci

Checklist to #

enabling updates
- by updating source list for debian, were using proxmox 7 and it is based off of Debian Bulleye
  - deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription
  - NOTE: to update this, if you are using any other version other than 7.x.x
- comment out enterprise apt source in /etc/apt/sources.list.d/pve-enterprise.list
  - prevent updating issues
- run apt update, apt dist-upgrade, reboot
- this will get proxmox updated using all the new sources
optional. configure storage, and run fdisk /dev/sdXXX to clear out the disks that are avail
- were gonna pass HDD to TrueNAS, so we’ll only have one drive. that is the Samsung 1Tb drive for all the OS
update DNS in /etc/resolve.conf to point to home server DNS

root@beast:~# cat /etc/resolv.conf
search nameserver
nameserver 10.1.10.1
nameserver 127.0.0.1
root@beast:~#

plug in dummy HDMI
Disable secured boot
enable IOMMU - for GPU passthrough

Installing Windows #

reinstalled windows, for a main dev machine. mainly fusion 360
- install without a problem, keys things to remember to setup from the GET GO is UEFI + without UEFI, Q35 machine. We wont able to do GPU passthrough

Backup and Restore Proxmox #

NOTE: this can only be done on the same hardware!!!!

Mount UBS Drive for Proxmox Backup · GitHub a. Formatted the entire disk with mkfs.ext4 /dev/sdb for the entire disk, and not just the partition
backup and restore script GitHub - huynle/proxmox-stuff: This is a collection of stuff that I wrote for Proxmox.
setup monthly cronjob, by putting the script in /etc/cron.monthly/

Troubleshoot #

lose qemu guest agent and network connectivity after a little time with PCI Nvidia card added
- most likely a kernel and nvidia firmware compatibility issue - look here