Setting up Proxmox on R720
Jun 5, 2022
3 minute read
  • Plan is to have server run Proxmox, with TrueNAS
  • Get HW
  • relatively simple installation of proxmox
    • plug in USB with latest proxmox ISO
    • boot up and select boot device with use UEFI boot
    • install, follow prompts. pretty straightforward prompts
      • I setup mine with static IP
  • wont be able to boot directly into NVMe PCIe
    • possible alternatives - use clover, or change DVD drive to ssd bay
  • Super Helper post to get started with Proxmox
  • enabled GPU Passthrough
    • changed /etc/default/grup to have the following line
      • GRUB_CMDLINE_LINUX_DEFAULT="quiet intel_iommu=on"
        • This was later updated to "quiet intel_iommu=on iommu=pt pci=noaer initcall_backlist=sysfb_init to help resolve pass two differen nvidia cards into two different VM
      • then run update-grub
    • edited /etc/modprobe.d/vfio.conf +
      options vfio-pci ids=<add all your device ids here separated by commas
      
    • added to /etc/modules
      vfio
      vfio_iommu_type1
      vfio_pci
      vfio_virqfd
      
      • ran update-initramfs -u -k all
      • then reboot
    • when adding a new VM, have the following settings
      • BIOS settings
        • q35 for machine type
        • ovmf for boot type
      • Add PCIE hardware.. pick your GPU from list
    • debugging why its not working, look at these files and settings
      • cat /etc/default/grub
      • cat /etc/modules
      • lsmod | grep vfio
      • cat /proc/cmdline
      • dmesg | grep -E 'IOMMU|DMAR'
      • find /sys/kernel/iommu_groups/ -type l
      • lspci

Checklist to #

  • enabling updates
    • by updating source list for debian, were using proxmox 7 and it is based off of Debian Bulleye
      • deb http://download.proxmox.com/debian/pve bullseye pve-no-subscription
      • NOTE: to update this, if you are using any other version other than 7.x.x
    • comment out enterprise apt source in /etc/apt/sources.list.d/pve-enterprise.list
      • prevent updating issues
    • run apt update, apt dist-upgrade, reboot
    • this will get proxmox updated using all the new sources
  • optional. configure storage, and run fdisk /dev/sdXXX to clear out the disks that are avail
    • were gonna pass HDD to TrueNAS, so we’ll only have one drive. that is the Samsung 1Tb drive for all the OS
  • update DNS in /etc/resolve.conf to point to home server DNS
[email protected]:~# cat /etc/resolv.conf
search nameserver
nameserver 10.1.10.1
nameserver 127.0.0.1
[email protected]:~# 
  • plug in dummy HDMI
  • Disable secured boot
  • enable IOMMU - for GPU passthrough

Installing Windows #

  • reinstalled windows, for a main dev machine. mainly fusion 360
    • install without a problem, keys things to remember to setup from the GET GO is UEFI
      • without UEFI, Q35 machine. We wont able to do GPU passthrough Windows11 settings

Troubleshoot #

  • lose qemu guest agent and network connectivity after a little time with PCI Nvidia card added
    • most likely a kernel and nvidia firmware compatibility issue - look here


comments powered by Disqus