Found this discussion thread, I'm going to try this
https://discussion.fedoraproject.org/t/after-install-fedora-44-nvidia-drivers-failed-to-load-and-it-falled-back-to-nouveau/189884/16
It worked!
For some reason the nouveau driver still loads but I set the option modeset=0 on it and now it doesn't grab the GPU, and then the nvidia drivers load
Ok, time to check if virtualbox still works