The technology has "been there" for a while, it's trivial do setup what you're asking for, the issue is that games have anti cheat engines that will get triggered by the virtualization and ban you.

I’ve been doing exactly that at home for a couple years now. First with Parsec, now Sunshine/Moonlight.

Host is Proxmox on Ryzen 5800x, 64gm RAM GPU is 2070 Super, with VGPU patched drivers from https://gitlab.com/polloloco/vgpu-proxmox

When I’m gaming I’ll dedicate the full 8Gb to my windows Vm, otherwise I split it in 2 or 4Gb chunks to Jellyfin or my home camera monitoring. 8gb can’t split very many ways, and most things require at least 2 to run.

Locally at home I can run 1440p 60fps rock solid over wifi on any device, from my phone/old laptop/apple tv/raspberry pi. Remote I can do 1080p60, but a bit more hit or miss depending on my network connection.

Experimenting with LLMs I’ve done through the same windows VM, or to a ubuntu dev VM. Works the same way. I’m thinking of transitioning my gaming VM to Linux too.

The amount of VRAM is the hard limitation to get past, the virtualization tech itself has been there for a while.

But to be perfectly honest……it really was just a “let’s see if I could do this” type task, direct GPU pass though is more straightforward and it’s not really worth splitting 8Gb these days. Unless you get a card with significantly more VRAM passthrough is much less work.

I bought a cheap used Dell R710 on Facebook marketplace for like $100 or so, as well as an ups, rack, 10g switch, etc, from various other sellers. All told, I’ve got about $500 in my server setup.

Installed proxmox on it. It’s “free” if you don’t buy a license. You just have to put up with a little nag screen when you open the control panel but it still works 100%, much like winrar.

Works great.

Edit: just realized this is in c/selfhosted AND I misunderstood the post. I’m gonna leave it here just on the off chance it’s useful to somebody, but I acknowledge it’s not what you’re looking for.

I know you asked about VMs, but fwiw there are GPU-capable containers now: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

Used one of these and the setup is as easy as it sounds. It can run Houdini, Stable Diffusion.

I've also wanted to do this for a while, but there were always a few too many barriers to actually spin up the project. Here's just a brain dump of things I've seen recently.

vGPUs continue to be behind a license. But there is now vgpu_unlock.

L1T just showed off PCIe "fabric" from Liqid that can switch physical devices between machines.

Turning VMs on and off isn't as slick as either of the above, but that is doable today. You'll just have to build all the switching automation yourself. That could just be a shell script running QEMU/libvirt commands, at a minimum.

You're not really describing your use-case here. Are you just trying to run a server that does all your rendering for you so you can play games elsewhere? Yes, that's totally possible.

If you're trying to describe a business...no, it's not possible, scalable, or profitable.

I'm curious as to what your intentions are here though.

You can use proxmox to do most of this. Currently my set will only pass-through the gpu to one VM. I have heard of splitting the power among VMs but I have not gone down that rabbit hole. If I want to play with llms I fire up that server, if I want to game, I shut that down and fire up my windows 10 vm.

I currently have a setup exactly like this, with a threadripper 2950x, an RX 6600, and a 2070 super.

Let me know if you have any questions in the specifics, but its 100% possible

Best part of this setup is being able to connect to both via sunshine on many displays at once

OK, but why?

Well, for fun and as a cool hobby project, I get that. That is enough to justify it, like any other crazy hobbyist project. Don't let me stop you.

But in the spirit of practicality and speaking hypothetically: Why set it up that way?

For self-hosting why not build a few standalone machines and run off that instead? The reason to do this large scale is optimizing resources so you can assign a smaller pool of hardware to users as they need it, right? For a home set of two or three users you'd probably notice the fluctuations in performance caused by sharing the resources on the gaming VMs and it would cost you the same or more than building a couple reasonable gaming systems and a home server/NAS for the rest. Way less, I bet, if you're smart about upgrades and hand-me-downs.

I've recently tried to do that using sunsine and different linux gaming distros and it was awful, the VM was working great for a few minutes and then suddenly crashes and I have to hard stop it.

All the people that I've seen talking about it on the internet are using Windows VMs so I guess that I'm doing something wrong or the only way to do it is through a Windows VM, which I'll not even try.

I've never done anything like this myself but maybe Sunshine with Moonlight might be worth a look

Craft computing has been chasing this for several years now. His most recent attempt being the most successful one. https://m.youtube.com/watch?v=RvpAF77G8_8

You should take a look at https://www.youtube.com/channel/UCp3yVOm6A55nx65STpm3tXQ he does have a serie about doing something like this and go in depth in it

As others have expressed- were already there. Understand though that the reason this hasn't caught on mainstream is the entire purpose of what you are asking is simple: it runs counter to the standards of commercial capitalism. We are talking about efficiency, self hosting, doing more with less, and cutting strings.

That said- understand that what you are undertaking is not dissimilar from building infrastructure in a company. You are building and expanding to meet your needs. Your needs are unique so there isn't a 'turn key' solution that will fit perfectly... so you need to try things and see what works.

As far as things you are talking about specifically: you are going to ultimately be dipping your toes into the virtualization world... so xcp-ng and proxmox are good choices. If you can get your hands on older copies and uh... source a key or two: esxi is also very beginner friendly but won't be able to upgrade thanks to their new pricing model. You seem like you are aware of the YouTube sphere so let me recommend 2GuysTech and the series on different hypervisors.

Once you decide on a hypervisor it's as 'simple' as building a PC to meet your needs. If you have one already I'd start there to get a feel for how much you can pull out of it to determine how much you may need. You can probably split up a single GPU or just pass it through (cost vs performance.). LLMs are power / resource hungry so that may require it's own GPU.

If power is cheap by you you can look into older server hardware but honestly this can be a messy space to dabble in (noise, heat, power costs.)

From there play with services that fit your needs.

It's very doable and there are some easier paths to take... certainly- but again the thing about homelabs is it's very custom. This is why the community (in general) is willing to help. We all have had to forge the same path.

Do you really need multiple VMs, can't you run all at one? The easiest would be to install some windows/Linux on a single machine. Then stream your games with Sunshine/Moonshine and connect over RDP/VPN?