The closest we have right now is probably Resonite. It's the successor to NeosVR; the original dev team managed to claw the rights to the code away from the cryptobro who owned Neos and started their own with the code. It's not open source, but you're able to code and model in-game.

We do need some kinda fediverse VR platform though. The biggest issue is that it'd probably be a lot more demanding than something like Matrix or Lemmy. However, I think it'd fix a lot of issues that platforms like VRC have (one of the biggest user complaints about VRC specifically is that they technically don't allow anything NSFW and have been dragging their heels when it comes to adding an adult-only instance option).