Jitsi is close but it purely does video calling.

Not sure what you mean by that. Turn off the camera and you've got an audio chat.

I want something that is a chat where guests can join a group with a link.

That's exactly how Jitsi works.

Signal is not self hosted but they also support videoconference style calls.