A few years ago I did stuff like this using Puppeteer. This is not one of the advertised use cases, but normally Puppeteer comes with a fully featured WebRTC implementation, and for my needs, also included the WebAudio spec. Using a combination of these, I could route, transform, mix, broadcast audio without changing or renegotiating the connection details, and also preserve the privacy of the peers by having only the IP of Puppeteer be actually exposed. It worked far better than I had ever expected.