@anolemmi It’s supposed to be automatic as to which volume level is adjusted. For example, if you adjust the volume while Siri is speaking, it adjusts the Siri volume (only). But I find it tricky to quickly grab the phone and adjust it before Siri stops talking.
It’s understandable that users are confused that invisible volume levels are being adjusted; and either the one they wanted was not adjusted or others were adjusted unintentionally.
Edit: tag @apple_enthusiast