I was concerned with this myself and planned to just add a physical card under my phone case, but I was suprised how little difference using the card normally made.

And if you care about privacy, you probably shouldn't make your purchases using a google app.