The counter argument to that is that it negatively impacts lower class people who are unable to take time off work to go do those things, thus disproportionately hindering lower class and minority rights.

And the counter argument to that is that there should be enough safety nets in place to allow all people to be able to take time off work as needed.

That would have people really confused. "We have to raise minimum wage to allow everyone the right to bear arms"