Try wearing very thin neoprene under your bigger gloves. It's been a game changer for me. I have a horrible habit of taking my gloves off from years of snowboarding and those have been awesome.