I do it because I can... I read release notes on every update and once you've configured a kernel for a particular machine you really don't need to touch the config, barring major changes like when PATA and SATA merged. Or of course if I'm adding a new piece of hardware.

I remove everything I don't need and compiling the kernel only takes a couple minutes. I use Gentoo and approach everything on my system the same way - remove the things I don't need to make it as minimal as possible.

Compiling your own kernel also makes it easier when you need to do a git bisect to determine when a bug was introduced to report it or try to fix it. I've also included kernel patches in my build years ago, but haven't needed to do that in a long time.

I used to compile a custom kernel for my phone to enable modules/drivers that weren't included by default by the maintainer.

It's not about performance for me, it's about control.