My changes take maybe a few seconds less, but I find that quite normal. It could be faster, but let's be honest how often does one run that? I run it maybe once a week or once a month. 25 seconds a week is nothing to me.
Normally I test packages with nix-shell -p $package then when I'm satisfied, it goes into my configuration and the time to do so doesn't matter anymore.