Lemmynsfw uses a custom version that that removes the blur for nsfw images but that's it I think.

Are there instances that run modified versions of the base Lemmy software? For example, that use their own sorting algorithms, or provide users ways to block instances or specific users, etc?

If one had developed code to do these things, why would one not upstream it so it's released in core lemmy and all instances can benefit from that capability?

The instances that have been developing custom features (IIRC lemm.ee and a couple others) have been contributing them back into the main Lemmy project, so everyone else ends up eventually getting them.

Edit: I've seen a community for instance owners and devs somewhere but don't remember the name, hopefully someone else remembers

Yes, Hexbear is highly customized. Beehaw is o believe. Not sure what else

The NSFW instances use a modified front end to allow for automatic picture expansion (and probably other things).