I've done symlinks into a separate directory before, but by far my favorite method is to just let ~ be a git repo. It's maximally simple, no other tooling needed besides git.

There are a few key steps to making this work well:

• echo '*' > ~/.gitignore: This way git status isn't full of untracked files. I can still git add -f what I actually want to track.
• git branch -m dots: For clarity in my shell prompt.
• [ -d "$HOME/.local/$(hostname)/bin" ] && PATH=$PATH:$HOME/.local/$(hostname)/bin and similar if there's config I want to apply only to certain hosts.