You do have to turn off battery saver for the background process. Phones tend to not like background processes. That would cause the behavior you're seeing.
Settings > Apps > App battery usage > Immich > Set to "unrestricted"
Also I have mine set to a ten minute delay, maybe that's why I haven't noticed. Maybe try adding a small delay to the load?
(Primarily I wanted a chance to delete photos before they uploaded.)