I don't know about this in depth, but from what another user in this thread said, a flatpak can't ask a portal to have access to two files at once. If I'm understanding correctly, that would explain why Librewolf needs permission to access ~/Downloads, since it can be downloading more than one file at once, and it needs access to all those files in ~/Downloads at the same time.

EDIT: I got a bit mixed up with what you were saying, but nevertheless, if this is true, then Librewofl would still need permission to access ~/Downloads and so be marked as "potentially unsafe".