If the encryption is ever hacked, knowing who you are is probably the least of anyone's concerns. I would imagine that any adversary could build a profile or plan a response without knowing a particular phone number.
"These two people are planning civil rights activism here on Friday," is just as useful as, "MLK Jr and Malcolm X are planning activism here on Friday."
Thankfully, they'd have to not only break encryption but also MitM the conversations, since Signal doesn't actually store chat data on their servers.