whitelist only email means you're reading your spam folder to find addresses to whitelist, you're reading the spam. or more realistically, you don't, you miss legit messages, you don't make connections with strangers and that's the end of that.
Instead the spaminess of each message should be visually graded with colored tags indicating certainty of the spaminess, this is "graylisting" or more specifically "shades of gray" listing