You didn't have to tell us you are a man, it was obvious. What you just did is provide a great example of why we need better education on gender based and sexual violence.
Firstly, there is almost a complete certainty that you know a rapist and may even be friendly with one. The overwhelming majority of sexual violence perpetrators are men and that violence is facilitated through the nuclear family model and social ostricization of men who challenge rape culture.
I have had multiple women in different generations of my family disclose sexual violence from men beloved by dozens of people who were none the wiser. I have sat at tables where men make jokes about getting women drunk on purpose like it doesn't reflect lived experiences only for other men to laugh along enthusiastically or nervously. I have been in classes where men discuss harassing women in the workplace without it being challenged. I could not name the amount of friends I've lost when I challenge them on their views, all of them have been men. Every one of those personal experiences reflects statistical data we have on sexual violence rates and rape culture reproduction.
You seem to have this fantasy that rapists are men who jump out of the darkness to prey on strangers, but they're your friends and family who prey on those you know. They do talk about it, all the time. Whether they're talking about the act of rape or why rape is okay, they talk about it. You cannot insulate yourself from this culture without abandoning the responsibility to challenge it.
When you say shit like, "not all men," you're admitting to an insecurity you have about the potential injustice of rape accusations and the association between manhood and sexual violence. Too bad, you're in it and have to choose to challenge it or continue to participate through inaction. The fear of false rape accusations is almost completely unfounded in a world where women face consequences for reporting and convictions of sexual assault are so difficult to get. False convictions are so statistically rare and acting like they arent shows that you value your comfort more than believing victims. Men are the vast majority of rapists and most women have experienced gender based or sexual violence at one point in their lives. As long as they exist in a culture where that violence will most likely come from men and most men do not challenge it, manhood will be associated with sexual violence. When you say that if a cartoon rapist should ever appear you'll give'em a wollop, you're subscribing to an imagination of sexual violence that obscures the reality of its frequency and potential vectors. You're telling the women around you that you do not care about what they experience and only think of rape as a vector for you to fulfill the masculine fantasy of violently punishing a rapist to protect the poor, helpless women.
This is all of course ignoring your fundamental misunderstanding of what rape is when you discuss it as though it's a form of sex. There is no discussion to be had about whether men kiss and tell about rape because rape is not sex. It is about violence and power. It happens at high rates in highly patriarchal societies exactly because of the brutal hierarchy of power that must be maintained for that patriarchy to exist.
I don't care if most men don't rape, most rapists are men and the greatest resistance we get to challenging rape culture is from men. If that bothers you, learn.