Residential proxies are a thing.

Also speaking from experience, especially asia based crawlers will send 5 requests at once from one IP, then 5 from the next etc. and the first IP wont appear again for several hours, making ip based rate limiting useless.

In my case i have honeypot links that block them, i block several data center adress ranges, have an automated whitelist for registered users, use some public blocklists and user agent filtering and that takes care of 99% of it. But it is non-trivial compared to rate limiting.