I think you're either looking for a tunnelling solution or a Web Application Firewall. And there are several options. ModSecurity, or SafeLine, BunkerWeb, Coraza, open-appsec. For (AI) crawlers and bad bots we have Anubis, Iocaine... or something like more traditional blocklists or something like OpenResty which is some modified Nginx plus modules...
The tunneling itself can be done with a simple reverse proxy like Nginx, Traefik, Caddy...
I've messed around a bit with ModSecurity. But I'm still looking for something aganist the crawlers.