My recommendation, and keep in mind this is really crude, but if you have a good DDoS solution already in place for the obvious stuff, the more devious stuff is going to be something that requires using a combination of logs and data that could be compiled even from a simple awstats page. Sort IPs by page count over hit count and the closer they are to a "1" ratio the higher they increase in bandwidth over time, just auto-ban them.
You could use the time while under Cloudflare to build a good baseline for "normal" user behavior and then define that as your method. You're right about it being difficult to script up a system for this, but it could maybe be done. Your solution already sounds pretty ingenious as it is, but you're right in that it almost becomes a full-time job just keeping ahead of everything when it comes to automating it.
I don't fault you for going with Cloudflare, but even with their assurances and transparency, I still don't trust them. The government will inevitably use them for wiretapping again, and because of gag orders, they will comply. Just as they have in the past.
This is just an ELK stack + webserver logs. It'd fail due to IPv4 exhaust and NAT/tor tunnels.
---
What sort of memory cache engine are you using? Are you doing any frontend caching (varnish)? Have you considered offloading static content(see edit). Are you separating the database server from the application server? What's the actual infrastructure look like for this SMF host. Are you doing master-master SQL clustering? Are you opposed to having CDNs serve static content (such as those on ip.bitcointalk.org or the image proxy... probably the actual server's IP address).
It looks as though ip.bitcointalk.org is actually hosted on digitalocean? Is the entire site hosted on digitalocean or just the image proxy server? With more information about the internal infrastructure of bitcointalk, we can actually help to improve it rather than throwing random ass suggestions that normally involve fairly standard SIEM technologies or basic elastic computing features.
---
Edit: A bit more exploring into ip.bitcointalk.org shows a *.bitcointalk.org wildcard cert. But it doesn't seem to be serving actual bitcointalk data when I'm telling my header is bitcointalk.org =(