Ninjastic
PostEdited versionsQuotes to this post
Post
Topic
Board Meta
Merits 12 from 7 users
Topic OP
@theymos: is it possible to get an exception from Cloudflare for scrapers?
by
LoyceV
on 29/12/2022, 21:47:20 UTC
⭐ Merited by Foxpup (4) ,NotATether (2) ,Daniel91 (2) ,NeuroticFish (1) ,shahzadafzal (1) ,vapourminer (1) ,DdmrDdmr (1)
Today, Cloudflare was blocking scraping bots again. For 8 hours and 24 minutes, my scraper wasn't working. Anything between posts 61513760 and 61515614 (1853 posts) is missing, and amongst others my notification bot wasn't working. I already delayed my Merit and Trust data updates (which normally start on Friday and Saturday morning).
This happened 18 days ago too.

Public request to theymos: Is it possible to get an exception from Cloudflare for popular scraping bots? I'm thinking of BPIP.org, Ninjastic.space and loyce.club, but there are probably a few others that provide useful features for the Bitcointalk community. It would be really nice if DDOS-protection doesn't interfere with features.