Re: shdvb has the best "Attitude" when it comes to sending trust (BPIP)
BTW, How do you manage to parse every user's profile without triggering Cloudflare? You've parsed ~23m pages so far. 
Time.
Theymos restricts it to one page per second per IP. I have a few parsers running in case one fails, and I parse about 3-4 pages a second.
Time. Theymos restricts it to one page per second per IP. I have a few parsers running in case one fails, and I parse about 3-4 pages a second.
I'm assuming you have multiple parsers that binds to different IPs to achieve 4 PPS without triggering the one PPS limit. Correct me if I'm wrong. Theymos restricts it to one page per second per IP. I have a few parsers running in case one fails, and I parse about 3-4 pages a second.