BTW, How do you manage to parse every user's profile without triggering Cloudflare? You've parsed ~23m pages so far. Wink

Time.

Theymos restricts it to one page per second per IP. I have a few parsers running in case one fails, and I parse about 3-4 pages a second.

Thanks a lot @Vod. I forget that darn sleep command. Tongue

Time.

Theymos restricts it to one page per second per IP. I have a few parsers running in case one fails, and I parse about 3-4 pages a second.

I'm assuming you have multiple parsers that binds to different IPs to achieve 4 PPS without triggering the one PPS limit. Correct me if I'm wrong.