Post
Topic
Board Project Development
Merits 5 from 1 user
Re: BPIP problems
by
unabomber
on 16/04/2018, 17:33:45 UTC
⭐ Merited by Vod (5)
No, my crawler does nothing but load the page - does not follow or index any links.

I have noticed there are some old profiles that still have an avatar from an external source, and they time out when I try to connect.  Examples:
https://bitcointalk.org/index.php?action=profile;u=3758
https://bitcointalk.org/index.php?action=profile;u=5153

I feel this is somehow related to that issue. 


Why is your crawler even downloading images? all I would expect it to do is just grab the HTML.