Post
Topic
Board Project Development
Re: BPIP problems
by
Vod
on 16/04/2018, 18:25:16 UTC
No, my crawler does nothing but load the page - does not follow or index any links.

I have noticed there are some old profiles that still have an avatar from an external source, and they time out when I try to connect.  Examples:
https://bitcointalk.org/index.php?action=profile;u=3758
https://bitcointalk.org/index.php?action=profile;u=5153

I feel this is somehow related to that issue.  


Why is your crawler even downloading images? all I would expect it to do is just grab the HTML.

Hmmm, good point.  I run the default webBrowser control in C#, which loads everything into a little browser window.  I'll see if there is a way to have it only download the code and not load anything else...

Edit:  I run the parser on a VM where I don't need to browse, so I just had to disable the loading of images from IE.  Thanks!