Quote from: unabomber on April 16, 2018, 05:33:45 PM

Quote from: Vod on April 16, 2018, 02:51:17 PM

No, my crawler does nothing but load the page - does not follow or index any links.

I have noticed there are some old profiles that still have an avatar from an external source, and they time out when I try to connect. Examples:
https://bitcointalk.org/index.php?action=profile;u=3758
https://bitcointalk.org/index.php?action=profile;u=5153

I feel this is somehow related to that issue.

Why is your crawler even downloading images? all I would expect it to do is just grab the HTML.

Hmmm, good point. I run the default webBrowser control in C#, which loads everything into a little browser window. I'll see if there is a way to have it only download the code and not load anything else...

Edit: I run the parser on a VM where I don't need to browse, so I just had to disable the loading of images from IE. Thanks!