Re: [ANN][AUTO-SWITCH] Profit-switch auto-exchange pool: CleverMining.com
so I may have missed this in the many pages here, but have we ever gotten an API?
The whole cloudflare/ddos protection stuff is nice to keep the pool up, but incredibly annoying for both checking hte pages and trying to write data scrapers to track my performance.
If there is no API (still) any one have any suggestions on how to scrape data from their miner pages? I cant seem to get around the cloudflare junk
I am having the same problem in trying to "data scrape" as you call it. I read on the CloudFlare site that their protection does not prevent web crawlers from scanning the site (so I am confident we can find a way to do it), and I'm thinking what I will try next is to write code to accept their cookie(s). Of course, my guess is worthless until I have it running.The whole cloudflare/ddos protection stuff is nice to keep the pool up, but incredibly annoying for both checking hte pages and trying to write data scrapers to track my performance.
If there is no API (still) any one have any suggestions on how to scrape data from their miner pages? I cant seem to get around the cloudflare junk
What I unsuccessfully tried in php was to file_get_contents, then do a javascript setTimeout of 10 seconds, after which I do a window.location.reload.
hmmm, well i guess if they say web crawlers can scan it, there has to be a way. what's the link where you read this?
I went the python route a few weeks back, but got no where. After reading your post, i fired wireshark back up and it looks like there's a hidden form that needs some special key to return in order to let us through ... so I also think we'll need some code to accept their cookie/whatever and respond back similar to how a browser would. still kind of pain to grab some data
update: ok so i did some digging. got this from the cloudflare site:
Javascript and cookies are required for the tests, and to record the fact that the tests were correctly passed. The page which your visitors see when in IUAM can be fully customized to reflect your branding. I'm Under Attack mode does not block search engine crawlers or your existing CloudFlare whitelist.
My guess is they whitelisted known and allowed search engine crawlers which is why those bots work where as ours dont. Add to it that it looks like Terk opted to put the domain in a permanent I'm Under Attack Mode and we are where we are. So pretty much we need to mimic the expected behavior if we want anything to work ... lovely