Quote from: toxic0n on April 23, 2014, 05:41:56 PM

Quote from: jordo on April 23, 2014, 06:33:13 AM

Quote from: GolfCabalist on April 22, 2014, 08:04:40 PM

Quote from: boxofspuds on April 22, 2014, 06:18:09 AM

Quote from: GolfCabalist on April 22, 2014, 05:01:20 AM

Quote from: boxofspuds on April 22, 2014, 04:29:43 AM

so I may have missed this in the many pages here, but have we ever gotten an API?

The whole cloudflare/ddos protection stuff is nice to keep the pool up, but incredibly annoying for both checking hte pages and trying to write data scrapers to track my performance.

If there is no API (still) any one have any suggestions on how to scrape data from their miner pages? I cant seem to get around the cloudflare junk

I am having the same problem in trying to "data scrape" as you call it. I read on the CloudFlare site that their protection does not prevent web crawlers from scanning the site (so I am confident we can find a way to do it), and I'm thinking what I will try next is to write code to accept their cookie(s). Of course, my guess is worthless until I have it running.

What I unsuccessfully tried in php was to file_get_contents, then do a javascript setTimeout of 10 seconds, after which I do a window.location.reload.

hmmm, well i guess if they say web crawlers can scan it, there has to be a way. what's the link where you read this?

I went the python route a few weeks back, but got no where. After reading your post, i fired wireshark back up and it looks like there's a hidden form that needs some special key to return in order to let us through ... so I also think we'll need some code to accept their cookie/whatever and respond back similar to how a browser would. still kind of pain to grab some data Undecided

update: ok so i did some digging. got this from the cloudflare site:
Javascript and cookies are required for the tests, and to record the fact that the tests were correctly passed. The page which your visitors see when in IUAM can be fully customized to reflect your branding. I'm Under Attack mode does not block search engine crawlers or your existing CloudFlare whitelist.

My guess is they whitelisted known and allowed search engine crawlers which is why those bots work where as ours dont. Add to it that it looks like Terk opted to put the domain in a permanent I'm Under Attack Mode and we are where we are. So pretty much we need to mimic the expected behavior if we want anything to work ... lovely

I neednt link you now, your text is exactly what I read. I did not reach the conclusion that you did about the crawlers being whitelisted, thanks for helping me think! Wink

As to "kind of a pain", well, it would be worth it so I could get some sleep, I'm plain stupid about having to check every hour and enter data into a spreadsheet. Auto-reading from CM is the only missing link. I'm databasing the exchange rates of 5 different coins once every five minutes, making graphs, all automatically.

See https://github.com/jordoh/miner-monitor/blob/master/pools/clever_mining.rb#L84-L102 for ruby code. The basic process is:

1. Get the 503 page from CloudFlare - it contains a verification code (in the HTML) and simple equation to solve (in the JS).
2. Make a get request for http://www.clevermining.com/cdn-cgi/l/chk_jschl?jschl_vc=&jschl_answer= with the referer header set to the page you are trying to access. If you got the answer right, you'll a 302 (redirect) response with a cf_clearance cookie.
3. Follow the redirect (i.e. request the original page you were trying to access), including the cf_clearance cookie.

Thanks for this, I was looking into making an Android widget for CM and looks like I will have a way to do so.

Do you have an example of this code in C#?

Buddy

Posts

Merits