Had a PM from comeonalready, which brings up a great question (and a reasonably long answer), figured I'd post it here just so everyone has the same information 
You're absolutely correct on those blocks. We've been tweaking (every day or so) to see what the right amount of time between coin switches is (how quickly we should react to new coins being most profitable). The answer right away might be "instantly", but there are some subtle problems with it unfortunately.
Switching coins extremely quickly (and for some coins, this is every 3-4 seconds in testing - will be better after we remove some super fast coins) causes a ton of extra load on the server in terms of bandwidth (sending all new work to a thousand clients), and this actually props up our orphan rate if we find a block during those times (and if we're switching low-diff coins quickly, we find blocks pretty rapidly) because our network upload gets a bit slow, and block propagation speed drops, leading to orphans. One solution that we've played with here is to raise our transaction fee minimums on a per-coin basis, leading to smaller blocks (quicker propagation), and less outgoing bandwidth on new work for miners, but at the same time, drops profitability since you're not getting as much in transaction fees, its also detrimental to the individual coin network.
Another issue is queued work on clients. A large number of clients, even after they've been asked to switch coins, will continue submitting work on the old coin for up to 30+ seconds (I think this is due to queued work, not 100% sure, if someone wants to test, please email me and we can setup a test!) Basically even after we've told clients "doge isn't profitable, go mine moon!", depending on the number of people, we still have a very good chance of finding a doge block in the coming 30 seconds. And if our switcher missed the block change by a split second (and didn't see it for 30 seconds later or something), we could easily be finding blocks.
That said, a LOT of this is tweaking based. And I'm constantly switching it up. New endpoints make this a lot easier too, since I can just switch one set of them (faster reaction time for USEast for example, slower for USWest) and see the results by zone pretty nicely. Just a matter of playing with it Smiley
Hi pw,
Another off-forum response because I am not looking to raise any doubts publicly about the pool. I am fully aware that it takes time to get things running smoothly. (Also, it is simply frustrating to have someone who does not understand the underlying issue respond with an obviously incorrect explanation taking us all off course. The only opinion I care to hear at this point is yours.)
The reason I chose to call attention to this doge block is because it took much longer than the 20-30 second example that you posted, in this case an entire 66 seconds for it to be solved, which suggests an algorithmic inefficiency to be examined more closely and optimized if possible.
85632 was the previous block solved by someone else. 85633 was our block, which was a 66 second block.
preceding doge block # 85632 2014-02-04 03:43:18 496705.02917333
(the reward not exceptional, but I can easily accept how at the time it was an appropriate candidate for better than average profitability)
our doge block # 85633 2014-02-04 03:44:24 8b67e7e8199c7faf046648f3cb7eed7c5b6b5efacd786cc1e39086c25f5777cf 193,881.10278679 valid
Again, note a block time of 66 seconds.
I was not able to determine from the stats page how long we were mining the preceding doge block, but I do know when (ideally, excluding operational latencies) we should have stopped.
With a doge network target of a new block every 60 seconds, if the switcher cannot get us away from a lousy reward block until even after a third block is expected on average (60 + 6 seconds), then the operational efficiency of the algorithm may ultimately result in no better than random results with doge!! You have a lot more data available to you that I do on the stats page, and I suggest taking a harder look at it. Thanks.
Another off-forum response because I am not looking to raise any doubts publicly about the pool. I am fully aware that it takes time to get things running smoothly. (Also, it is simply frustrating to have someone who does not understand the underlying issue respond with an obviously incorrect explanation taking us all off course. The only opinion I care to hear at this point is yours.)
The reason I chose to call attention to this doge block is because it took much longer than the 20-30 second example that you posted, in this case an entire 66 seconds for it to be solved, which suggests an algorithmic inefficiency to be examined more closely and optimized if possible.
85632 was the previous block solved by someone else. 85633 was our block, which was a 66 second block.
preceding doge block # 85632 2014-02-04 03:43:18 496705.02917333
(the reward not exceptional, but I can easily accept how at the time it was an appropriate candidate for better than average profitability)
our doge block # 85633 2014-02-04 03:44:24 8b67e7e8199c7faf046648f3cb7eed7c5b6b5efacd786cc1e39086c25f5777cf 193,881.10278679 valid
Again, note a block time of 66 seconds.
I was not able to determine from the stats page how long we were mining the preceding doge block, but I do know when (ideally, excluding operational latencies) we should have stopped.
With a doge network target of a new block every 60 seconds, if the switcher cannot get us away from a lousy reward block until even after a third block is expected on average (60 + 6 seconds), then the operational efficiency of the algorithm may ultimately result in no better than random results with doge!! You have a lot more data available to you that I do on the stats page, and I suggest taking a harder look at it. Thanks.
These small doge blocks are just issues that need tweaking (slightly faster response time). The only times I've seen this happen are in special circumstances (where we didn't intend to mine this block, it just happened). When I've seen it, we start mining a high value block (say 700k or so), and between the time we start mining, and the time we check profitability again (can take 10+ seconds due to exchange pricing delays, block processing cpu spikes, etc), someone already mined the high-value block, and we mined a small block immediately after it. I've also seen it happen (not on doge, but on luckycoin), where we will notice the block value change, tell miners to switch, and have some miners (especially on very high intensity) still submit shares 2-3 seconds after being requested to switch. When these shares come in, we still process them for a bit (until that coin's block changes), and if someone submits a valid block for a coin we just stopped mining, we'll still submit it to the network (why the hell not!), and can get smaller block rewards.
In the first of the cases, we can fix it by watching closer (which just needs some tweaks), the second case is just a matter of a few seconds of jump time between coins, which gets solved (slightly) with more endpoints (lower latency).
In the first of the cases, we can fix it by watching closer (which just needs some tweaks), the second case is just a matter of a few seconds of jump time between coins, which gets solved (slightly) with more endpoints (lower latency).
I thought perhaps the reward estimate might have been off at the time, but I've been following the recent estimates posted on the stats page and the subsequent generation rewards in the block chain and they indeed do track properly. I then considered the possibility that we very quickly found the next block after the one deemed to be profitable -- before being able to switch to another coin, but there was a 66 second gap between the two blocks being found, so why didn't we switch to another coin? What happened? At 193,881 doge reward, the profitability for that block would have fallen below even that of litecoin, which always has sufficient exchange volume, so how is it that we were still allocating our resources to doge under those circumstances?
You're absolutely correct on those blocks. We've been tweaking (every day or so) to see what the right amount of time between coin switches is (how quickly we should react to new coins being most profitable). The answer right away might be "instantly", but there are some subtle problems with it unfortunately.
Switching coins extremely quickly (and for some coins, this is every 3-4 seconds in testing - will be better after we remove some super fast coins) causes a ton of extra load on the server in terms of bandwidth (sending all new work to a thousand clients), and this actually props up our orphan rate if we find a block during those times (and if we're switching low-diff coins quickly, we find blocks pretty rapidly) because our network upload gets a bit slow, and block propagation speed drops, leading to orphans. One solution that we've played with here is to raise our transaction fee minimums on a per-coin basis, leading to smaller blocks (quicker propagation), and less outgoing bandwidth on new work for miners, but at the same time, drops profitability since you're not getting as much in transaction fees, its also detrimental to the individual coin network.
Another issue is queued work on clients. A large number of clients, even after they've been asked to switch coins, will continue submitting work on the old coin for up to 30+ seconds (I think this is due to queued work, not 100% sure, if someone wants to test, please email me and we can setup a test!) Basically even after we've told clients "doge isn't profitable, go mine moon!", depending on the number of people, we still have a very good chance of finding a doge block in the coming 30 seconds. And if our switcher missed the block change by a split second (and didn't see it for 30 seconds later or something), we could easily be finding blocks.
That said, a LOT of this is tweaking based. And I'm constantly switching it up. New endpoints make this a lot easier too, since I can just switch one set of them (faster reaction time for USEast for example, slower for USWest) and see the results by zone pretty nicely. Just a matter of playing with it Smiley