I have 10 X RX580 and after increasing the Virtual Memory to 48000 I still got the errors. After I increased to 64000 it worked smoothly (it may also depend on how many cards you have).
This version is not so easy to use because you have to adjust "-h" and "-dmem" values often, I will try to improve it soon.
1. "-dmem 1" requires twice more memory, make sure you have enough virtual memory. For example, if you have 6 GPUs 8GB, set virtual memory to 6*8=48GB.
2. Check how much memory miner allocates, it shows something like "(Memory used: 5128MB)", make sure you have this memory on your GPU.
3. You can see low hashrate if you use almost all memory on GPU, try to reduce "-h" value a bit.
Guys if you are getting the "GpuMiner dk2an0 failed" error, be sure to note Claymore's comments above. I was initially getting the same error with my RX480 8GB rig. Lowering to -h 1024 will solve the issue but make the program only see 4GB on my cards. After changing page file to 48GB, the miner runs beautifully with default -h and -dmem settings.
Ellesmere RX480 8GB, same modded BIOS, default miner settings:
CryptoNote Miner 9.7 - 850 H/s
CryptoNote Miner 10 - 987 H/s
Thanks Claymore!