No, you are not alone. I've tried 10.0 and 10.5 and get the same behavior. The hash rate slowly declines over time. I'll start at 30.15 and after an hour it'll be at 27.5 or so. Sometimes it creeps down farther. I've set the -minspeed 28 to try to get a restart when it happens. Occasionally it will get stuck just over 28 so it doesn't reboot. I might tick it up a bit to 29 to see if I can maintain the rate.
I'm on Win10 64-bit FCU using the Aug 23rd Beta BlockChain drivers in a test machine with a single RX580.
To be honest, if I sit and watch it for a bit I can sometimes see it drop the rate while I watch. I suspect it has a thread or two dying inside and isn't able to relaunch them. If yours is always after you walk away, I would head straight into the Control Panel and check to see if you have some power saving mode selected after some period of inactivity.