OK so after some time I'm 90% sure it was thermal throttling. Thank you again eckmar.
Strange though, that MEM and GPU clocks are not downclocked in such case.
Throttling is downclocking. What are the GPU temps? If they were too hot they would cool when you reboot.
If rebooting doesn't fix it the problem may be outside your rig like networking issues, pool issues etc.