I have a similar issue but with Scrypt on multipools (Clevermining, in this case). Nwools noted that it's a known issue with BFGminer that the hashrate will drop as the multipool switches blockchains. I'm using a mix of Gridseed Orbs and a Blizzard. I'm not sure if this has been fixed in version 5.0.
Are you using a multipool, Dimoza? If yes, that might be issue.