You don't need to restart when making the changes from the interface - only if you're changing the batch file etc.
To prove this, drop your gpu clock to something very low and watch what happens.
I only ever tweak my miners via the menus then, once i find the sweet spot, i lock it in with the batch file.
Doesn't answer your question as to why it drops though. Do you have --auto-gpu/fan/mem/temp set in the config?
auto-fan is on. auto-gpu is not. I set target-temp etc, but (according to the readme) this shouldn't effect the clock speed because auto-gpu is not on.