The difficulty won't drop, because miners with less than 1M SC can't mine trusted blocks. So it hasn't any effect on the actual blocks/time. You can decide to not mine SC, and the difficulty will drop a little bit.
The difficulty will drop. The difficulty for trusted blocks and normal blocks isn't related.