Even though I don't believe I have anything to worry about, I'd still prefer to add more signers to the multi-sig so I don't have full control anymore. This would make it safer for everyone, I really do not like the fact that users have to trust me.
Let's assume there are multiple trusted signers, and the system is nicely decentralized. Would it still be possible for you to pull an exit scam by creating notes that give you access to large funds? How would the signers know whether or not the note was created legitimately after a deposit?