Setbacks there are none unless you talk in your sleep and you sleep among strangers maybe?
What about brain injury or illness? I know that chances of that happening are not very high, but they still exist and shit happens so no way I would trust my memory when it comes to my life savings.
That's why I said it has no drawbacks as long as you have a physical copy of that private key!
OP asked how secure is but only after mentioning he has a piece of paper with the seed from his wallet, I still don't see any disadvantages to it.
Besides, if we talk about brain injury, depending on how serious a memory loss is you might even forget where you hid your private key or that bitcoin is all together!