Another privacy implication is when your public master key is leaked or you send it to 3rd party API (e.g. to retrieve balance), but i guess it's also obvious for you.
Why?
Generally forgetfulness, amnesia or just any other thing that can affect your memory, the list goes on.
I definitely wouldn't take the risk and blame myself later on for forgetting a seed phrase. Just having a physical copy of it and keeping it safe would be fine for most.
Additionally, there's risk you either,
1. Remember the words on wrong order.
2. Remember wrong words which sounds similar (such as "this" and "these").
3. Don't remember the position of words you remember.
Brute forcing or finding which part is wrong aren't easy.