It is way less secure than using a strong cipher with the same secret data.
Your mechanism leaks bits of the plaintext, which is always bad.

You could have just used your 4 dates or whatever shit you are using and use a proper encryption cipher.
Then no single bits would have been leaked and you'd be pretty fine.

How do I explain to my mother to AES decrypt "71TjQQYPkadCq8qUA6Lqt7FhUBEjPSzgDSbBA6spbtD/j8v3JXp9Vpco0H8rS/TK2/IOMS0aHF5QIyLihGuP2dSgdoKdyDrb82O72tNPdT4=" and ensure to type it out correctly?

Birthdays and anniversaries everyone remembers, and with 24 seed words you can shift it with up to 8 dates. Never said it's unbreakable, but it's not easy to break either, it gives you plenty of time to react in case of theft and it's simple enough by knowing the dates to do it by hand.