This is the bottom line, really. It doesn't matter how good your system is or how strong your memory is, when everything from a stroke or an aneurysm or even a simple fall and blow to the head could render the whole thing useless. If I gave you a piece of paper and said "Use this to back up your seed phrase, but there is a not insignificant chance it will spontaneously combust with no warning whatsoever and you'll lose everything", you would of course refuse. Relying on your memory is really no different.
And this is true not only for memorizing the seed, but also for some homebrew crypto schemes - someone can easily lock themselves out if they forget their algorithm; or with hiding the seed very deep, like for example burying it somewhere and then suffering from memory loss and not being able to find the location.