Your brain is much more susceptible to damage, and inaccuracies than other physical items.
This is the bottom line, really. It doesn't matter how good your system is or how strong your memory is, when everything from a stroke or an aneurysm or even a simple fall and blow to the head could render the whole thing useless. If I gave you a piece of paper and said "Use this to back up your seed phrase, but there is a not insignificant chance it will spontaneously combust with no warning whatsoever and you'll lose everything", you would of course refuse. Relying on memory is really no difference.
Nobody should, but we may have to in case the government seizes all our belongings.
You should have multiple off-site back ups anyway. Backing up everything in the same location as your main wallet (i.e. in your house) provides no protection against fire, flood, or other natural disasters. If the government or law enforcement raid your house, your off-site back ups will be safe provided you have kept their existence private. Use passphrased wallets, or even better multi-sig, for this set up for additional protection.