What about if a disease occurs to you and you can't remember it? You'll lose the wallet!
Not a good idea!
I thought the same thing as you did. For example, if a bitcoin's owner is suddenly being diagnosed with Alzheimer or Dementia - it's a huge possibility that he/she will lose the memory of many things, including the password for brain wallet!