I could memorize a 12 word seed and a passphrase

- yes but its much harder to memorize and you would also not be able to quickly transfer it to a family member without revealing the entropy in the transmission

you would have a much greater chance of hacking this than you would of hacking 12 random BIP39 words.

- yes, but it still would not be feasible. Even if I would tell you where the location is you would not be able to recover the key. You would need to spend about 1$ in computational costs per password candidate you try just for the 9 square meter area I told you the key comes from. You would also not know how much is in my wallet so you might spend a billion dollar on a brute force and then end up with a wallet containing 1btc...
One could also choose to create a geokey with a location and 12 random BIP39 words as password if the paranoia or wallet is big enough.