Will it more difficult to get the key?
Most likely yes, but :
1. It's useless if attacker know you use brainwallet & know this method
2. Unless you write down passphrase for brain wallet, you will forget your passphrase or/and your clever method
3. It's still far less secure than simply use CSPRNG to generate your private key/seed