You can. Depending on your method of generating it, it could take a reasonably long time for you to crack it.

I highly doubt anyone wold be able to memorize such a confusing passphrase accurately over long periods of time without confusing which of the letters are changed.

BIP39 is actually very useful in terms of the properties that it has. The range of words allows easy correction of the seeds, there is a built in checksum with the seed, it doesn't have to rely on the user's judgement to make a secure password. I bet if people are actually forced to create their own passphrase this way, I would see loads of people asking how to bruteforce their passphrase because they cannot figure out which symbols they inserted. What you've described is basically brainwallet which has been cracked thoroughly and isn't recommended for use. It could be more secure it being salted and it would take a much longer time to bruteforce.

You can certainly keep a passphrase this way, I wouldn't recommend it but it's definitely possible.