Even if someone knows the 6 words of your phrase they will only end up importing a different wallet unless they have full access of your whole phrase.
About brute forcing seed phrase, on a CPU (I mean computer), 4 words can be brute forced in some minutes, while 5 words can be brute forced in some hours. 6 words shouldn't take more than 2 or more days. I have not practically tested this before though. As long as an attacker has the remaining words correctly and not scattered or disarranged, brute forcing it is possible.