The odds of a counterwallet passphrase being brute-force discovered are roughly the same as any Bitcoin private key being brute-force discovered.
Indeed this is not what to worry about. What to worry about, are trojans, rootkits, keyloggers, screen scrapers, nosy coworkers, etc.