If they don't have access to the server seeds, why shouldn't they? After all, they can just use another alias and you won't know.
It's called the appearance of impropriety. If something looks shady, some people aren't going to delve into the facts to find out if it's actually shady or if it's above board...they are just going to move on.