Your miner will be insecure, but that's not a problem as long as you are using a different password for your user account.

If an attacker gets one of your workers' passwords by sniffing your unencrypted HTTP traffic, I think the worst thing he could do is execute a denial of service attack in your name in an attempt to get you banned from your pool.