Not at all. It is only the hashrate calculations that are affected. The accepted shares, are calculated correctly and payouts are also calculated correctly.
I've figured out what was going on. I was using sgminer 5, and upon connecting to the pool the diff was getting set to 20.480. I've just tried it with 4.1 and the diff is .16 and shares are actually going through at a decent rate like they should.
If anyone else experiences this with sgminer 5 (which is still a beta I think) then use an older version!