Your mining hashrate isn't stable mainly because of the tweaking you've done to your card.

Try playing around the overclock settings. Alter the core and memory clocks until you find a stable setting.
Changes in the hashrate should be about +5/-5.

Look for rejected shares in the miner.