First, try asking in the "Mining" forum where all the hardware geeks lurk.
Second, is your miner running while you tweak the memory clock rate? My system crashes if I tweak card settings while running Phoenix.
Also did you try with some other tweaking tools? I personally use ATI Tray Tools since they work best with my two Sapphire Vapor-X 5870. I've tried Sapphire Trixx, MSI Afterburner and AMD GPU Clock Tool, but only ATT works the way it is supposed to work.