I have a Gigabyte Gaming 1070 Ti and a MSI 960. I am currently using the AORUS software to do the OC'ing with their preset "Overclock" function. I can and have used MSI Afterburner, but sadly I really don't know the first thing about what I should look for and how to establish stability with low risk.
I don't mind if the wattage goes up a little (electricity is pretty cheap), but I do want the hashrate to go up considerably. All I really want to know is what setting should I use that others may have had some success with.
The only other algo I really use is Neoscrypt (which is more memory intensive).
That Overclock button is probably for gaming.
I would suggest getting something like the free Nvidia Inspector.
First, put the Power Target to 70-75%. This way it's less probable anything will crash.
There are two parameters you can tweak:
Base Offset (core clock) and Memory Offset.
You can leave Memory offset at 0, or even put it in negative as it doesn't affect this particular algo almost at all.
Increase the core clock slowly. You should reach a point where additional increasing doesn't increase the hashing power.
If you get to a point where the miner crashes, you obviously ran over all limits.