I'm looking for a way to set a custom fan curve or set a target temperature that will ramp up the fan speed instead of throttling the GPU.
Have you looked into MSI Afterburner? It does a good job with the stock fans on most graphics cards (yes, even those by brands other than MSI).