If the fans are blowing out of the case remove the fan and find a better place.
If they are blowing into the graphics card and the case fan is slower keep it. If it is alot faster (which is highly unlikely) remove it as it might damage the graphic fan bearings
your reasoning int entirely correct. It might be that it is slower due to more force needed to spin because of the air from the extra fan pushing the original fan in the other way than it is spinning.
SO Iorbas needs to do some more testing.