This is my understanding as well, but one problem with increasing the scratchpad size to defeat ASICs is that it also reduces the hashrate of GPU/CPU miners; perhaps monotonically So that seems like a shoot-yourself-in-the-foot kind of solution.
Why is that a problem? If the whole network decreases hashrate by 50% nobody loses but the ASICs...
Because changing the computation - ie, tweaking the algo - renders the ASIC obsolete (the computation is literally hard-coded into silicon) and this won't affect GPU/CPU software at all (just change the code and recompile).
EDIT - sorry, I didn't address your actual rebuttal that if the entire network hashrate decreases then - presumably - so will difficulty. Yes, that may work.