I'll ask again since nobody replied the last time...
If difficulty doesn't matter, why do pools even have a choice on what difficulty to go with? Why is there VarDiff? Since difficulty is meaningless, people who spent all the time developing those systems (as well as those who implemented them into their miners) wasted their time on something that supposedly means absolutely nothing.
Since difficulty has no real value, every pool/miner should be set at a hard-coded 1024 or 2048 or (insert difficulty here). We clearly have no need for more than one option.
Ideally you want the lowest difficulty that doesn't overload the servers or your bandwidth, which will vary on your hashrate. If you have one GPU, you don't want the diff so high that you aren't telling the pool that you are working very often. If you have a GPU farm, there is no need to have a low diff that is constantly sending the pool shares.
As I discussed in more detail a dozen posts or so above, it doesn't matter over the long-term, but lower difficulty will give more reasonable day-to-day variance to slow hashers.