... So then even when the difficulty is at retarded proportions, the productivity won't necessarily have to decrease?
When total network was 1GHash/sec a block took 10 minutes, same as now. The whole point of difficulty is to keep productivity the same.