Those time stamps are there for other reasons that may someday in the future be useful (like contract enforcements, or marking an anniversary, or something) It does not have any affect on the algorithms that govern the block chain.
Are you stating that timestamps aren't used to calculate the difficulty? Are you stating there are no possible manipulations of the difficulty via timestamps that could be exploited? If yes, where I can read the analysis?
Edit: I am genuinely interested in analysis of difficulty attacks as it helps me with my work. So I am curious if you know something I don't. Because I am not 100% certain there are no such exploits.
Edit#2: I realize it can be a pain to refute such general attack vectors, and the onus should be on the attacker to prove he has an attack. This is what BCX's reputation has afforded him. I thinking he won't trash his reputation.