Iptables has a couple counterintuitive concepts that aren't flat apparent, reminds me of a crazy commission determination engine I worked on once.  I can't give you a better answer than that because I'm not highly knowledgeable about it myself and my unix machines are all fine now and it's been awhile since I had an iptables problem.  I **think** I solved my last issue by an answer at stackexchange.

You can keep digging at iptables via that suggestion and you may find your answer, but DDOS attack and protection is very hard even for the folks that are experts in that or can throw lots of money to those folks to help.  Honestly I'd start with switching VPS to one that provides DDOS as part of the package if I couldn't solve in a couple days.  Sometimes throwing money at it *IS* the best way to solve a problem.