Okay I solved this problem by doing a double spend.

For all future readers:

What I did was export the private keys off of Blockchain and into my own wallet (bitcoin-qt). Then I made a payment to myself. This one got confirmed, which makes the older one a double spend.

After a while, the old ones that never confirmed should disappear.