I think I've figured it out. It's either that the transaction can be a max of 20 inputs, or the size of the transaction cannot exceed more than 384k bytes. The size of the transaction was the issue, not the number of inputs.

This presents a problem in that in order to get all of my remaining bitcoin out of my wallet, I'd have to do about 10 transactions. Is there no way around this?