OBC and POBC don't rely on the Bitcoin transaction rules either to verify transactions: the Bitcoin protocol has no knowledge of colors, so it's not possible to rely on that.
If you make an invalid transaction, depending on how inputs and outputs are setup, you may uncolor some or all of the coins you are transferring. If that is the case, they cannot be spent again (they have been uncolored, and no longer exist).
If there was such a thing as a color-aware client and miners were running it, it could reject such transactions.
You can check the example at the bottom of the document, it walks though the coloring process.
Okay, that makes sense. The spec is a bit light on discussing what constitutes a valid vs. invalid transaction. Can there exist a valid transaction with invalid outputs, or does any invalid output make the transaction invalid?
If address A has 100 units of color C, and I make a transaction that transfers 200 units of C from address A to address B, what is the result? Does A still contain 100 coins of color C?
Edit: Ah, it appears from the example in the spec that this is a valid transaction, but the coloring would be destroyed by this transaction (similar to Output 9 in the spec).